Build a Continuous Integration pipeline using GitHub, Docker and Jenkins on Oracle Cloud Infrastucture

Abhiram Annangi
Principal Product Manager - Cloud Marketplace

In my previous blog post, we discussed how to deploy Jenkins on Oracle Cloud Infrastructure, and dynamically scale it by leveraging the Master/slave architecture of Jenkins, using the Oracle Cloud Infrastructure Compute plugin. In this post, let’s look into how to setup a Continuous Integration pipeline on Oracle Cloud Infrastructure (OCI), utilizing the Jenkins setup we already created.

The most important step for continuous delivery of software is Continuous Integration (CI). CI is a development practice where developers commit their code changes (usually small and incremental) to a centralized source repository, which in turns kicks off a set of automated build and tests. This gives them an opportunity to capture the bugs early and automatically before passing them on to production. Continuous Integration pipeline usually involves a series of steps the software takes starting from code commit to performing basic automated linting/static analysis, capturing dependencies and finally building the software along with performing some basic unit tests before creating a build artifact. Source Code Management systems like Github, Gitlab etc. offer web hooks integration to which CI tools like Jenkins can subscribe to start running automated builds and tests after each code check-in. In this tutorial let's look at how to run a continuous integration pipeline using Jenkins on OCI.

Pre-requisites

To run through this tutorial, you will need the following

• Install and configure Jenkins as discussed in this blog post
• A Github account
• Access to Oracle Cloud Infrastructure Registry

Overview

Jenkins, like a few other CI/CD softwares provides us flexibility to define the entire build pipeline (build/test/deploy) programmatically. This is called pipeline as code. The pipelines in Jenkins are defined in Groovy DSL in a special file called Jenkinsfile. In this tutorial, utilizing the Jenkinsfile, we shall create an automated Continuous Integration (CI) pipeline where a code commit or a pull request to Github triggers the following Pipeline jobs in Jenkins and return the status to Github indicating whether it failed or succeeded.

1. Checkout source code from Github
2. Fetch necessary code dependencies
3. Build a Docker image
4. Perform a set of unit tests
5. Push the docker image to a private Docker registry - Oracle Cloud Infrastructure Registry

Oracle Cloud Infrastructure Registry is an Oracle-managed registry that enables you to simplify your development to production workflow. We will be using this as a private registry to push docker images.

Note: We already setup Jenkins to operate in Master/slave mode, where master instance purely plays a management role in the build process, while the entire build/test happens on Jenkins slave node(s)

Let's look at how to create this Jenkins CI build pipeline on Oracle Cloud Infrastructure, in a few easy steps:

Step 1: Configuring Jenkins slaves

• Login to the Jenkins Master instance and navigate to Cloud config section under Manage Jenkins > Configure System menu
• Under the Advanced section in Instance Templates, edit the init script to include the following. This init script is used to install and run Java, Git and Docker-engine. We will be using Docker to build and push docker images.

sudo yum update -y
sudo yum install -y java git docker-engine
sudo systemctl start docker
sudo systemctl enable docker

• Make sure the Labels field under the same Instance Templates section has jenkinslaves as it's value. We will be using this label in our Jenkinsfile.
• The above init script installs the Git executable in /usr/bin/git directory on the slave nodes. Go to Manage Jenkins > Global Tool Configuration and edit the Path to Git executable for Jenkins to locate it.
• Finally, navigate to Manage Jenkins > Manage Plugins section, under Available tab, search for Blue Ocean plugin and Github Pipeline for Blue Ocean plugin. Install them without restart. The Blue Ocean plugin creates a more sophisticated visualization for the build pipeline on Jenkins. It also makes it easier to integrate with SCMs like Github. We will be utilizing these plugins in steps 4 and 5.

Step 2: Configuring Oracle Cloud Infrastructure Registry

• Our Jenkins pipeline pushes the final docker build artifacts to Oracle Cloud Infrastructure Registry. We shall require an auth token to access it. If you already have an auth token available, you can skip this step.
• If not, please refer to this documentation to generate an auth token in a few easy steps. Keep this token handy, as we will be using this later to configure our pipeline.

Step 3: Configuring Jenkinsfile and Dockerfile

• Let’s configure the Jenkinsfile for running a build pipeline and Dockerfile to build the resulting Docker image.
• The Jenkinsfile for our setup specifies an agent with label jenkinslave, which tells Jenkins to run the build jobs on the slave node, instead of on the master (since we configured the slave node with label jenkinslave in step 1 of our configuration). The stages of the build pipeline contain 1) Fetch Dependencies 2) Build Docker image 3) Test image 4) Push image to Registry, as shown

Note:  For the purpose of this demo we are just running shell commands with credentials in clear text, to push the docker image. The ideal way would be to install plugin cloudbees docker build and publish and use it's wrappers within Jenkinsfile.

• Update the Jenkinsfile with your OCI Registry credentials. The user name is "OCI Tenancy name/OCI username" (For ex: foo/abhiram.annangi@oracle.com). The password is the OCI auth token you generated in step 2. In this example, I used the Registry in Ashburn (iad.ocir.io), but if your tenancy is in a different region, use the appropriate region specific Registry name (For ex: Registry in phx has phx.ocir.io)
• In the Dockerfile, we just set the maintainer, health checks and expose an arbitrary port. The Dockerfile looks like

Step 4: Integrating Jenkins with Github

• Clone this Github repository used in this tutorial to your personal Github.
• If you already have not created Github access token, create one. You will be needing it to have Jenkins to scan through your private repositories.
• Go to Open Blue Ocean in your Jenkins main dashboard and create a new pipeline by using the steps listed in this post.

Note: The above integration with Github creates an on-demand build pipeline. For auto-triggering builds for any changes in your Github, you should subscribe to Github webhook. The official documentation from Cloudbees, for GitHub integration with Jenkins using webhooks can be found here.

• Once you successfully link Github with Jenkins, the build pipeline will automatically get triggered.
• This will take a few minutes, as the Jenkins master dynamically launches a Jenkins slave instance to run the build process. Jenkins job gets queued until the Jenkins slave agent with the label jenkinslave gets provisioned.

• If you go to OCI console, you should be seeing a Jenkins slave instance being provisioned. The build will begin as soon as the slave instance is provisioned and the init scripts are executed.

Step 5: Build execution

• Once the Jenkins slave instance is provisioned, you should see the entire build pipeline go through. This is how it should look on the Blue Ocean dashboard.

• The docker image is finally pushed to Oracle Cloud Infrastructure Registry. You can log into your OCI console and under Containers > Registry you will notice the docker image.

This concludes this tutorial for setting up a Continuous Integration pipeline using Jenkins on Oracle Cloud Infrastructure.

Suggested Enhancements

• Configuring Jenkins with SSL using a reverse proxy - By default, Jenkins comes with an embedded Winstone server which enables Jenkins to run as a standalone application. Winstone supports basic web server functionalities, however if you are planning for a production deployment of Jenkins, we recommend using some kind of reverse proxy servers to front your Jenkins installation. This helps secure Jenkins with SSL to protect passwords and other sensitive data. You can use an Nginx or Squid reverse proxy or even an OCI Load balancer service to terminate SSL. Here is a blog post to configure Squid on Oracle Cloud Infrastructure.
• Containerizing Jenkins Master - In this tutorial we were running Jenkins master as a virtual machine and scaling out Jenkins slave virtual machine instances dynamically and on-demand. Since Jenkins master is mostly involved in directing the traffic to slave instances for build jobs, it can as well be run as a docker container as opposed to a full fledged virtual machine, while still utilizing Oracle Cloud Infrastructure Compute plugin for Jenkins. This gives better resource efficiency.

In my next blog post, we shall complete the continuous delivery by setting up a Jenkins continuous deployment (CD) pipeline to pull the docker image from our private registry and deploy it into Kubernetes cluster, using Oracle Container Engine for Kubernetes (OKE)