Build a Continuous Integration pipeline using GitHub, Docker and Jenkins on Oracle Cloud Infrastucture

Abhiram Annangi
Principal Product Manager - Cloud Marketplace

In my previous blog post, we discussed how to deploy Jenkins on Oracle Cloud Infrastructure, and dynamically scale it by leveraging the Master/slave architecture of Jenkins, using the Oracle Cloud Infrastructure Compute plugin. In this post, let’s look into how to setup a Continuous Integration pipeline on Oracle Cloud Infrastructure (OCI), utilizing the Jenkins setup we already created.

The most important step for continuous delivery of software is Continuous Integration (CI). CI is a development practice where developers commit their code changes (usually small and incremental) to a centralized source repository, which in turns kicks off a set of automated build and tests. This gives them an opportunity to capture the bugs early and automatically before passing them on to production. Continuous Integration pipeline usually involves a series of steps the software takes starting from code commit to performing basic automated linting/static analysis, capturing dependencies and finally building the software along with performing some basic unit tests before creating a build artifact. Source Code Management systems like Github, Gitlab etc. offer web hooks integration to which CI tools like Jenkins can subscribe to start running automated builds and tests after each code check-in. In this tutorial let's look at how to run a continuous integration pipeline using Jenkins on OCI.

Pre-requisites

To run through this tutorial, you will need the following
  • Install and configure Jenkins as discussed in this blog post
  • A Github account
  • Access to Oracle Cloud Infrastructure Registry

Overview

Jenkins, like a few other CI/CD softwares provides us flexibility to define the entire build pipeline (build/test/deploy) programmatically. This is called pipeline as code. The pipelines in Jenkins are defined in Groovy DSL in a special file called Jenkinsfile. In this tutorial, utilizing the Jenkinsfile, we shall create an automated Continuous Integration (CI) pipeline where a code commit or a pull request to Github triggers the following Pipeline jobs in Jenkins and return the status to Github indicating whether it failed or succeeded.
  1. Checkout source code from Github
  2. Fetch necessary code dependencies
  3. Build a Docker image
  4. Perform a set of unit tests
  5. Push the docker image to a private Docker registry - Oracle Cloud Infrastructure Registry
Oracle Cloud Infrastructure Registry is an Oracle-managed registry that enables you to simplify your development to production workflow. We will be using this as a private registry to push docker images.
Note: We already setup Jenkins to operate in Master/slave mode, where master instance purely plays a management role in the build process, while the entire build/test happens on Jenkins slave node(s)
Let's look at how to create this Jenkins CI build pipeline on Oracle Cloud Infrastructure, in a few easy steps:

Step 1: Configuring Jenkins slaves

  • Login to the Jenkins Master instance and navigate to Cloud config section under Manage Jenkins > Configure System menu
  • Under the Advanced section in Instance Templates, edit the init script to include the following. This init script is used to install and run Java, Git and Docker-engine. We will be using Docker to build and push docker images.
sudo yum update -y
sudo yum install -y java git docker-engine
sudo systemctl start docker
sudo systemctl enable docker
  • Make sure the Labels field under the same Instance Templates section has jenkinslaves as it's value. We will be using this label in our Jenkinsfile.
  • The above init script installs the Git executable in /usr/bin/git directory on the slave nodes. Go to Manage Jenkins > Global Tool Configuration and edit the Path to Git executable for Jenkins to locate it.
  • Finally, navigate to Manage Jenkins > Manage Plugins section, under Available tab, search for Blue Ocean plugin and Github Pipeline for Blue Ocean plugin. Install them without restart. The Blue Ocean plugin creates a more sophisticated visualization for the build pipeline on Jenkins. It also makes it easier to integrate with SCMs like Github. We will be utilizing these plugins in steps 4 and 5.

Step 2: Configuring Oracle Cloud Infrastructure Registry

  • Our Jenkins pipeline pushes the final docker build artifacts to Oracle Cloud Infrastructure Registry. We shall require an auth token to access it. If you already have an auth token available, you can skip this step.
  • If not, please refer to this documentation to generate an auth token in a few easy steps. Keep this token handy, as we will be using this later to configure our pipeline.

Step 3: Configuring Jenkinsfile and Dockerfile

  • Let’s configure the Jenkinsfile for running a build pipeline and Dockerfile to build the resulting Docker image.
  • The Jenkinsfile for our setup specifies an agent with label jenkinslave, which tells Jenkins to run the build jobs on the slave node, instead of on the master (since we configured the slave node with label jenkinslave in step 1 of our configuration). The stages of the build pipeline contain 1) Fetch Dependencies 2) Build Docker image 3) Test image 4) Push image to Registry, as shown
Note:  For the purpose of this demo we are just running shell commands with credentials in clear text, to push the docker image. The ideal way would be to install plugin cloudbees docker build and publish and use it's wrappers within Jenkinsfile.
  • Update the Jenkinsfile with your OCI Registry credentials. The user name is "OCI Tenancy name/OCI username" (For ex: foo/abhiram.annangi@oracle.com). The password is the OCI auth token you generated in step 2. In this example, I used the Registry in Ashburn (iad.ocir.io), but if your tenancy is in a different region, use the appropriate region specific Registry name (For ex: Registry in phx has phx.ocir.io)
  • In the Dockerfile, we just set the maintainer, health checks and expose an arbitrary port. The Dockerfile looks like

Step 4: Integrating Jenkins with Github

  • Clone this Github repository used in this tutorial to your personal Github.
  • If you already have not created Github access token, create one. You will be needing it to have Jenkins to scan through your private repositories.
  • Go to Open Blue Ocean in your Jenkins main dashboard and create a new pipeline by using the steps listed in this post.
Note: The above integration with Github creates an on-demand build pipeline. For auto-triggering builds for any changes in your Github, you should subscribe to Github webhook. The official documentation from Cloudbees, for GitHub integration with Jenkins using webhooks can be found here.
  • Once you successfully link Github with Jenkins, the build pipeline will automatically get triggered.
  • This will take a few minutes, as the Jenkins master dynamically launches a Jenkins slave instance to run the build process. Jenkins job gets queued until the Jenkins slave agent with the label jenkinslave gets provisioned.
  • If you go to OCI console, you should be seeing a Jenkins slave instance being provisioned. The build will begin as soon as the slave instance is provisioned and the init scripts are executed.

Step 5: Build execution

  • Once the Jenkins slave instance is provisioned, you should see the entire build pipeline go through. This is how it should look on the Blue Ocean dashboard.
  • The docker image is finally pushed to Oracle Cloud Infrastructure Registry. You can log into your OCI console and under Containers > Registry you will notice the docker image.
This concludes this tutorial for setting up a Continuous Integration pipeline using Jenkins on Oracle Cloud Infrastructure.

Suggested Enhancements

  • Configuring Jenkins with SSL using a reverse proxy - By default, Jenkins comes with an embedded Winstone server which enables Jenkins to run as a standalone application. Winstone supports basic web server functionalities, however if you are planning for a production deployment of Jenkins, we recommend using some kind of reverse proxy servers to front your Jenkins installation. This helps secure Jenkins with SSL to protect passwords and other sensitive data. You can use an Nginx or Squid reverse proxy or even an OCI Load balancer service to terminate SSL. Here is a blog post to configure Squid on Oracle Cloud Infrastructure.
  • Containerizing Jenkins Master - In this tutorial we were running Jenkins master as a virtual machine and scaling out Jenkins slave virtual machine instances dynamically and on-demand. Since Jenkins master is mostly involved in directing the traffic to slave instances for build jobs, it can as well be run as a docker container as opposed to a full fledged virtual machine, while still utilizing Oracle Cloud Infrastructure Compute plugin for Jenkins. This gives better resource efficiency.
In my next blog post, we shall complete the continuous delivery by setting up a Jenkins continuous deployment (CD) pipeline to pull the docker image from our private registry and deploy it into Kubernetes cluster, using Oracle Container Engine for Kubernetes (OKE)

Comments

Post a Comment

Popular posts from this blog

Flutter for Single-Page Scrollable Websites with Navigator 2.0

Image
  Single page scrollable websites are everywhere. In this website design pattern, all the content is presented in one long-scrolling layout that contains multiple sections. Visitors can scroll or jump to a section by clicking buttons of a sticky menu. This pattern is a good fit for small content such as brochure websites, software library documentation, portfolios, and landing pages that are used to convert users [ 1 ]. Designers also love this pattern because it is simple, clean, and enables cool scroll animations. Jetbrains Mono single-page scrollable website Hive docs multi-page scrollable website In this series of articles, we will explore how to build a  single-page scrollable website  using Flutter. We will benefit from the Navigator 2.0 API to provide a good navigation experience to the users. You can find the source code of this series’ sample apps on my  Github page . The implementation details of the sample apps will be explained in the following articles: ...
Read more

A Data Science Portfolio is More Valuable than a Resume

Image
A Data Science Portfolio is More Valuable than a Resume Benjamin Obi Tayo Ph.D. Following Nov 20  · 5 min read D ata Science is a practical field. Hands-on skills are very important especially when you are interested in working outside academia as a practicing data scientist. In academia, you need more theoretical and research skills. While in-depth knowledge in the theoretical foundations of data science is important, as a practicing data scientist, hands-on experience is very crucial and one way to showcase you hands-on skills is via building a very great portfolio. Companies interested in hiring you would definitely be asking you for a portfolio, as it gives evidence of your strengths in fundamental data science concepts. While a data science resume is important, a data science portfolio is more valuable than a resume. This article will discuss 3 important platforms that can be used for portfolio buildi...
Read more

Better File Storage in Oracle Cloud

Image
Use Oracle Application Express with Oracle Cloud Infrastructure REST APIs to upload, download, and manage big files in Oracle Cloud. By Adrian Png November 27, 2019 Oracle Cloud Infrastructure  provides a comprehensive suite of REST APIs for accessing and managing cloud resources. For example, administrators can use REST APIs to create and manage autonomous databases and initiate backups. Oracle Application Express (Oracle APEX)  makes working with REST easy. The  APEX_WEB_SERVICE  package, available since Oracle APEX 4.0, provides developers with a function for consuming REST APIs. Oracle APEX 5.0 introduced the  APEX_JSON  package, allowing easy parsing of JSON responses. Oracle APEX 18.1 introduced the Web Source Module (WSM), which enables developers to access REST services and use the data in APEX components such as reports, interactive reports, and interactive grids. These features, when used together with Oracle Cloud Infrastructure REST ...
Read more