Let’s say you’ve got a CI/CD pipeline and you would like to run builds in containers. You could just configure Docker on one of your machines and point your builds there, but why not use something a bit more scalable? Enter Kubernetes, a leading container orchestration platform, which luckily offers several options for using its self-contained pods as on-demand CI builders. Things like sharing sources between containers and networking will be handled for you, so all you’ll have to worry about is specifying the desired image.

In this blog, I’ll be exploring two of those options in commonly used CI tools, namely Gitlab and Jenkins, and will explain how to configure the Gitlab-runner or Jenkins Kubernetes plugin to run on-demand CI builders on a Kubernetes (or “K8s”) cluster.

What do I need?

This guide assumes you already have a running Kubernetes cluster and either Gitlab or Jenkins, running within the cluster or externally. If you don’t, and you’re able to use Helm (basically a K8s package manager), there are predefined deployments for both Gitlab and Jenkins in the form of Helm charts which also configures the runner or plugin for you. If you want to experiment with K8s locally, you could use Minikube or the Edge channel of the Docker client (only for Mac at the time of writing).

The examples below can also be cloned from my git repo here.

Gitlab

The Gitlab-runner supports Kubernetes out-of-the-box. It acts as an intermediary between the Gitlab service itself, containing the webserver, git repos, etc. and the Kubernetes API. The API handles all requests for creating pods, services, and basically anything else you can do on a K8s cluster. For example, when you commit to your repo in Gitlab and you’ve specified

 in your

, it sends the specified image to the runner. The runner then requests a pod from the K8s API containing the node container alongside a “helper” container which handles git cloning and artifacts. Any script in your pipeline will then be executed in the node container and your sources will automatically be available in a volume in said container.

To use the runner on your cluster, you’ll have to configure a couple of things, starting with a service-account and role-based access for the runner pod. This authorizes Gitlab-runner to create pods on the cluster for each CI job and is done by applying a configuration file in YAML with kubectl. Note that this assumes you’re OK with giving the runner permission to do anything in the specified namespace, but you can tweak permissions as needed:

[/crayon]
Save this in a YAML file and if needed modify the namespace or other values, then apply with

. Afterwards, let’s configure the runner. We’ll need a runner registration token for that, so go to Gitlab’s

page to get the token. Then create a configmap which configures the runner by mounting a few files within the Gitlab-runner pod:

[/crayon]
Note that the

option can be used in case Gitlab is running on the same cluster as your runner. Simply specifying the local hostname of Gitlab at

will still cause the runner to do git clones via the external URL, so use the clone-url option to speed up cloning to the runner.

Apply using

again. Finally we’ll create the Gitlab-runner deployment, which runs the actual container within the cluster:

[/crayon]
After applying this deployment, Kubernetes will deploy the gitlab-runner pod, which should automatically register itself to your Gitlab instance. Verify this by running

to see if the pod is running and

to check the logs of the runner.

That’s it! Now go run something. If you don’t have a pipeline yet and want to test this setup, create a new project in Gitlab from a template (such as the “NodeJS Express” template) and modify any file to trigger the predefined pipeline.

Jenkins

Interaction between Jenkins and Kubernetes is handled by (you guessed it) a plugin. This makes configuration slightly easier than Gitlab, but the downside is, of course, yet another plugin. When you specify a

with a desired CI image in a Jenkinsfile, the plugin communicates with the Kubernetes API and requests a pod containing your container alongside a “jnlp-slave” container. This JNLP container acts as a buildslave and handles things such as communication back to Jenkins, git checkouts, and workspace volumes. You’ll find an example further below.

First of all, install the plugin. You’ll also need to make sure Jenkins has enabled the JNLP agent listener on port

, found on Jenkins’

page under “Agents”. If Jenkins is running on this cluster you’ll also need to authorize Jenkins with the service account and role based access configuration:

[/crayon]
Create a YAML file with these contents, modifying any relevant values and then apply with

. Then we’ll configure the plugin itself under “Manage Jenkins” > “Configure System”. If Jenkins is running on the cluster the only settings you’ll need are displayed in the image below. Otherwise, you’ll also need to setup certificate authentication between Jenkins and Kubernetes.
Jenkins kubernetes config
To make it easier, this can be converted to Groovy, so these settings won’t have to be entered manually when your Jenkins container gets killed. The following can either be saved as a Groovy file in your Jenkins image’s

directory or posted via the script console API at

with

as the body. That script being:

[/crayon]
After this, you should be ready to run your first pipeline in K8s pods. Here’s an example to test this setup:

[/crayon]
The

contains the configuration for containers you want to use for CI. Anything specified outside of a

block is executed within the JNLP container, which is where git and any Jenkins plugins are available. Anything within the container block is executed in whichever container name you specify. See the plugin readme for more options.

Which is “better”?

It depends. Gitlab has tighter integration with Kubernetes, also offering options besides CI such as automated production deployments and canary releases. Its pipeline syntax is also far more readable and easier than Jenkinsfiles, though the latter allows for more complex scripting. On the other hand, Gitlab’s CI works best if your sourcecode is also versioned there, which may not be an option for many of you, making Jenkins more flexible. So basically, pick whichever CI tool you prefer based on its features, as the act of running CI in K8s pods is pretty similar between the two.

What else is there?

Take a look at Brigade by Microsoft, which is an event-driven JavaScript framework for Kubernetes. Combined with “Kashti” dashboards, it makes for an interesting CI/CD option. I plan on looking into it in the coming weeks, so you can expect to see a similar post based on Brigade soon.