Autoscaling GitLab Runner Instances on Google Cloud Platform
Reading time7 Minutes
tl;dr: Migrating GitLab CI jobs to Google Cloud Platform is possible with little effort due to the good support provided by GitLab and relieves the load off your hardware.
This can be worthwhile even for small projects or private GitLab instances without generating major costs.
Most of the jobs only run for a relatively short time, but require some power during that time, so a cloud provider that charges by the minute would be a good choice.
The goal of the setup is to have a lightweight GitLab Runner5 running on the Gitlab hardware, which takes CI/CD jobs, creates VMs for them in the Google Cloud, and terminates them after execution. If no jobs are currently running, no costs are incurred for computing power in this way.
Google Cloud Platform Project
First, we create a project, a service user, and a storage bucket for the cache in Google Cloud Platform.
If you haven’t registered with Google Cloud Platform yet, you get $300 budget for the first year. Only once this budget is used up will there be any real costs.
After the successful registration / login, we can now create a new project with a speaking name.
Within the project we now create a new service account under IAM & Admin > Service Accounts. This account needs the following permissions to start instances later and to access the cache:
- Compute Instance Admin (v1).
- Service Account User
- Storage Object Admin
Depending on how many projects are running in your GCP, the permissions can be further refined or accepted as is for now.
The access data for the ServiceAccount can then be created under KEYS > Add Key > Create new key and downloaded as JSON.
Now that the configuration in GCP is complete, we install the GitLab Runner and configure it accordingly.
GitLab Runner - Docker Machine
The VMs started in the GCP must therefore themselves support Docker. Ideally, one uses an appropriate image for this, Google recommended CoreOS8 for a long time, but this is deprecated and should be replaced by Container-Optimized OS9.
Unfortunately, the Docker Machine Version10 used by GitLab Runner is also no longer actively maintained, so there is an issue here with Google’s Container-Optimized OS images. A workaround is to use a “normal” Linux image like
debian-10 and then install Docker in it.
But the better solution is to replace Docker machine with the GitLab fork11, which also copes with Container-Optimized OS.
The GitLab Runner can either be installed directly on the system or started via Docker.
For simplicity, we’ll use Docker here and build a customized image that includes the fork version of Docker-Machine. The configuration for the GitLab Runner lies in the
./config-gcp folder. We need to copy the GCP
client_secret.json file created earlier here too.
version: "3.8" services: gitlab-runner-gcp: build: . volumes: - "./config-gcp:/etc/gitlab-runner" environment: - "GOOGLE_APPLICATION_CREDENTIALS=/etc/gitlab-runner/client_secret.json"
FROM gitlab/gitlab-runner:latest RUN wget -q https://gitlab-docker-machine-downloads.s3.amazonaws.com/v0.16.2-gitlab.11/docker-machine-Linux-x86_64 -O /usr/bin/docker-machine && \ chmod +x /usr/bin/docker-machine
We can now start the GitLab Runner with
docker-compose up, don’t worry if it complains about invalid configurations, we’ll come to this next.
Creating gitlab-runner-test_gitlab-runner-gcp_1 ... done Attaching to gitlab-runner-test_gitlab-runner-gcp_1 gitlab-runner-gcp_1 | Runtime platform arch=amd64 os=linux pid=8 revision=2ebc4dc4 version=13.9.0 gitlab-runner-gcp_1 | Starting multi-runner from /etc/gitlab-runner/config.toml... builds=0 gitlab-runner-gcp_1 | Running in system-mode. gitlab-runner-gcp_1 | gitlab-runner-gcp_1 | Configuration loaded builds=0 gitlab-runner-gcp_1 | listen_address not defined, metrics & debug endpoints disabled builds=0 gitlab-runner-gcp_1 | [session_server].listen_address not defined, session endpoints disabled builds=0 gitlab-runner-gcp_1 | ERROR: Failed to load config stat /etc/gitlab-runner/config.toml: no such file or directory builds=0 gitlab-runner-gcp_1 | ERROR: Failed to load config stat /etc/gitlab-runner/config.toml: no such file or directory builds=0
Each GitLab Runner must register with the GitLab instance. For this purpose there is the
gitlab-runner register command. We can find the required credentials in the GitLab instance under Admin Area > Overview > Runners.
root@docker:/opt/gitlab-runner# docker exec -it gitlab-runner-test_gitlab-runner-gcp_1 gitlab-runner register Runtime platform arch=amd64 os=linux pid=19 revision=2ebc4dc4 version=13.9.0 Running in system-mode. Enter the GitLab instance URL (for example, https://gitlab.com/): https://YOUR_GITLAB_INSTANCE/ Enter the registration token: GITLAB_RUNNER_REGISTRATION_TOKEN Enter a description for the runner: [4847c6296d8f]: gitlab-runner-test Enter tags for the runner (comma-separated): OPTIONAL_TAGS Registering runner... succeeded runner=XYZABCDE Enter an executor: custom, docker, parallels, virtualbox, docker+machine, docker-ssh, shell, ssh, docker-ssh+machine, kubernetes: docker+machine Enter the default Docker image (for example, ruby:2.6): alpine:latest Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
GitLab Runner now creates
config-gcp/config.toml file based on the input, which we need to customize next.
Configuration - General
# Spawn up to 5 machines concurrent = 5 check_interval = 0 [session_server] session_timeout = 1800 [[runners]] name = "gitlab-runner-gce" url = "https://YOUR_GITLAB_INSTANCE/" token = "SOME_TOKEN" executor = "docker+machine" limit = 5 [runners.docker] tls_verify = false image = "alpine:latest" privileged = false disable_entrypoint_overwrite = false oom_kill_disable = false disable_cache = false volumes = ["/cache"] shm_size = 0
Configuration - Google Cloud Platform
First you need to decide, which machine types you want to create in GCP, in most cases one of the
E2 instance types should fit perfect. I use
e2-highcpu-4 which has 4 vCPUs and 4GB Memory and is quite cheap12.
I also recommend enabling the
google-preemptible flag in your config. This reduces the costs for an instance dramatically (usually the prementible instance costs ~33% of the normal instance) with the disadvantage, that the instance might be terminated by Google. In this (rather rare) case the job fails, and you need to retry it.
You can have a look at the complete documentation on the GitLab Runner configuration13 and adjust the settings for your scenario.
Make sure to set the
google-zone to the same zone as your Storage Bucket, otherwise you might increase the latency.
[runners.machine] IdleCount = 0 IdleTime = 30 MachineDriver = "google" MachineName = "auto-scale-runner-%s" MachineOptions = [ "google-project=GOOGLE_CLOUD_PROJECT", # Depending on your requirements, choose another instance "google-machine-type=e2-highcpu-4", # When running the forked docker-machine, you should use cos-stable "google-machine-image=cos-cloud/global/images/family/cos-stable", # Otherwise you can use debian-10 # "google-machine-image=debian-cloud/global/images/family/debian-10", "google-preemptible=true", "google-zone=europe-north1-a", "engine-registry-mirror=https://mirror.gcr.io" ]
Configuration - Google Storage Cache
Since a new container starts for each job, tools like npm have to re-download all dependencies each time. To reduce this time, project dependencies can be cached14. This cache is then downloaded, unpacked, before the job is run and saved updated after a success.
Among other things, GitLab Runner also supports Google Cloud Storage as a cache hoster, so the configuration is very straightforward15.
[runners.cache] Type = "gcs" Path = "gitlab-runner" Shared = false [runners.cache.gcs] BucketName = "YOUR_BUCKET_NAME" CredentialsFile = "/etc/gitlab-runner/client_secret.json"
Configuration - Summary
config.toml should now look similar to this:
# Spawn up to 5 machines concurrent = 5 check_interval = 0 [session_server] session_timeout = 1800 [[runners]] name = "gitlab-runner-gce" url = "https://YOUR_GITLAB_INSTANCE/" token = "SOME_TOKEN" executor = "docker+machine" limit = 5 [runners.docker] tls_verify = false image = "alpine:latest" privileged = false disable_entrypoint_overwrite = false oom_kill_disable = false disable_cache = false volumes = ["/cache"] shm_size = 0 [runners.cache] Type = "gcs" Path = "gitlab-runner" Shared = false [runners.cache.gcs] BucketName = "YOUR_BUCKET_NAME" CredentialsFile = "/etc/gitlab-runner/client_secret.json" [runners.machine] IdleCount = 0 IdleTime = 30 MachineDriver = "google" MachineName = "auto-scale-runner-%s" MachineOptions = [ "google-project=GOOGLE_CLOUD_PROJECT", # Depending on your requirements, choose another instance "google-machine-type=e2-highcpu-2", # When running the forked docker-machine, you should use cos-stable "google-machine-image=cos-cloud/global/images/family/cos-stable", "google-preemptible=true", "google-zone=europe-north1-a", "engine-registry-mirror=https://mirror.gcr.io" ]
Now that we have configured both the GitLab Runner and GCP, new CI pipeline jobs are now processed by Google instances.
Here it is worth checking again that the instances have the correct settings. Later changes to the config.toml are taken up without restarting the GitLab Runner, so it is quite easy to try out larger / smaller machine types.
After finishing the first job of a project, you should see some content in your cache bucket and the following logs in subsequent jobs:
Restoring cache Checking cache for develop... Downloading cache.zip from https://storage.googleapis.com/.... Successfully extracted cache
Spring Boot Application in OpenShift / OKDNow that we have packaged an existing Spring Boot application into a Docker Image, we can deploy it to a Kubernet cluster as well.
In this example the additional features of OpenShift/OKD are used to enable a continuous deployment of the application.