Understanding GitLab Runner

2019 March 22

GitLab offers an excellent continuous integration (CI) service. They even provide some free worker machines shared among all users, but they're underpowered, oversubscribed, and each user is limited to 2000 minutes per month (you can check your usage in your settings).

One nice feature about GitLab CI is that it lets you Bring Your Own Compute. We can set up our own machines running the GitLab Runner service, register them on GitLab, and then our projects can start sending CI jobs to those machines.

This post tries to explain how GitLab Runner works so that when you try to install, launch, and configure it, you'll know what questions to ask.

Definitions

First, we need to clear up some terminology, especially "GitLab Runner", which is overloaded:

  • Your GitLab instance is the GitLab service hosting your projects. It is identified by its URL. For most of us, it will be https://gitlab.com.

  • Groups in GitLab are like organizations in GitHub. Individual users can be members of groups, and groups can have projects.

  • From the perspective of a GitLab instance user, a GitLab Runner is a worker machine that executes your project's GitLab CI jobs, through work stealing. Each Runner, as it becomes available, continually sends requests to the GitLab instance, asking to be assigned jobs. Different Runners can have different capabilities (e.g. size, speed, or operating system), and they include these capabilities in their requests via tags that the GitLab Runner administrator includes in its configuration. To match your jobs to the right Runners, you have to tag your jobs as well; only Runners with all the same tags as a job will be eligible to execute that job. A Runner can be specific (available to individually chosen projects), group (available to all projects in a group), or shared (available to every project in the GitLab instance).

  • From the perspective of a GitLab Runner administrator, a GitLab Runner is an abstract object within a single gitlab-runner process, which can host multiple GitLab Runners. In this sense, a GitLab Runner is not a machine waiting for jobs the way a user might think of it. Instead, a GitLab Runner is like a service responsible for executing one or many jobs and reporting their progress back to the GitLab instance. It can execute the job on the same machine as its host gitlab-runner process, or on a different machine. That machine might exist before the job is submitted, or it might be launched on the fly in response to the job's submission.

From this point on, I will consistently use "GitLab Runner" to refer to the abstract workers and "gitlab-runner" to refer to the software or process.

Commands

When you install gitlab-runner, know that it is generally installed as a long-running background service. On Linux, you can check on it with systemctl:

$ systemctl | grep gitlab
gitlab-runner.service             loaded active running   GitLab Runner
$ systemctl status gitlab-runner.service
● gitlab-runner.service - GitLab Runner
   Loaded: loaded
   Active: active (running) since Fri 2019-03-01 11:05:36 CST; 3 weeks ago
 Main PID: 929 (gitlab-runner)
    Tasks: 16 (limit: 19660)
   CGroup: /system.slice/gitlab-runner.service
           └─929 /usr/bin/gitlab-runner run

There are a few commands for interacting with the service: install, uninstall, start, stop, restart, status:

$ sudo gitlab-runner status
gitlab-runner: Service is running!

These commands are deprecated and will eventually be removed. stop and restart have corresponding signals, but the rest do not. It seems that GitLab expects us to use our platform's service manager (e.g. systemd) for the rest.

# Restart the service with a signal...
$ sudo killall -SIGHUP gitlab-runner
# ...or with systemd.
$ sudo systemctl restart gitlab-runner.service

Registration

GitLab Runners participate in a distributed work-stealing algorithm. They continually poll a GitLab instance for jobs that match their tags, execute them, and communicate the progress and results back. Even though your gitlab-runner service is running, it is empty: it is not hosting any GitLab Runners. It can host more than one, which is a good thing, because each GitLab Runner is configured with many parameters, and every combination of parameters we want to use requires a distinct GitLab Runner.

GitLab Runners are configured in the configuration file for gitlab-runner. The file's location depends on how you run gitlab-runner. The background service on Linux, which runs as root, uses the configuration file at /etc/gitlab-runner/config.toml.

Due to the idiosyncracies of GitLab Runner tokens, you cannot just copy the configuration of a GitLab Runner. Instead, Runners must be added incrementally with gitlab-runner register, which takes a registration token and exchanges it with the GitLab instance for a runner token that is stored in the configuration file. It is this registration process that (1) tells your GitLab Runner which GitLab instance to ask for jobs and (2) tells your GitLab instance that it is allowed to assign your jobs to your GitLab Runner.

Where to find the token you need depends on the type of Runner you want to register: shared, specific (project), or group.

$ sudo gitlab-runner register
Running in system-mode.

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
https://gitlab.com/
Please enter the gitlab-ci token for this runner:
abcd1234
Please enter the gitlab-ci description for this runner:
[ubuntu]: linux-t2.micro
Please enter the gitlab-ci tags for this runner (comma separated):
linux,t2.micro
Whether to run untagged builds [true/false]:
[false]:
Whether to lock the Runner to current project [true/false]:
[true]:
Registering runner... succeeded                     runner=abcdefgh
Please enter the executor: parallels, kubernetes, ssh, virtualbox,
docker+machine, docker-ssh+machine, docker, docker-ssh, shell:
shell
Runner registered successfully. Feel free to start it, but if it's
running already the config should be automatically reloaded!

Configuration

After you register your Runners, you can edit their configurations and have the gitlab-runner service reload them.

Each Runner is configured with its own executor that defines how it executes jobs. There is a shell executor that executes jobs on the same machine as the Runner. There is an ssh executor to execute jobs on an existing remote machine. The docker+machine executor uses Docker Machine to launch cloud instances, as needed, and execute jobs in Docker containers on those instances.

You may choose to configure a fleet of Runners offering a range of sizes, speeds, operating systems, or installed dependencies. If you do, be sure to assign tags to those Runners so that your jobs can differentiate among them.