It's templates all the way down

Benjamin Tissoires and I have been busy anthophila and working on the freedesktop CI templates. This post is primarily of interest if you're working on GitLab, specifically if your repo is hosted on gitlab.freedesktop.org. If either of those applies, prepare to be distracted from the current pandemic, otherwise maybe just prepare to be entertained. I'll do my best to be less miserable than the news.

We all know that CI/CD really helps with finding bugs early. If you don't know that yet, insert a jedi handwave before the previous sentence and now you do. GitLab is the git forge now used by freedesktop.org and it comes with a built-in CI system. I'm leaving out the difficult bits such as actually setting the thing up because this is obviously all handled by Heinzelmännchen and just readily available, hooray. I'm also going to assume that you roughly know how to write GitLab CI jobs or, failing that, at least know how to read YAML without screaming. So for this post, we start with the basic problem that your .gitlab-ci.yml is getting unwieldy, repetitive or generally just kinda sucks to maintain. Which is roughly where libinput and libevdev were a while back which caused Benjamin to start the ci-templates.

Now, what do we want? (other than a COVID-19 cure) Reproducible tests, possibly on different distributions, with the same base system across tests. For my repos the goal was basically "test on the common distributions to catch certain bugs early". [1] For Mesa, the requirement is closer to "have a fixed set of images that 'never' change so tests are reproducible". Both goals have much in common.

Your first venture into CI will look like this:

myjob:
  image: fedora:31
  before_script:
    - dnf update -y
    - dnf install -y onepackage twopackage threepackage floor
  script:
    - meson builddir && ninja -C builddir test

So, in short: take a Fedora 31 docker image, update it [2], install the required packages and then run the actual test part - meson and ninja. Easy.

This works fine but it takes approximately forever because dnf update is slow and you're potentially pulling down gigs of packages on every test run. Which is fun, but less so when you have 10 different jobs and they all do that. So let's call this step 1 and pretend we're more advanced than that. Step 2 is where you start building an image you re-use, steps 3 to N are the bits where you learn more than you want to know about docker, podman, skopeo and how many typos you can put into a YAML file. So, ad break, and we jump right to the part where enlightenment is just around the corner or wherever enlightenment lurks these days.

Using the CI Templates

Here's the .gitlab-ci.yml to build a Fedora 31 images with ci-templates and run the test on that image:

include:
  - project: 'freedesktop/ci-templates'
    ref: 123456deadbeef
    file: '/templates/fedora.yml'

variables:
   # project name of the upstream repo
   FDO_UPSTREAM_REPO: someproject/name

stages:
  - prep
  - test

myimage:
  extends: .fdo.container-build@fedora
  stage: prep
  variables:
    FDO_DISTRIBUTION_VERSION: '31'
    FDO_DISTRIBUTION_PACKAGES: 'onepackage twopackage threepackage floor'
    FDO_DISTRIBUTION_TAG: '2020-03-20.0'

myjob:
  extends: .fdo.distribution-image@fedora
  stage: test
  script:
    - meson builddir && ninja -C builddir test
  variables:
    FDO_DISTRIBUTION_VERSION: '31'
    FDO_DISTRIBUTION_TAG: '2020-03-20.0'

Now, you guessed correctly that the .fdo and FDO_ prefixes are used by the templates. There is a bunch of stuff hidden here. Basically, this will:

• check if the image exists in your personal project's registry and use that, but if not
• check if the image exists in the given upstream project's registry and use that, but if not
• create a Fedora 31 image with the given packages installed and pushes it with the tag to the registry
• use that image (whether newly created or pre-existing) and run the tests on it

There are a few more details too, but that's roughly the summary of it. For existing tags, the the myimage job effectively becomes a noop and the myjob job will re-use the image. The image will be in your registry so you can podman run it locally to reproduce a bug.

To build a new image, simply change the tag. Either because you want newer packages or you need extra (or less packages). And the nice thing here: you will build a new image as part of your merge request and run the CI against that new image. But upstream and every other MR will keep using the old image - right up until your MR is merged at which point every (future) MR will use that new updated image.

Want to build a Debian Stretch image? Replace Fedora and 31 with debian and stretch. Same for Ubuntu, Centos, Alpine and Arch though for those two you don't need a version number.

Templating the templates

"But, but, Peter, I want to test on eleventy different distribution like you do" I hear you say. Well, fear not, for this is where the ci-fairy comes in. How about we *gasp* generate the .gitlab-ci.yml file from a base configuration? That can't possibly be a bad idea, so let's do that! First, we save our configuration into the .gitlab-ci/config.yml:

distributions:
  - name: fedora
    tag: 12345
    version: 30
  - name: ubuntu
    tag: abcde
    version: '19.10'
  # and so on, and so forth

packages:
  - curl
  - wget
  - gcc

There is no specific requirement on the structure of the config file, ci-fairy simply loads it and passes it to Jinja2. Your template could thus look like this .gitlab-ci/ci.template file:

include:
{% for d in distributions %}
   - project: 'freedesktop/ci-templates'
     ref: 123456deadbeef
     file: '/templates/{{d.name}}.yml'
{% endfor %}

stages:
  - prep
  - test

{% for d in distributions %}

.{{d.name}}.{{d.version}}:
  variables:
    FDO_DISTRIBUTION_VERSION: '{{d.version}}'
    FDO_DISTRIBUTION_TAG: '{{d.tag}}'

myimage.{{d.name}}.{{d.version}}:
  extends:
    - .fdo.container-build@{{d.name}}
    - .{{d.name}}.{{d.version}}
  stage: prep
  variables:
    FDO_DISTRIBUTION_PACKAGES: "{{' '.join(packages)}}"

myjob.{{d.name}}.{{d.version}}:
  extends:
    - .fdo.distribution-image@{{d.name}}
    - .{{d.name}}.{{d.version}}
  stage: test
  script:
    - meson builddir && ninja -C builddir
{% endfor %}

And to locally generate our .gitlab-ci.yml, all we need to do is

$ pip3 install git+http://gitlab.freedesktop.org/freedesktop/ci-templates
$ cd path/to/project
$ ci-fairy generate-template
$ ci-fairy lint  # checks the resulting YAML for syntax errors
$ git commit .gitlab-ci.yml

And, for reference, the file we generated here looks like this:

include:
   - project: 'freedesktop/ci-templates'
     ref: 123456deadbeef
     file: '/templates/fedora.yml'
   - project: 'freedesktop/ci-templates'
     ref: 123456deadbeef
     file: '/templates/ubuntu.yml'

stages:
  - prep
  - test

.fedora.30:
  variables:
    FDO_DISTRIBUTION_VERSION: '30'
    FDO_DISTRIBUTION_TAG: '12345'

myimage.fedora.30:
  extends:
    - .fdo.container-build@fedora
    - .fedora.30
  stage: prep
  variables:
    FDO_DISTRIBUTION_PACKAGES: "curl wget gcc"

myjob.fedora.30:
  extends:
    - .fdo.distribution-image@fedora
    - .fedora.30
  stage: test
  script:
    - meson builddir && ninja -C builddir

.ubuntu.19.10:
  variables:
    FDO_DISTRIBUTION_VERSION: '19.10'
    FDO_DISTRIBUTION_TAG: 'abcde'

myimage.ubuntu.19.10:
  extends:
    - .fdo.container-build@ubuntu
    - .ubuntu.19.10
  stage: prep
  variables:
    FDO_DISTRIBUTION_PACKAGES: "curl wget gcc"

myjob.ubuntu.19.10:
  extends:
    - .fdo.distribution-image@ubuntu
    - .ubuntu.19.10
  stage: test
  script:
    - meson builddir && ninja -C builddir

Aside from the templating a new thing here is the e.g. .fedora.30 template what we extend from. This is an easy way to avoid having to set things like the distribution version and the tag multiple times. And a few things of note: the tag is job-specific (not distribution-specific). So you could have two Fedora 30 images with two different tags. This is also just an example I typed out, a real-world .gitlab-ci.yml will look more complex and different. So only rely on the above to get an idea of what's possible.

A word for non-gitlab.freedesktop.org users: You can use the remote: include directive to use the templates from elsewhere. ci-fairy isn't tied to freedesktop.org either but you'll have to provide more flags to get what you want instead of relying on the default behaviours.

The documentation for CI Templates has more, go and peruse my pretties.

[1] For months the CI was basically just a build test because I couldn't run the test suite in a container
[2] Updating isn't always required but sooner or later you run into a dependency issue if you don't