It's templates all the way down - part 2

In Part 1 I've shown you how to create your own distribution image using the freedesktop.org CI templates. In Part 2, we'll go a bit further than that by truly embracing nested images.

Our assumption here is that we have two projects (or jobs), with the second one relying heavily on the first one. For example, the base project and a plugin, or a base project and its language bindings. What we'll get out of this blog post is a setup where we have

• a base image in the base project
• an image extending that base image in a different project
• automatic rebuilds of that extended image when the base image changes

And none of your contributors have to care about this. It's all handled automatically and filing a MR against a project will build against the right image. So let's get started.

Our base project has CI that pushes an image to its registry. The .gitlab-ci.yml contains something like this:

.fedora32:
  variables:
    FDO_DISTRIBUTION_VERSION: '32'
    FDO_DISTRIBUTION_TAG: 'base.0'

build-img:
  extends:
    - .fedora32
    - .fdo.container-build@fedora
  variables:
    FDO_DISTRIBUTION_PACKAGES: "curl wget"

This will build a fedora/32:base.0 image in the project's container registry. That image is built once and then re-used by any job extending .fdo.distribution-image@fedora. So far, so Part 1.

Now, the second project needs to test things on top of this base image, for example language bindings for rust. You want to use the same image that the base project uses (and has successfully completed its CI on) but you need some extra packages or setup. This is where the FDO_BASE_IMAGE comes in. In our dependent project, we have this:

.fedora32:
  variables:
    FDO_DISTRIBUTION_VERSION: '32'
    FDO_DISTRIBUTION_TAG: 'rust.0'

build-rust-image:
  extends:
    - .fedora32
    - .fdo.container-build@fedora
  variables:
    FDO_BASE_IMAGE: "registry.freedesktop.org/baseproject/name/fedora/32:base.0"
    # extra packages we want to install and things we need to set up
    FDO_DISTRIBUTION_PACKAGES: "rust cargo"
    FDO_DISTRIBUTION_EXEC: "bash -x some-setup-script.sh"

test-rust:
  extends:
    - .fedora32
    - .fdo.distribution-image@fedora
  script:
   - cargo build myproject-bindings

And voila, you now have two images: the base image with curl and wget in the base project and an extra image with rust and cargo in the dependent project. And all that is required is to reference the FDO_BASE_IMAGE, everything else is the same. Note how the FDO_BASE_IMAGE is a full path in this example since we assume it's in a different project. For dependent images within the same project, you can just use the image path without the host.

The dependency only matters while the image is built, after that the dependent image is just another standalone image. So even if the base project removes the base image, you still have yours to test on.

But eventually you will need to change the base image and you want the dependent image to update as well. The best solution here is to have a CI job as part of the base repo that pokes the dependent repo's CI whenever the base image updates. The CI templates add the pipeline id as label to an image when it is built. In your base project, you can thus have a job like this:

poke-dependents:
   extends:
     - .fedora32
     - .fdo.distribution-image@fedora
   image: something-with-skopeo-and-jq
   script:
     # FDO_DISTRIBUTION_IMAGE still has indirections
     - DISTRO_IMAGE=$(eval echo ${FDO_DISTRIBUTION_IMAGE})
     # retrieve info from the registry and extract the pipeline id
     - JSON_IMAGE=$(skopeo inspect docker://$DISTRO_IMAGE)
     - IMAGE_PIPELINE_ID=$(echo $JSON_IMAGE | jq -r '.Labels["fdo.pipeline_id"]')
     - |
       if [[ x"$IMAGE_PIPELINE_ID" == x"$CI_PIPELINE_ID" ]]; then
          curl -X POST 
               -F "token=$AUTH_TOKEN_VALUE"
               -F "ref=master" 
               -F "variables[SOMEVARIABLE]=somevalue"
               https://gitlab.freedesktop.org/api/v4/projects/dependent${SLASH}project/trigger/pipeline
       fi
   variables:
     SLASH: "%2F"

Let's dissect this: First, we use the .fdo.distribution-image@fedora template to get access to FDO_DISTRIBUTION_IMAGE. We don't need to use the actual image though, anything with skopeo and jq will do. Then we fetch the pipeline id label from the image and compare it to the current pipeline ID. If it is the same, our image was rebuilt as part of the pipeline and we poke the other project's pipeline with a SOMEVARIABLE set to somevalue. The auth token is a standard GitLab token you need to create to allow triggering the pipeline in the dependent project.

In that dependent project you can have a job like this:

rebuild-extra-image:
  extends: build-extra-image
  rules:
    - if: '$SOMEVARIABLE == "somevalue"'
  variables:
    FDO_FORCE_REBUILD: 1

This job is only triggered where the variable is set and it will force a rebuild of the container image. If you want custom rebuilds of images, set the variables accordingly.

So, as promised above, we now have a base image and a separate image building on that, together with auto-rebuild hooks. The gstreamer-plugins-rs project uses this approach. The base image is built by gstreamer-rs during its CI run which then pokes gstreamer-plugins-rs to rebuild selected dependent images.

The above is most efficient when the base project knows of the dependent projects. Where this is not the case, the dependent project will need a scheduled pipeline to poll the base project and extract the image IDs from that, possibly using creation dates and whatnot. We'll figure that out when we have a use-case for it.