User-specific XKB configuration - part 2

This is the continuation from this post.

Several moons have bypassed us [1] in the time since the first post, and Things Have Happened! If you recall (and of course you did because you just re-read the article I so conveniently linked above), libxkbcommon supports an include directive for the rules files and it will load a rules file from $XDG_CONFIG_HOME/xkb/rules/ which is the framework for custom per-user keyboard layouts. Alas, those files are just sitting there, useful but undiscoverable.

To give you a very approximate analogy, the KcCGST format I described last time are the ingredients to a meal (pasta, mince, tomato). The rules file is the machine-readable instruction set to assemble your meal but it relies a lot on wildcards. Feed it "spaghetti, variant bolognese" and the actual keymap ends up being the various components put together: "pasta(spaghetti)+sauce(tomato)+mince". But for this to work you need to know that spag bol is available in the first place, i.e you need the menu. This applies to keyboard layouts too - the keyboard configuration panel needs to present a list so the users can clickedy click-click on whatever layout they think is best for them.

This menu of possible layouts is provided by the xkeyboard-config project but for historical reasons [2], it is stored as an XML file named after the ruleset: usually /usr/share/X11/xkb/rules/evdev.xml [3]. Configuration utilities parse that file directly which is a bit of an issue when your actual keymap compiler starts supporting other include paths. Your fancy new layout won't show up because everything insists on loading the system layout menu only. This bit is part 2, i.e. this post here.

If there's one thing that the world doesn't have enough of yet, it's low-level C libraries. So I hereby present to you: libxkbregistry. This library has now been merged into the libxkbcommon repository and provides a simple C interface to list all available models, layouts and options for a given ruleset. It sits in the same repository as libxkbcommon - long term this will allow us to better synchronise any changes to XKB handling or data formats as we can guarantee that the behaviour of both components is the same.

Speaking of data formats, we haven't actually changed any of those which means they're about as easy to understand as your local COVID19 restrictions. In the previous post I outlined the example for the KcCGST and rules file, what you need now with libxkbregistry is an XKB-compatible XML file named after your ruleset. Something like this:

$ cat $HOME/.config/xkb/rules/evdev.xml
<?xml version="1.0" encoding="UTF-8"?>                                           
<!DOCTYPE xkbConfigRegistry SYSTEM "xkb.dtd">                                    
<xkbConfigRegistry version="1.1">                                                
  <layoutList>                                                                   
    <layout>                                                                     
      <configItem>                                                               
        <name>us</name>                                                      
      </configItem>                                                              
      <variantList>                                                              
        <variant>                                                                
          <configItem>                                                           
            <name>banana</name>                                                  
            <shortDescription>banana</shortDescription>                              
            <description>US (Banana)</description>                           
          </variant>                                                             
      </variantList>                                                             
  </layoutList>                                                                  
  <optionList>                                                                   
    <group allowMultipleSelection="true">                                        
      <configItem>                                                               
        <name>custom</name>                                                      
        <description>Custom options</description>                                
      </configItem>                                                              
      <option>                                                                   
      <configItem>                                                               
        <name>custom:foo</name>                                                  
        <description>Map Tilde to nothing</description>                          
      </configItem>                                                              
      </option>                                                                  
      <option>                                                                   
      <configItem>                                                               
        <name>custom:baz</name>                                                  
        <description>Map Z to K</description>                                    
      </configItem>                                                              
      </option>                                                                  
    </group>                                                                     
  </optionList>                                                                  
</xkbConfigRegistry>   

This looks more complicated than it is: we have models (not shown here), layouts which can have multiple variants and options which are grouped together in option group (to make options mutually exclusive). libxkbregistry will merge this with the system layouts in what is supposed to be the most obvious merge algorithm. The simple summary of that is that you can add to existing system layouts but you can't modify those - the above example will add a "banana" variant to the US keyboard layout without modifying "us" itself or any of its other variants. The second part adds two new options based on my previous post.

Now, all that is needed is to change every user of evdev.xml to use libxkbregistry. The gnome-desktop merge request is here for a start.

[1] technically something that goes around something else doesn't bypass it but the earth is flat, the moon is made of cheese, facts don't matter anymore and stop being so pedantic about things already!
[2] it's amazing what you can handwave away with "for historical reasons". Life would be better if there was less history to choose reasons from.
[3] there's also evdev.extras.xml for very niche layouts which is a separate file for historical reasons [2], despite there being a "popularity" XML attribute

It's templates all the way down - part 2

In Part 1 I've shown you how to create your own distribution image using the freedesktop.org CI templates. In Part 2, we'll go a bit further than that by truly embracing nested images.

Our assumption here is that we have two projects (or jobs), with the second one relying heavily on the first one. For example, the base project and a plugin, or a base project and its language bindings. What we'll get out of this blog post is a setup where we have

• a base image in the base project
• an image extending that base image in a different project
• automatic rebuilds of that extended image when the base image changes

And none of your contributors have to care about this. It's all handled automatically and filing a MR against a project will build against the right image. So let's get started.

Our base project has CI that pushes an image to its registry. The .gitlab-ci.yml contains something like this:

.fedora32:
  variables:
    FDO_DISTRIBUTION_VERSION: '32'
    FDO_DISTRIBUTION_TAG: 'base.0'

build-img:
  extends:
    - .fedora32
    - .fdo.container-build@fedora
  variables:
    FDO_DISTRIBUTION_PACKAGES: "curl wget"

This will build a fedora/32:base.0 image in the project's container registry. That image is built once and then re-used by any job extending .fdo.distribution-image@fedora. So far, so Part 1.

Now, the second project needs to test things on top of this base image, for example language bindings for rust. You want to use the same image that the base project uses (and has successfully completed its CI on) but you need some extra packages or setup. This is where the FDO_BASE_IMAGE comes in. In our dependent project, we have this:

.fedora32:
  variables:
    FDO_DISTRIBUTION_VERSION: '32'
    FDO_DISTRIBUTION_TAG: 'rust.0'

build-rust-image:
  extends:
    - .fedora32
    - .fdo.container-build@fedora
  variables:
    FDO_BASE_IMAGE: "registry.freedesktop.org/baseproject/name/fedora/32:base.0"
    # extra packages we want to install and things we need to set up
    FDO_DISTRIBUTION_PACKAGES: "rust cargo"
    FDO_DISTRIBUTION_EXEC: "bash -x some-setup-script.sh"

test-rust:
  extends:
    - .fedora32
    - .fdo.distribution-image@fedora
  script:
   - cargo build myproject-bindings

And voila, you now have two images: the base image with curl and wget in the base project and an extra image with rust and cargo in the dependent project. And all that is required is to reference the FDO_BASE_IMAGE, everything else is the same. Note how the FDO_BASE_IMAGE is a full path in this example since we assume it's in a different project. For dependent images within the same project, you can just use the image path without the host.

The dependency only matters while the image is built, after that the dependent image is just another standalone image. So even if the base project removes the base image, you still have yours to test on.

But eventually you will need to change the base image and you want the dependent image to update as well. The best solution here is to have a CI job as part of the base repo that pokes the dependent repo's CI whenever the base image updates. The CI templates add the pipeline id as label to an image when it is built. In your base project, you can thus have a job like this:

poke-dependents:
   extends:
     - .fedora32
     - .fdo.distribution-image@fedora
   image: something-with-skopeo-and-jq
   script:
     # FDO_DISTRIBUTION_IMAGE still has indirections
     - DISTRO_IMAGE=$(eval echo ${FDO_DISTRIBUTION_IMAGE})
     # retrieve info from the registry and extract the pipeline id
     - JSON_IMAGE=$(skopeo inspect docker://$DISTRO_IMAGE)
     - IMAGE_PIPELINE_ID=$(echo $JSON_IMAGE | jq -r '.Labels["fdo.pipeline_id"]')
     - |
       if [[ x"$IMAGE_PIPELINE_ID" == x"$CI_PIPELINE_ID" ]]; then
          curl -X POST 
               -F "token=$AUTH_TOKEN_VALUE"
               -F "ref=master" 
               -F "variables[SOMEVARIABLE]=somevalue"
               https://gitlab.freedesktop.org/api/v4/projects/dependent${SLASH}project/trigger/pipeline
       fi
   variables:
     SLASH: "%2F"

Let's dissect this: First, we use the .fdo.distribution-image@fedora template to get access to FDO_DISTRIBUTION_IMAGE. We don't need to use the actual image though, anything with skopeo and jq will do. Then we fetch the pipeline id label from the image and compare it to the current pipeline ID. If it is the same, our image was rebuilt as part of the pipeline and we poke the other project's pipeline with a SOMEVARIABLE set to somevalue. The auth token is a standard GitLab token you need to create to allow triggering the pipeline in the dependent project.

In that dependent project you can have a job like this:

rebuild-extra-image:
  extends: build-extra-image
  rules:
    - if: '$SOMEVARIABLE == "somevalue"'
  variables:
    FDO_FORCE_REBUILD: 1

This job is only triggered where the variable is set and it will force a rebuild of the container image. If you want custom rebuilds of images, set the variables accordingly.

So, as promised above, we now have a base image and a separate image building on that, together with auto-rebuild hooks. The gstreamer-plugins-rs project uses this approach. The base image is built by gstreamer-rs during its CI run which then pokes gstreamer-plugins-rs to rebuild selected dependent images.

The above is most efficient when the base project knows of the dependent projects. Where this is not the case, the dependent project will need a scheduled pipeline to poll the base project and extract the image IDs from that, possibly using creation dates and whatnot. We'll figure that out when we have a use-case for it.

xisxwayland checks for Xwayland ... or not

One of the more common issues we encounter debugging things is that users don't always know whether they're running on a Wayland or X11 session. Which I guess is a good advertisement for how far some of the compositors have come. The question "are you running on Xorg or Wayland" thus comes up a lot and suggestions previously included things like "run xeyes", "grep xinput list", "check xrandr" and so on and so forth. None of those are particularly scriptable, so there's a new tool around now: xisxwayland.

Run without arguments it simply exits with exit code 0 if the X server is Xwayland, or 1 otherwise. Which means use can use it like this:

$ cat my-xorg-only-script.sh
#!/bin/bash

if xisxwayland; then
   echo "This is an Xwayland server!";
   exit 1
fi

...

Or, in the case where you have a human user (gasp!), you can ask them to run:

$ xisxwayland --verbose
Xwayland: YES

And even non-technical users should be able to interpret that.

Note that the script checks for Xwayland (hence the name) via the $DISPLAY environment variable, just like any X application. It does not check whether there's a Wayland compositor running but for most use-cases this doesn't matter anyway. For those where it matters you get to write your own script. Congratulations, I guess.

Wayland doesn't support $BLAH

Gather round children, it's analogy time! First, some definitions:

• Wayland is a protocol to define the communication between a display server (the "compositor") and a client, i.e. an application though the actual communication is usually done by a toolkit like GTK or Qt.
• A Wayland Compositor is an implementation of a display server that (usually but not necessary) handles things like displaying stuff on screen and handling your input devices, among many other things. Common examples for Wayland Compositors are GNOME's mutter, KDE's KWin, weston, sway, etc.[1]

And now for the definitions we need for our analogy:

• HTTP is a protocol to define the communication between a web server and a client (usually called the "browser")
• A Web Browser is an implementation that (sometimes but not usually) handles things like displaying web sites correctly, among many other things. Common examples for Web Browsers are Firefox, Chrome, Edge, Safari, etc. [2]

And now for the analogy:

The following complaints are technically correct but otherwise rather pointless to make:

• HTTP doesn't support CSS
• HTTP doesn't support adblocking
• HTTP doesn't render this or that website correctly

And much in the same style, the following complaints are technically correct but otherwise rather pointless to make:

• Wayland doesn't support keyboard shortcuts
• Wayland doesn't support screen sharing
• Wayland doesn't render this or that application correctly

In most cases you may encounter (online or on your setup), saying "Wayland doesn't support $BLAH" is like saying "HTTP doesn't support $BLAH". The problem isn't in with Wayland itself, it's a missing piece or bug in the compositor you're using.

Likewise, saying "I don't like Wayland" is like saying "I don't like HTTP".The vast majority of users will have negative feelings towards the browser, not the transport protocol.

[1] Because they're implementations of a display server they can speak multiple protocols and some compositors can also be X11 window managers, much in the same way as you can switch between English and your native language(s).
[2] Because they're implementations of a web browser they can speak multiple protocols and some browsers can also be FTP file managers, much in the same way as... you get the point

High resolution wheel scrolling in the desktop stack

This is a follow up from the kernel support for high-resolution wheel scrolling which you totally forgot about because it's already more then a year in the past and seriously, who has the attention span these days to remember this. Anyway, I finally found time and motivation to pick this up again and I started lining up the pieces like cans, for it only to be shot down by the commentary of strangers on the internet. The Wayland merge request lists the various pieces (libinput, wayland, weston, mutter, gtk and Xwayland) but for the impatient there's also an Fedora 32 COPR. For all you weirdos inexplicably not running the latest Fedora, well, you'll have to compile this yourself, just like I did.

Let's recap: in v5.0 the kernel added new axes REL_WHEEL_HI_RES and REL_HWHEEL_HI_RES for all devices. On devices that actually support high-resolution wheel scrolling (Logitech and Microsoft mice, primarily) you'll get multiple hires events before the now-legacy REL_WHEEL events. On all other devices those two are in sync.

Integrating this into the userspace stack was a bit of a mess at first, but I think the solution is good enough, even if it has a rather verbose explanation on how to handle it. The actual patches to integrate ended up being relatively simple. So let's see why it's a bit weird:

When Wayland started, back in WhoahReallyThatLongAgo, scrolling was specified as the wl_pointer.axis event with a value in pixels. This works fine for touchpads, not so much for wheels. The early versions of Weston decreed that one wheel click was 10 pixels [1] and, perhaps surprisingly, the world kept on turning. When libinput was forked from Weston an early change was that wheel events would have two values - degrees of movement and click count ("discrete steps"). The wayland protocol was expanded to include the discrete steps as wl_pointer.axis_discrete as well. Then backwards compatibility reared its ugly head and Mutter, Weston, GTK all basically said: one discrete step equals 10 pixels so we multiply the discrete value by 10 and, perhaps surprisingly, the world kept on turning.

This worked out well enough for a few years but with high resolution wheels we ran into a problem. Discrete steps are integers, so we can't send partial values. And the protocol is defined in a way that any tweaking of the behaviour would result in broken clients which, perhaps surprisingly, is a Bad Thing. This lead to the current proposal of separate events. LIBINPUT_EVENT_POINTER_AXIS_WHEEL and for Wayland the wl_pointer.axis_v120 event, linked to above. These events are (like the kernel events) a parallel event stream to the previous events and effectively replace the LIBINPUT_EVENT_POINTER_AXIS and Wayland wl_pointer.axis/axis_discrete pair for wheel events (not so for touchpad or button scrolling though).

The compositor side of things is relatively simple: take the events from libinput and pass the hires ones as v120 events and the lowres ones as v120 events with a value of zero. The client side takes the v120 events and uses them over wl_pointer.axis/axis_discrete unless one is zero in which case you can discard all axis events in that wl_pointer.frame. Since most client implementation already have the support for smooth scrolling (because, well, touchpads do exist) it's relatively simple to integrate - the new events just feed into the smooth scrolling code. And since you already have to do wheel emulation for that (because, well, old clients exist) wheel emulation is handled easily too.

All that to provide buttery smooth [2] wheel scrolling. Or not, if your hardware doesn't support it. In which case, well, live with the warm fuzzy feeling that someone else has a better user experience now. Or soon, anyway.

[1] with, I suspect, the scientific measurement of "yeah, that seems about alright"
[2] like butter out of a fridge, so still chunky but at least less so than before

It's templates all the way down

Benjamin Tissoires and I have been busy anthophila and working on the freedesktop CI templates. This post is primarily of interest if you're working on GitLab, specifically if your repo is hosted on gitlab.freedesktop.org. If either of those applies, prepare to be distracted from the current pandemic, otherwise maybe just prepare to be entertained. I'll do my best to be less miserable than the news.

We all know that CI/CD really helps with finding bugs early. If you don't know that yet, insert a jedi handwave before the previous sentence and now you do. GitLab is the git forge now used by freedesktop.org and it comes with a built-in CI system. I'm leaving out the difficult bits such as actually setting the thing up because this is obviously all handled by Heinzelmännchen and just readily available, hooray. I'm also going to assume that you roughly know how to write GitLab CI jobs or, failing that, at least know how to read YAML without screaming. So for this post, we start with the basic problem that your .gitlab-ci.yml is getting unwieldy, repetitive or generally just kinda sucks to maintain. Which is roughly where libinput and libevdev were a while back which caused Benjamin to start the ci-templates.

Now, what do we want? (other than a COVID-19 cure) Reproducible tests, possibly on different distributions, with the same base system across tests. For my repos the goal was basically "test on the common distributions to catch certain bugs early". [1] For Mesa, the requirement is closer to "have a fixed set of images that 'never' change so tests are reproducible". Both goals have much in common.

Your first venture into CI will look like this:

myjob:
  image: fedora:31
  before_script:
    - dnf update -y
    - dnf install -y onepackage twopackage threepackage floor
  script:
    - meson builddir && ninja -C builddir test

So, in short: take a Fedora 31 docker image, update it [2], install the required packages and then run the actual test part - meson and ninja. Easy.

This works fine but it takes approximately forever because dnf update is slow and you're potentially pulling down gigs of packages on every test run. Which is fun, but less so when you have 10 different jobs and they all do that. So let's call this step 1 and pretend we're more advanced than that. Step 2 is where you start building an image you re-use, steps 3 to N are the bits where you learn more than you want to know about docker, podman, skopeo and how many typos you can put into a YAML file. So, ad break, and we jump right to the part where enlightenment is just around the corner or wherever enlightenment lurks these days.

Using the CI Templates

Here's the .gitlab-ci.yml to build a Fedora 31 images with ci-templates and run the test on that image:

include:
  - project: 'freedesktop/ci-templates'
    ref: 123456deadbeef
    file: '/templates/fedora.yml'

variables:
   # project name of the upstream repo
   FDO_UPSTREAM_REPO: someproject/name

stages:
  - prep
  - test

myimage:
  extends: .fdo.container-build@fedora
  stage: prep
  variables:
    FDO_DISTRIBUTION_VERSION: '31'
    FDO_DISTRIBUTION_PACKAGES: 'onepackage twopackage threepackage floor'
    FDO_DISTRIBUTION_TAG: '2020-03-20.0'

myjob:
  extends: .fdo.distribution-image@fedora
  stage: test
  script:
    - meson builddir && ninja -C builddir test
  variables:
    FDO_DISTRIBUTION_VERSION: '31'
    FDO_DISTRIBUTION_TAG: '2020-03-20.0'

Now, you guessed correctly that the .fdo and FDO_ prefixes are used by the templates. There is a bunch of stuff hidden here. Basically, this will:

• check if the image exists in your personal project's registry and use that, but if not
• check if the image exists in the given upstream project's registry and use that, but if not
• create a Fedora 31 image with the given packages installed and pushes it with the tag to the registry
• use that image (whether newly created or pre-existing) and run the tests on it

There are a few more details too, but that's roughly the summary of it. For existing tags, the the myimage job effectively becomes a noop and the myjob job will re-use the image. The image will be in your registry so you can podman run it locally to reproduce a bug.

To build a new image, simply change the tag. Either because you want newer packages or you need extra (or less packages). And the nice thing here: you will build a new image as part of your merge request and run the CI against that new image. But upstream and every other MR will keep using the old image - right up until your MR is merged at which point every (future) MR will use that new updated image.

Want to build a Debian Stretch image? Replace Fedora and 31 with debian and stretch. Same for Ubuntu, Centos, Alpine and Arch though for those two you don't need a version number.

Templating the templates

"But, but, Peter, I want to test on eleventy different distribution like you do" I hear you say. Well, fear not, for this is where the ci-fairy comes in. How about we *gasp* generate the .gitlab-ci.yml file from a base configuration? That can't possibly be a bad idea, so let's do that! First, we save our configuration into the .gitlab-ci/config.yml:

distributions:
  - name: fedora
    tag: 12345
    version: 30
  - name: ubuntu
    tag: abcde
    version: '19.10'
  # and so on, and so forth

packages:
  - curl
  - wget
  - gcc

There is no specific requirement on the structure of the config file, ci-fairy simply loads it and passes it to Jinja2. Your template could thus look like this .gitlab-ci/ci.template file:

include:
{% for d in distributions %}
   - project: 'freedesktop/ci-templates'
     ref: 123456deadbeef
     file: '/templates/{{d.name}}.yml'
{% endfor %}

stages:
  - prep
  - test

{% for d in distributions %}

.{{d.name}}.{{d.version}}:
  variables:
    FDO_DISTRIBUTION_VERSION: '{{d.version}}'
    FDO_DISTRIBUTION_TAG: '{{d.tag}}'

myimage.{{d.name}}.{{d.version}}:
  extends:
    - .fdo.container-build@{{d.name}}
    - .{{d.name}}.{{d.version}}
  stage: prep
  variables:
    FDO_DISTRIBUTION_PACKAGES: "{{' '.join(packages)}}"

myjob.{{d.name}}.{{d.version}}:
  extends:
    - .fdo.distribution-image@{{d.name}}
    - .{{d.name}}.{{d.version}}
  stage: test
  script:
    - meson builddir && ninja -C builddir
{% endfor %}

And to locally generate our .gitlab-ci.yml, all we need to do is

$ pip3 install git+http://gitlab.freedesktop.org/freedesktop/ci-templates
$ cd path/to/project
$ ci-fairy generate-template
$ ci-fairy lint  # checks the resulting YAML for syntax errors
$ git commit .gitlab-ci.yml

And, for reference, the file we generated here looks like this:

include:
   - project: 'freedesktop/ci-templates'
     ref: 123456deadbeef
     file: '/templates/fedora.yml'
   - project: 'freedesktop/ci-templates'
     ref: 123456deadbeef
     file: '/templates/ubuntu.yml'

stages:
  - prep
  - test

.fedora.30:
  variables:
    FDO_DISTRIBUTION_VERSION: '30'
    FDO_DISTRIBUTION_TAG: '12345'

myimage.fedora.30:
  extends:
    - .fdo.container-build@fedora
    - .fedora.30
  stage: prep
  variables:
    FDO_DISTRIBUTION_PACKAGES: "curl wget gcc"

myjob.fedora.30:
  extends:
    - .fdo.distribution-image@fedora
    - .fedora.30
  stage: test
  script:
    - meson builddir && ninja -C builddir

.ubuntu.19.10:
  variables:
    FDO_DISTRIBUTION_VERSION: '19.10'
    FDO_DISTRIBUTION_TAG: 'abcde'

myimage.ubuntu.19.10:
  extends:
    - .fdo.container-build@ubuntu
    - .ubuntu.19.10
  stage: prep
  variables:
    FDO_DISTRIBUTION_PACKAGES: "curl wget gcc"

myjob.ubuntu.19.10:
  extends:
    - .fdo.distribution-image@ubuntu
    - .ubuntu.19.10
  stage: test
  script:
    - meson builddir && ninja -C builddir

Aside from the templating a new thing here is the e.g. .fedora.30 template what we extend from. This is an easy way to avoid having to set things like the distribution version and the tag multiple times. And a few things of note: the tag is job-specific (not distribution-specific). So you could have two Fedora 30 images with two different tags. This is also just an example I typed out, a real-world .gitlab-ci.yml will look more complex and different. So only rely on the above to get an idea of what's possible.

A word for non-gitlab.freedesktop.org users: You can use the remote: include directive to use the templates from elsewhere. ci-fairy isn't tied to freedesktop.org either but you'll have to provide more flags to get what you want instead of relying on the default behaviours.

The documentation for CI Templates has more, go and peruse my pretties.

[1] For months the CI was basically just a build test because I couldn't run the test suite in a container
[2] Updating isn't always required but sooner or later you run into a dependency issue if you don't

A tale of missing touches

libinput 1.15.1 had a new feature: it matched the expected touch count with the one actually seen as opposed to the one advertised by the kernel. That is good news for ALPS devices whose kernel driver lies about their capabilities because these days who doesn't. However, in some cases that feature had the side-effect of reducing the touch count to zero - meaning libinput would ignore any touch. This caused a slight UX degradation.

After a bit of debugging and/or cursing, the issue was identified as a libevdev issue, specifically - the way libevdev replays events after a SYN_DROPPED event. And after several days of fixing things, adding stuff to the CI and adding meson support for libevdev so the CI can actually run a few useful things, it's time for a blog post to brain-dump and possibly entertain the occasional reader such as you are. Congratulations, I guess.

The Linux kernel's evdev protocol is a serial protocol where all events have a type, a code and a value. Events are grouped by EV_SYN.SYN_REPORT events, so the event type is EV_SYN (0), the event code is SYN_REPORT (also 0). The value is usually (but not always), you guessed it, zero. A SYN_REPORT signals that the current event sequence (also called a "frame") is to be interpreted as one hardware event [0]. In the simplest case, two hardware events from a mouse could look like this:

EV_REL   REL_X        1
EV_SYN   SYN_REPORT   0
EV_REL   REL_X        1
EV_REL   REL_Y        1
EV_SYN   SYN_REPORT   0

While we have five evdev events here, those represent one hardware event with an x movement of 1 and a second hardware event with a diagonal movement by 1/1. Glorious, we all understand evdev now (if not, read this and immediately afterwards this, although that second post will be rather reinforced by this post).

Life as software developer would be quite trivial but our universe hates us and we need an extra event code called SYN_DROPPED. This event is used by the kernel when events from the device come in faster than you're reading them. This shouldn't happen given that most input devices scan out at the casual rate of every 7ms or slower and we're not exactly running on carrier pigeons here. But your compositor has been a busy bee rendering all these browser windows containing kitten videos and thus completely neglected to check whether you've moved the finger on the touchpad recently. So the kernel sort-of clears the current event buffer and positions a shiny steaming SYN_DROPPED in there to notify the compositor of its wrongdoing. [1]

Now, we could assume that every evdev client (libinput, every Xorg driver, ...) knows how to handle SYN_DROPPED events correctly but we're self-aware enough that we don't. So SYN_DROPPED handling is wrapped via libevdev, in a way that lets the clients use almost exactly the same processing paths they use for normal events. libevdev gives you a notification that a SYN_DROPPED occured, then you fetch the events one-by-one until libevdev tells you have the complete current state of the device, and back to kittens you go. In pseudo-code, your input stack's event loop works like this:

while (user_wants_kittens):
   event = libevdev_get_event()

   if event is a SYN_DROPPED:
     while (libevdev_is_still_synchronizing):
        event = libevdev_get_event()
        process_event(event)
   else:
        process_event(event)

Now, this works great for keys where you get the required events to release or press new keys. This works great for relative axes because meh, who cares [2]. This works great for absolute axes because you just get the current state of the device and done. This works great for touch because, no wait, that bit is awful.

You see, the multi-touch protocol is ... special. It uses the absolute axes, but it also multiplexes over those axes via the slot protocol. A normal two-touch event looks like this:

EV_ABS ABS_MT_SLOT         0
EV_ABS ABS_MT_POSITION_X   123
EV_ABS ABS_MT_SLOT         1
EV_ABS ABS_MT_POSITION_X   456
EV_ABS ABS_MT_POSITION_Y   789
EV_ABS ABS_X               123
EV_SYN SYN_REPORT          0

The first two evdev events are slot 0 (first touch [3]), the second two evdev events are slot 1 (second touch [3]). Both touches update their X position but the second touch also updates its Y position. But for single-touch emulation we also get the normal absolute axis event [3]. Which is equivalent to the first touch [3] and can be ignored if you're handling the MT axes [3] (I'm getting a lot of mileage out of that footnote). And because things aren't confusing enough: events within an evdev frame are position-independent except the ABS_MT axes which need to be processed in sequence. So that ABS_X events could be anywhere within that frame, but the ABS_MT axes need to be grouped by slot.

About that single-touch emulation... We also have a single-touch multi-touch protocol via EV_KEY. For devices that can only track N fingers but can detect N+M fingers, we have a set of BTN_TOOL defines. Two fingers down sets BTN_TOOL_DOUBLETAP, three fingers down sets BTN_TOOL_TRIPLETAP, etc. Those are just a bitfield though, so no position data is available. And it tops out at BTN_TOOL_QUINTTAP but then again, that's a good maximum backed by a lot of statistical samples from users hands. On many devices, we have to combine that single-touch MT protocol with the real MT protocol. Synaptics touchpads on PS/2 only support 2 finger positions but detect up 5 touches otherwise [4]. And remember the ALPS devices? They say they have 4 slots but may only send data for two or three, so we have to detect this at runtime and switch to the BTN_TOOL bits for some touches.

So anyway, now that we unfortunately all understand the MT protocol(s), let's look at that libevdev bug. libevdev checks the slot states after SYN_DROPPED to detect whether any touch has stopped or started during SYN_DROPPED. It also detects whether a touch has changed, i.e. the user lifted the finger(s) and put the finger(s) down again while SYN_DROPPED was happening. For those touches it generates the events to stop the original touch, then events to start the new touch. This needs to be done over two event frames, i.e. with a SYN_REPORT in between [5]. But the implementation ended up splitting those changes - any touch that changed was terminated in the first event frame, any touch that outright stopped was terminated in the second event frame. That in itself wasn't the problem yet, the problem was that libevdev didn't emulate the single-touch multi-touch protocol with those emulated frames. So we ended up with event frames where slots would terminate but the single-touch protocol didn't update until a frame later.

This doesn't matter for most users. Both protocols were still correct-enough in their own bubble, only once you start mixing protocols was where it all started getting wonky. libinput does this because it has to, too many devices out there only track two fingers. So if you want three-finger tapping and pinch gestures, you need to handle both protocols. Despite this we didn't notice until we added the quirk for ALPS devices. Because now libinput sometimes noticed that after a SYN_DROPPED there were no fingers on the touchpad (because they all stopped/changed) but the BTN_TOOL bits were still on so clearly we have a touchpad that cannot track all fingers it detects - in this case zero. [6]

So to recap: libinput's auto-adjustment of the touch count for buggy touchpad devices failed thanks to libevdev's buggy workaround of the device sync. The device sync we need because we can't rely on userspace handling touches correctly across SYN_DROPPED. An event which only gets triggered because the compositor is too buggy to read input events in time. I don't know how to describe it exactly, but what I can see all the way down are definitely not turtles.

And the sad thing about it: if we didn't try to correct for the firmware and accepted that gestures are just broken on ALPS devices because the kernel driver is lying to us, none of the above would have mattered. Likewise, the old xorg synaptics driver won't be affected by this because it doesn't handle multitouch properly anyway, so it doesn't need to care about these discrepancies. Or, in other words and much like real life: the better you try to be, the worse it all gets.

And as the take-home lesson: do upgrade to libinput 1.15.2 and do upgrade to libevdev 1.9.0 when it's out. Your kittens won't care but at least that way it won't make me feel like I've done all this work in vain.

[0] Unless the SYN_REPORT value is nonzero but let's not confuse everyone more than necessary
[1] A SYN_DROPPED is per userspace client, so a debugging tool reading from the same event node may not see that event unless it too is busy with feline renderings.
[2] yes, you'll get pointer jumps because event data is missing but since you've been staring at those bloody cats anyway, you probably didn't even notice
[3] usually, but not always
[4] on those devices, identifying a 3-finger pinch gesture only works if you put the fingers down in the correct order
[5] historical reasons: in theory a touch could change directly but most userspace can't handle it and it's too much effort to add now
[6] libinput 1.15.2 leaves you with 1 finger in that case and that's good enough until libevdev is released