Thursday, February 20, 2020

A tale of missing touches

libinput 1.15.1 had a new feature: it matched the expected touch count with the one actually seen as opposed to the one advertised by the kernel. That is good news for ALPS devices whose kernel driver lies about their capabilities because these days who doesn't. However, in some cases that feature had the side-effect of reducing the touch count to zero - meaning libinput would ignore any touch. This caused a slight UX degradation.

After a bit of debugging and/or cursing, the issue was identified as a libevdev issue, specifically - the way libevdev replays events after a SYN_DROPPED event. And after several days of fixing things, adding stuff to the CI and adding meson support for libevdev so the CI can actually run a few useful things, it's time for a blog post to brain-dump and possibly entertain the occasional reader such as you are. Congratulations, I guess.

The Linux kernel's evdev protocol is a serial protocol where all events have a type, a code and a value. Events are grouped by EV_SYN.SYN_REPORT events, so the event type is EV_SYN (0), the event code is SYN_REPORT (also 0). The value is usually (but not always), you guessed it, zero. A SYN_REPORT signals that the current event sequence (also called a "frame") is to be interpreted as one hardware event [0]. In the simplest case, two hardware events from a mouse could look like this:

EV_REL   REL_X        1
EV_SYN   SYN_REPORT   0
EV_REL   REL_X        1
EV_REL   REL_Y        1
EV_SYN   SYN_REPORT   0
While we have five evdev events here, those represent one hardware event with an x movement of 1 and a second hardware event with a diagonal movement by 1/1. Glorious, we all understand evdev now (if not, read this and immediately afterwards this, although that second post will be rather reinforced by this post).

Life as software developer would be quite trivial but our universe hates us and we need an extra event code called SYN_DROPPED. This event is used by the kernel when events from the device come in faster than you're reading them. This shouldn't happen given that most input devices scan out at the casual rate of every 7ms or slower and we're not exactly running on carrier pigeons here. But your compositor has been a busy bee rendering all these browser windows containing kitten videos and thus completely neglected to check whether you've moved the finger on the touchpad recently. So the kernel sort-of clears the current event buffer and positions a shiny steaming SYN_DROPPED in there to notify the compositor of its wrongdoing. [1]

Now, we could assume that every evdev client (libinput, every Xorg driver, ...) knows how to handle SYN_DROPPED events correctly but we're self-aware enough that we don't. So SYN_DROPPED handling is wrapped via libevdev, in a way that lets the clients use almost exactly the same processing paths they use for normal events. libevdev gives you a notification that a SYN_DROPPED occured, then you fetch the events one-by-one until libevdev tells you have the complete current state of the device, and back to kittens you go. In pseudo-code, your input stack's event loop works like this:

while (user_wants_kittens):
   event = libevdev_get_event()

   if event is a SYN_DROPPED:
     while (libevdev_is_still_synchronizing):
        event = libevdev_get_event()
        process_event(event)
   else:
        process_event(event)
Now, this works great for keys where you get the required events to release or press new keys. This works great for relative axes because meh, who cares [2]. This works great for absolute axes because you just get the current state of the device and done. This works great for touch because, no wait, that bit is awful.

You see, the multi-touch protocol is ... special. It uses the absolute axes, but it also multiplexes over those axes via the slot protocol. A normal two-touch event looks like this:

EV_ABS ABS_MT_SLOT         0
EV_ABS ABS_MT_POSITION_X   123
EV_ABS ABS_MT_SLOT         1
EV_ABS ABS_MT_POSITION_X   456
EV_ABS ABS_MT_POSITION_Y   789
EV_ABS ABS_X               123
EV_SYN SYN_REPORT          0
The first two evdev events are slot 0 (first touch [3]), the second two evdev events are slot 1 (second touch [3]). Both touches update their X position but the second touch also updates its Y position. But for single-touch emulation we also get the normal absolute axis event [3]. Which is equivalent to the first touch [3] and can be ignored if you're handling the MT axes [3] (I'm getting a lot of mileage out of that footnote). And because things aren't confusing enough: events within an evdev frame are position-independent except the ABS_MT axes which need to be processed in sequence. So that ABS_X events could be anywhere within that frame, but the ABS_MT axes need to be grouped by slot.

About that single-touch emulation... We also have a single-touch multi-touch protocol via EV_KEY. For devices that can only track N fingers but can detect N+M fingers, we have a set of BTN_TOOL defines. Two fingers down sets BTN_TOOL_DOUBLETAP, three fingers down sets BTN_TOOL_TRIPLETAP, etc. Those are just a bitfield though, so no position data is available. And it tops out at BTN_TOOL_QUINTTAP but then again, that's a good maximum backed by a lot of statistical samples from users hands. On many devices, we have to combine that single-touch MT protocol with the real MT protocol. Synaptics touchpads on PS/2 only support 2 finger positions but detect up 5 touches otherwise [4]. And remember the ALPS devices? They say they have 4 slots but may only send data for two or three, so we have to detect this at runtime and switch to the BTN_TOOL bits for some touches.

So anyway, now that we unfortunately all understand the MT protocol(s), let's look at that libevdev bug. libevdev checks the slot states after SYN_DROPPED to detect whether any touch has stopped or started during SYN_DROPPED. It also detects whether a touch has changed, i.e. the user lifted the finger(s) and put the finger(s) down again while SYN_DROPPED was happening. For those touches it generates the events to stop the original touch, then events to start the new touch. This needs to be done over two event frames, i.e. with a SYN_REPORT in between [5]. But the implementation ended up splitting those changes - any touch that changed was terminated in the first event frame, any touch that outright stopped was terminated in the second event frame. That in itself wasn't the problem yet, the problem was that libevdev didn't emulate the single-touch multi-touch protocol with those emulated frames. So we ended up with event frames where slots would terminate but the single-touch protocol didn't update until a frame later.

This doesn't matter for most users. Both protocols were still correct-enough in their own bubble, only once you start mixing protocols was where it all started getting wonky. libinput does this because it has to, too many devices out there only track two fingers. So if you want three-finger tapping and pinch gestures, you need to handle both protocols. Despite this we didn't notice until we added the quirk for ALPS devices. Because now libinput sometimes noticed that after a SYN_DROPPED there were no fingers on the touchpad (because they all stopped/changed) but the BTN_TOOL bits were still on so clearly we have a touchpad that cannot track all fingers it detects - in this case zero. [6]

So to recap: libinput's auto-adjustment of the touch count for buggy touchpad devices failed thanks to libevdev's buggy workaround of the device sync. The device sync we need because we can't rely on userspace handling touches correctly across SYN_DROPPED. An event which only gets triggered because the compositor is too buggy to read input events in time. I don't know how to describe it exactly, but what I can see all the way down are definitely not turtles.

And the sad thing about it: if we didn't try to correct for the firmware and accepted that gestures are just broken on ALPS devices because the kernel driver is lying to us, none of the above would have mattered. Likewise, the old xorg synaptics driver won't be affected by this because it doesn't handle multitouch properly anyway, so it doesn't need to care about these discrepancies. Or, in other words and much like real life: the better you try to be, the worse it all gets.

And as the take-home lesson: do upgrade to libinput 1.15.2 and do upgrade to libevdev 1.9.0 when it's out. Your kittens won't care but at least that way it won't make me feel like I've done all this work in vain.

[0] Unless the SYN_REPORT value is nonzero but let's not confuse everyone more than necessary
[1] A SYN_DROPPED is per userspace client, so a debugging tool reading from the same event node may not see that event unless it too is busy with feline renderings.
[2] yes, you'll get pointer jumps because event data is missing but since you've been staring at those bloody cats anyway, you probably didn't even notice
[3] usually, but not always
[4] on those devices, identifying a 3-finger pinch gesture only works if you put the fingers down in the correct order
[5] historical reasons: in theory a touch could change directly but most userspace can't handle it and it's too much effort to add now
[6] libinput 1.15.2 leaves you with 1 finger in that case and that's good enough until libevdev is released

Thursday, February 6, 2020

User-specific XKB configuration - part 1

The xkeyboard-config project is the repository for all XKB descriptions, or "keyboard layouts" as the layman would say. But languages are weird and thus xkeyboard-config contains an obscene amount of different layouts. And of course there are additional layouts that are more experimental than common [1].

The fault, as usual, lies with us (the pronoun, not the layout). XKB is weird and its flexible to the point of driving even bananas bananas but due to some historic accidents it's largely non-editable. All XKB files are installed in system folders and we all know the 11th commandment was "thou shalt not edit things in /usr/share". But, luckily, that is all about to change. Or rather: it has changed as of libxkbcommon 0.10.0, released Jan 20 2020.

xkeyboard-config provides two types of files. The ones that actually set up your keyboard layout and the ones that allow you to keep sane while doing so, despite your best efforts to the contrary.

KcCGST

Let's look at the first set of files. XKB uses "keycodes, compat, geometry, symbols, types", conveniently if unpronouncably called KcCGST. Keycodes map your "physical" scancode to an internal code-name. For example, your key with the digit 1 on it is AE01 (alphabetic key, row E from bottom, key number 1 from left). Then you map those keycodes into symbols (1 and !). This happens based on the key's type which defines the combination of modifiers to produce the symbols [2]. Compat is largely magic weird stuff (locking modifiers, pointer control) and geometry would let you draw a pretty picture of your keyboard if it was defined for your keyboard which it won't be.

To see the full keyboard layout simply run xkbcomp -xkb $DISPLAY - and marvel. xkeyboard-config keeps all these parts so your X server or Wayland compositor can load it at runtime depending on your layout.

RMLVO

But when it comes to actual layout selection we like our users enough that we don't make them handle KcCGST but rather provide them with RMLVO instead - "rules, models, layouts, variants, options". You select layout "us" and something converts this into the right components to actually load. Run setxkbmap -layout us -print to see this happening.

"layouts" is what you'd usually associate with a country (except politics is still a thing, so more weirdness here) and "variants" are variations of those. Layout "us" gives you QWERTY and "fr" gives you AZERTY but the "us(dvorak)" variant gives you whatever heresy dvorak applies to those physical keys. And of course, things don't stop there - options are tack-on thingies that do stuff. Like remapping caps lock to compose so you're less capable of shouting at me. Come to think of it, it should really be enabled by default for that reason. You can combine multiple options largely at-will. "models" are largely obsolete (except where they aren't) thanks to the Linux kernel evdev interface which makes all keyboard look the same. But they used to be a thing and maybe one day they'll make a comeback like bell bottom jeans. Disliked by everyone but some weirdos insist on using them.

Rules

Finally, we have rules and thus come to the core of the matter of this post. Rules are magic files that tell the various tools how to go from RMLVO to KcCGST. It's a weird format but it's quite understandable, just open /usr/share/X11/xkb/rules/evdev and have a looksie. It'll make you the popular kid at the next frat party.

Many many moons ago before the Y2K bug was even in its larvae stage, the idea was that you could configure all of those because every UNIX tool had to be more flexible than your yoga teacher. I'm unsure to what extent this was actually ever the case but around 2007-ish the old keyboard driver got deprecated and the evdev driver made it's grand entrance. And one side-effect of that was that things broke. evdev uses different keycodes, so all those users that copy-pasted unnecessary XKB configuration into their xorg.conf now had broken keys because they were applying the wrong rules. After whacking enough moles that we got in trouble with the RSPCA we started hardcoding the "evdev" ruleset everywhere. The xorg.conf option "XKBRules" became a noop and thus stopped breaking users' setups.

Except that it also stopped users from deploying their own rules files - something that probably didn't really matter anyway. This had some unintended side-effects though. First, to have a working custom XKB layout you basically had to get it merged upstream. Yes, you could edit the files locally but they'd just be overwritten next time you update the packages. Second, getting rid of hardcoded things is hard so we're stuck with the evdev ruleset for the forseeable future. This was the situation until, well, now.

User-specific rules and layouts

The new libxkbcommon release changes two things: it prepends $XDG_CONFIG_HOME/xkb/ to the lookup path for XKB rules (and other files). So any file in that path will be picked before the system paths. This makes it possible to have KcCGST files in your home directory and actually use them. This was somewhat possible before by passing the right flags to the various tools but now it's on by default - at least where libxkbcommon is used (Wayland).

Secondly, rules files now support an include statement. This means you can set your own rules and include system rules. Because everything is hardcoded to evdev this effectively means your new rule file will be $XDG_CONFIG_HOME/xkb/rules/evdev and have at least one line: ! include %S/rules/evdev. If you do just that, you get the evdev ruleset from the system installation path. And any lines you add before or after that line will be loaded. Have a look at the git commit for the details but the summary is that you'll have a rules file that looks like this:

$ cat $XDG_CONFIG_DIR/xkb/rules/evdev
! option                =       symbols
  custom:foo            =       +custom(foo)
  custom:bar            =       +custom(baz)

! include %S/evdev
This file will define the option->symbol mappings as above and then include the system-provided evdev rules file, i.e. it'll behave like before with those two added. To get those to do something, you need to have the actual symbols files:
$ cat $XDG_CONFIG_HOME/xkb/symbols/custom
partial alphanumeric_keys
xkb_symbols "foo" {
    key <TLDE> {        [      VoidSymbol ]       };
};

partial alphanumeric_keys
xkb_symbols "baz" {
    key <AB01> {        [      k, K ]       };
};

And voila, you can now use the XKB option "custom:foo" and/or "custom:bar" to remap your tilde or Z key. The rest is left to the reader as an exercise in creativity.

Remaining work

The libxkbcommon change was only the first part of the full feature. The remaining parts is to have libxkbcommon actually resolve XDG_CONFIG_HOME when running in gnome-shell which doesn't work right now thanks to secure_getenv() always returning NULL. That's an issue with gnome-shell in particular thanks to the rt-scheduler feature, enabled by default on Fedora already.

The second part, and harder, is to make the new options appear in the various graphical configuration tools. xkeyboard-config ships an XML file [3] that lists every possible combination with some human description for it. This XML file is used by the various tools directly but none of those tools support XML's XI:Include statements. So we'd either have to update all those tools to extend the parsing accordingly or, most likely a smarter long-term solution, write a wrapper library that provides a stable API to get at the same info. That way we can update the include paths under the hood without having to update every tool. Of course this requires every tool to update to the new library first, so, well, chicken, egg, usual problem. Anyway, we'll get there eventually.

[1] For example, I suspect a meetup of icelandic dvorak users doesn't qualify for a group discount.
[2] Each key has "levels" with one symbol each and modifiers that switch between those levels. Most keys have two levels - normal and shift. But there's a key type for EIGHT_LEVEL_ALPHABETIC_LEVEL_FIVE_LOCK and you can cry or laugh and both reactions are appropriate
[3] ask your grandparents about that, it's basically JSON for old people

Tuesday, January 28, 2020

libinput's timer offset errors

Let's say you have a friend (this wouldn't happen to you, of course, just that friend) who is staring at their system logs and wonder why it is full of messages similar to this:

 libinput error: client bug: timer event5 debounce short: offset negative (-7ms)
And the question is of course - what is going on here and why hasn't this been fixed yet. Now, the libinput documentation explains this already but it's always worthwhile to fire out a blog post into the void in the hope someone reads it.

libinput uses a specific model to communicate with the Wayland compositor (or the X server). There is a single epoll file descriptor and that fd will trigger whenever something happens that's of interest to libinput. When that fd triggers, the compositor is expected to call libinput_dispatch() which is the main "do stuff" function of libinput.

The actual trigger doesn't matter, it could be an event from a device but it could be something else. The caller doesn't have to care. All that matters is that there is code like this:

if (libinput_fd_triggered_in_select)
   libinput_dispatch();
And then libinput will do the right thing. Whether you also want events from libinput is almost orthogonal to this.

libinput uses timerfd internally so any timeouts also trigger the epoll fd. Timeouts are scheduled based on the event's time stamp, so if you get an event with timestamp T, a timeout of 180ms will be scheduled for time T + 180ms. So the process looks something like this:

T(0): kernel button event
T(0): libinput_dispatch(): schedule timeout for T(0+180)
...
T(180): epoll fd triggers
T(180): libinput_dispatch(): process timeout
...
This works generally fine. Even with some delays we don't generally need to worry about the timeouts and they still trigger as expected. But some of the timeouts are "short", as in 8ms short. And this is where these warnings may trigger.

Let's say your compositor is busy doing some rendering. The epoll fd triggers with a button event but the compositor is too busy to handle it immediately. Instead, it finishes whatever it's doing and only then calls libinput_dispatch():

T(0): kernel button event
...
T(12): libinput_dispatch(): schedule timeout for T(0+8)

libinput error: client bug: timer event5 debounce short: offset negative (-4ms)
libinput will still use the event's timestamp instead of the wall clock time so the scheduled timeout is no longer in the future. And that is when the error message will be printed. This isn't a libinput bug, it's always a bug in the compositor. Especially gnome-shell is still struggling with these instances and while great strives have been made to make it more responsive, it can still happen.

The error message may seem cryptic, but it provides a bunch of useful information: event5 is your event node, "debounce short" is the timer name so we know where we got stuck. And 4ms gives us an indication how much we got delayed.

And for the record: the other end of this issue - delayed libinput_dispatch() after a timeout should have triggered is handled quietly by libinput. For example, if you have a physical event queued and a timeout expiry, we will process the earlier one first to make sure the sequences are handled correctly.