Tuesday, June 22, 2010

xmodmap keyboard deconfiguration

Reading this post on linux.com last weekend made me cringe. Not because it is factually wrong (it isn't), but because what's described here is the reason for a great many bugreports and some rather uninformed hatemail. One thing that startled me in particular is the hint that "to really take charge, you need to dig into the X keyboard extension" when the rest of the article doesn't deal with XKB at all. This prompted me to write this blurb here.

The abstraction cardhouse

In X, keyboard input is abstracted. The keyboard sends a keycode to the client. The client then translates that keycode into a keysym (a glyph, in most cases, more or less), given a keycode-keysym table. Keycodes should be treated as arbitrary by the client. Keysyms on the other hand are constant, the keysym for "♥" is always the same (0x1002665). In the core protocol, that lookup table was reasonably straightforward (for some value of "straightforward"), with several modifiers providing access to different levels in the table. The table being based on the max number of modifiers. For 105 keys and two modifiers, the table was essentially 210 symbols wide (modifiers can be "shift", "alt", etc.).

That was deemed to be insufficient, causing the development of the X Keyboard Extension, lovingly referred to as "XKB" or "that bloody nail in my coffin". I won't go into details, but the lookup tables are a tad more complicated. XKB is integrated around core so that XKB and core applications can coexist nicely. Up to a point that for a long time we kept XKB state and core state separately inside the server instead of generating it on-the-fly. What's unfortunate though is that the XKB protocol spec describes how to get the core lookup table from the XKB data. It also describes how to extrapolate from core to XKB. It doesn't however cover the full circle. So if you go from XKB to core back to XKB, guesswork happens.

Anyway, what you need to take from this is that the general rule of thumb is - any commandline tool uses core, any tool that allows you to select a layout, variant or model uses XKB.
And core doesn't support device specifiers, so it can only ever work on the default device or core device, more on that below.

Pulling out the cards

All the above was mostly good until 2007. That's when input hotplug happened. And 2009, when MPX happened. Now none of the old tools work properly anymore. Not because we've broken them, they still do what they've always done. But the environment has changed and what they do isn't sufficient anymore. So before you think about writing an angry comment that we've broken your configuration: stop using input and output hotplugging, stop using suspend/resume, stop using a desktop environment that manages your keyboard configuration. Then you'll notice everything still works as before and you can xmodmap away.

So why did everything break since?
Back before input hotplug we had one device that was deemed the "core keyboard". Whenever anything configured the device it set the configuration on this core device. Since xset, xmodmap and friends work on core only, the XKB configuration was then modified from that core table. All that was fine, because the core keyboard was hardly ever changed. Now with input-hotplug and MPX, we have the physical devices, attached to what is now called "master devices" but still serve the same purpose as the old "core keyboard". However, we've greatly increased the flexibility of the devices and physical devices are more often than not configured directly now. What we then end up with is a master device that has multiple keyboards configured, all with possible different keyboard configurations. In fact, non-US layout users in Fedora 12 are practically guaranteed to have multiple keyboard configurations. We only apply the user-selected layout to actual keyboard devices, not to devices with keys (Power Button, Video Bus, etc.). These devices come up with the "US" default.

In the case of a user with two physical keyboards, one qwerty, one azerty, the matching master device needs to adjust the keyboard configuration of the device in use. This means that it will switch between the two configurations as the devices are used. That works fine too. But if you now use xmodmap to change the the keyboard config on the master device, this configuration only stays as long as the same physical keyboard device is being used.
Now, the simple fix for this is to apply the new configuration to all devices instead of just the master device. And that's exactly what we do. However, this doesn't work with hotplugging. Once a new device is plugged in, it comes up with the configuration configured in the xorg.conf again. And that config doesn't include whatever xmodmap settings were applied later. So once that device is used, the settings are lost again or at least lost on this device.
Now sprinkle into this the fact that core doesn't really support multiple groups (layouts) on the same keyboard, that keycodes may be different between devices, that with MPX a device that was a slave to an xmodmap configured master device may end up being attached to a master that has a different XKB configuration and you've got the start of an interesting career.

The fix for all this is simple: make the session handle all keyboards, existing and hotplugged ones, and let it update the config accordingly. Whenever you're using a tool like xmodmap it is your responsibility to reapply the config after changing keyboards. Which brings me back to the article above - it curiously didn't mention that. So I strongly recommend that instead of configuring your keyboard with xmodmap you spend your time helping your favourite desktop environment to provide tools to cover your use-case.

PS: surprisingly "deconfiguration" doesn't exist in the dictionary but it seems the best word to describe what xmodmap does.


kiilerix said...

Thanks for an interesting article. You make it clear what we used to do and why that no longer works - and that we thus should use session / desktop tools instead. But what do these tools (such as gnome-keyboard-properties, I assume) do instead and how are things connected now?

Where do libraries/APIs like libxkbfile and libxklavier fit into a modern picture?

Do the session / desktop tools do the same as setxkbmap and use the XKB layouts and the libxkbfile API?

Is layout switching handled inside XKB, so that the desktop tool only is used to load keyboard layouts initially? Are keyboard switcher applets handled through some other API?

Who are still using the old "core" API and its tools if they no longer works correctly? Will the "core" view of keyboard layouts be deprecated/removed soon?

I assume that setxkbmap and xkbcomp are the exceptions to "commandline tool uses core". Is there any particular reason that command line tools uses core and not XKB - or is it just that nobody needed them enough to write XKB tools?

From https://bugs.freedesktop.org/show_bug.cgi?id=16164 I got the impression that XKB was slightly broken by design, but that is only the xkm format which isn't that essential?

Unknown said...

That's all well and good and makes sense, but what are people like me who cobble together desktops from pieces supposed to do?

This is starting to sound like an unintentional penalization of anyone who attempts to avoid a muddle of custom, semi-opaque config file formats. (I'm just glad the whole HAL XML xorg.conf mess was undone)

Exiro said...

> So I strongly recommend that instead of configuring your keyboard with xmodmap you spend your time helping your favourite desktop environment to provide tools to cover your use-case.

What happened to the UNIX philosophy? "Do one thing and do it well." What if I decide I no longer like this DE, or something like the GNOME 2 --> 3 transition happens again, or my favourite DE is Unity and I want to move away from Ubuntu at one point? Redo it from scratch each time? This is horrible advice. Worst way to conclude an article.

Peter Hutterer said...

Exiro: no-one stops you from using xmodmap. you'll just have to re-run it all the time, since xmodmap does one thing, and one thing well. but only one thing...

Petr said...


thank you for your post, this explains the problems I've been having. In my case, the keyboard repeat rate keeps resetting by itself seemingly at random, the desktop environment (lxde) is not reapplying the desired settings.

Do you know how to detect these "deconfiguration" events? I could write a small daemon which periodically checks current values and reapplies the setting if changed. However, that does not look very nice. Is there a better way of doing this?