Making sense of backtraces with addr2line

When the X server crashes it prints a backtrace to the log file. This backtrace looks something like this:

(EE) Backtrace:
(EE) 0: /usr/bin/Xorg (OsLookupColor+0x129) [0x473759]
(EE) 1: /lib64/libpthread.so.0 (__restore_rt+0x0) [0x3cd140f74f]
(EE) 2: /lib64/libc.so.6 (__select_nocancel+0xa) [0x3cd08ec78a]
(EE) 3: /usr/bin/Xorg (WaitForSomething+0x1ac) [0x46a8fc]
(EE) 4: /usr/bin/Xorg (SendErrorToClient+0x111) [0x43a091]
(EE) 5: /usr/bin/Xorg (_init+0x3b0a) [0x42c00a]
(EE) 6: /lib64/libc.so.6 (__libc_start_main+0xf5) [0x3cd0821d65]
(EE) 7: /usr/bin/Xorg (_start+0x29) [0x428c35]
(EE) 8: ? (?+0x29) [0x29]

This is a forced backtrace from the current F20 X Server package, generated by killall -11 Xorg. There is not a lot of human-readable information but you can see the call stack, and you can even recognise some internal functions. Now, in Fedora we compile with libunwind which gives us relatively good backtraces. Without libunwind, your backtrace may look like this:

(EE) 
(EE) Backtrace:
(EE) 0: /opt/xorg/bin/Xorg (xorg_backtrace+0xb5) [0x484989]
(EE) 1: /opt/xorg/bin/Xorg (0x400000+0x8d1a4) [0x48d1a4]
(EE) 2: /lib64/libpthread.so.0 (0x3cd1400000+0xf750) [0x3cd140f750]
(EE) 3: /lib64/libc.so.6 (__select+0x33) [0x3cd08ec7b3]
(EE) 4: /opt/xorg/bin/Xorg (WaitForSomething+0x3dd) [0x491a45]
(EE) 5: /opt/xorg/bin/Xorg (0x400000+0x3561b) [0x43561b]
(EE) 6: /opt/xorg/bin/Xorg (0x400000+0x43761) [0x443761]
(EE) 7: /opt/xorg/bin/Xorg (0x400000+0x9baa8) [0x49baa8]
(EE) 8: /lib64/libc.so.6 (__libc_start_main+0xf5) [0x3cd0821d65]
(EE) 9: /opt/xorg/bin/Xorg (0x400000+0x25df9) [0x425df9]

So, even less information and it certainly makes it hard to figure out where to even get started. Luckily there is a tool to get some useful info out of that: eu-addr2line. All you need is to install the debuginfo package for the crashing program. Then it's just a matter of copying addresses.

$ eu-addr2line -e /opt/xorg/bin/Xorg 0x48d1a4
/home/whot/xorg/xserver/os/osinit.c:132

Alright, this is useful now, I can download the source package and check where it actually goes wrong. But wait - it gets even better. Let's say you have a driver module in the callstack:

(EE) Backtrace:
(EE) 0: /opt/xorg/bin/Xorg (xorg_backtrace+0xb5) [0x484989]
(EE) 1: /opt/xorg/bin/Xorg (0x400000+0x8d1a4) [0x48d1a4]
(EE) 2: /lib64/libpthread.so.0 (0x3cd1400000+0xf750) [0x3cd140f750]
(EE) 3: /opt/xorg/lib/libinput.so.0 (libinput_dispatch+0x19) [0x7ffff1e51593]
(EE) 4: /opt/xorg/lib/xorg/modules/input/libinput_drv.so (0x7ffff205b000+0x2a12) [0x7ffff205da12]
(EE) 5: /opt/xorg/bin/Xorg (xf86Wakeup+0x1b1) [0x4af069]
(EE) 6: /opt/xorg/bin/Xorg (WakeupHandler+0x83) [0x444483]
(EE) 7: /opt/xorg/bin/Xorg (WaitForSomething+0x3fe) [0x491a66]
(EE) 8: /opt/xorg/bin/Xorg (0x400000+0x3561b) [0x43561b]
(EE) 9: /opt/xorg/bin/Xorg (0x400000+0x43761) [0x443761]
(EE) 10: /opt/xorg/bin/Xorg (0x400000+0x9baa8) [0x49baa8]
(EE) 11: /lib64/libc.so.6 (__libc_start_main+0xf5) [0x3cd0821d65]
(EE) 12: /opt/xorg/bin/Xorg (0x400000+0x25df9) [0x425df9]

You can see that we have a xf86libinput driver (libinput_drv.so) which in turn loadsd libinput.so. You can debug the crash the same way now, just change the addr2line argument:

$ eu-addr2line -e /opt/xorg/lib/libinput.so.0 libinput_dispatch+0x19 
/home/whot/code/libinput/src/libinput.c:603

Having this information of course doesn't mean you can fix any bug. But when you're reporting a bug it can be invaluable. If I have access to the same rpms that you're running it's possible to look up the context of the crash in the source. Or, even better, since you already have access to those you can make debugging a lot easier by attaching the required bits and pieces to a bug report. Seeing a bug report where a reporter already narrowed down where it crashes makes it a lot easier than guessing based on hex numbers what went wrong.

Lenovo T440 touchpad button configuration

The T440 has a rather unusual touchpad with the buttons painted on top of the touchpad rather than the bottom. In addition, the separate set of buttons for the trackstick have gone the way of the dodo. Moving the software-emulated buttons up on the touchpad is obviously quite important for trackstick users but it throws up a bunch of problems. There are some limitations with the current synaptics X.Org driver: we can only have one region each designated for the right and the middle button. The rest of the touchpad is a left button click. In the case of the T440, the default Windows config has a right button up the top and another one at the bottom of the touchpad. An ASCII-art of that would look like this:

+----------------------------+
| LLLLLLLLLL MMMMM RRRRRRRRR |
|                            |
|                            |
|                            |
|                            |
|                            |
|                            |
| LLLLLLLL          RRRRRRRR |
+----------------------------+

We simply can't do that at the moment, best we can is split the touchpad so that the whole right side is a right-click and a strip in the middle that is a middle click. So the best we can do is:

+----------------------------+
| LLLLLLLLLL MMMMM RRRRRRRRR |
| LLLLLLLLLL MMMMM RRRRRRRRR |
| LLLLLLLLLL MMMMM RRRRRRRRR |
| LLLLLLLLLL MMMMM RRRRRRRRR |
| LLLLLLLLLL MMMMM RRRRRRRRR |
| LLLLLLLLLL MMMMM RRRRRRRRR |
| LLLLLLLLLL MMMMM RRRRRRRRR |
| LLLLLLLLLL MMMMM RRRRRRRRR |
| LLLLLLLLLL MMMMM RRRRRRRRR |
+----------------------------+

I'm working on a solution for the proper config, but for now you'll have to be content with this.

The easiest approach for local configuration is a new InputClass section in the form:

Section "InputClass"
    Identifier "t440 top buttons"
    MatchDriver "synaptics"
    #                         right btn|middle btn
    Option "SoftButtonAreas" "60% 0 0 0 40% 60% 0 0"
EndSection

Drop that into /etc/X11/xorg.conf.d/99-t440-synaptics.conf and you're good to go.

The problem is finding a generic solution to this that we can ship in a distribution. That requires a two-step progress. The touchpads look the same as all others, the only differentiator we have is the DMI information on the box. We can't check that in the xorg.conf snippets yet (Daniel Martin is working on a MatchDMI tag, but it won't happen until server 1.16). For now, we need a udev rule to help the xserver.

ACTION!="add|change", GOTO="touchpad_quirks_end"
KERNEL!="event*", GOTO="touchpad_quirks_end"
ENV{ID_INPUT_TOUCHPAD}!="1", GOTO="touchpad_quirks_end"

ATTR{[dmi/id]product_version}=="*T440*", \
  ENV{ID_INPUT.tags}="touchpad_softbutton_top"

LABEL="touchpad_quirks_end"

If our product matches T440, we tag the touchpad, and that tag is something we can match against. Our revised InputClass section now looks like this:

Section "InputClass"
    Identifier "t440 top buttons"
    MatchDriver "synaptics"
    MatchTag "touchpad_softbutton_top"
    Option "SoftButtonAreas" "60% 0 0 0 40% 60% 0 0"
EndSection

I've pushed this configuration into Fedora now (rawhide, F20, F19), let's see what the feedback is. Having the whole right-side of the touchpad work as right button may cause a few issues so this is one change I may have to revert in the future.

Wacom Serial devices and inputattach in a systemd world

If you don't have one of the Wacom serial devices, you probably don't need to worry about the below. If you do, read on.

Wacom serial devices are supported by the xf86-input-wacom driver directly, but a much nicer way is to have the kernel deal with it and thus the device exposed as a normal evdev device to userspace. This way, the device is also picked up by libwacom and, in turn, by the GNOME Wacom panel for configuration and calibration. For the device to work as evdev device you need inputattach (in the linuxconsoletools package on Fedora).

The simplest approach if your device is connected to /dev/ttyS0 is:

$> inputattach --daemon --w8001 /dev/ttyS0

but of course we want to automate this. We used to have a udev rule that fires up inputattach but recent changes [1] in udev broke that ability: udev is now decidedly for short-running processes only, long-running processes started by a udev rule will be killed. The solution to that is a udev rule paired with a systemd service.

First, the systemd wacom-inputtattach@.service service file, which is quite simple:

[Unit]
Description=inputattach for Wacom ISDv4-compatible serial devices

[Service]
Type=simple
ExecStart=/usr/bin/inputattach -w8001 /dev/%I

Note the @ in the name, we use what comes after through the %I directive as the device file, e.g. ttyS0. Note also that we don't use the --daemon flag because we want systemd to take care of our process instead of forking it and letting it guess what the PID is, etc. This service can now be started with

$> systemctl start wacom-inputattach@ttyS0.service

and of course the information is available in the journal, etc.

Now we just need something to start the service automatically, and that's handled in a simple udev rule. Here's the whole udev file for easy copy/paste:

ACTION!="add|change", GOTO="wacom_end"

SUBSYSTEM=="tty|pnp", SUBSYSTEMS=="pnp", ATTRS{id}=="WACf*", \
  ENV{ID_MODEL}="Serial Wacom Tablet $attr{id}", \
  ENV{ID_INPUT}="1", ENV{ID_INPUT_TABLET}="1", \
  ENV{NAME}="Serial Wacom Tablet $attr{id}"
SUBSYSTEM=="tty|pnp", SUBSYSTEMS=="pnp", ATTRS{id}=="FUJ*", \
  ENV{ID_MODEL}="Serial Wacom Tablet $attr{id}", \
  ENV{ID_INPUT}="1", ENV{ID_INPUT_TABLET}="1", \
  ENV{NAME}="Serial Wacom Tablet $attr{id}"

SUBSYSTEM=="tty|pnp", KERNEL=="ttyS[0-9]*", ATTRS{id}=="WACf*", \
  TAG+="systemd", ENV{SYSTEMD_WANTS}+="wacom-inputattach@%k.service"

LABEL="wacom_end"

Wacom's serial devices have IDs starting with WACf, so we match on that. The first two "SUBSYSTEM..." lines match and set the name and some ID_foo tags so that the server recognises the tablet as input device and can match against xorg.conf.d InputClass snippets. With just those first two lines, we will have the xf86-input-wacom driver work for serial devices, even if inputattach didn't start. These two lines haven't changed in years either. The new and systemd-specific bit is the third line. Again we match but this time we set the systemd tag, tell systemd the service we want to start (%k is replaced with the kernel device, e.g. ttyS0) and we're done. Reboot [2] and you should be the happy viewer of something like this:

$> systemctl status wacom-inputattach@ttyS0.service   
wacom-inputattach@ttyS0.service - "inputattach for Wacom ISDv4-compatible serial devices"
   Loaded: loaded (/usr/lib/systemd/system/wacom-inputattach@.service; static)
   Active: active (running) since Thu 2013-12-12 08:03:00 EST; 24min ago
 Main PID: 562 (inputattach)
   CGroup: /system.slice/system-wacom\x2dinputattach.slice/wacom-inputattach@ttyS0.service
           └─562 /usr/bin/inputattach -w8001 /dev/ttyS0

[1] I say recent but really, it could've been any time since F18 or so, I don't test this particular device very often
[2] yeah yeah, there's a command to trigger this directly, you don't need to reboot, I know

Update 2013/12/13: remove quotes around Description in unit file

argcheck - assert on steroids

I guess about 80% of the time coding is spent handling error messages and dealing with the situation where the input isn't what it should be. That's ok for a real project, but sometimes it's just easier to say "this shouldn't happen but make sure I see an error if it does". For example when prototyping, it's enough to know that something went wrong so we can throw out the output. Or get notified when a value that should be within a range is outside that range, etc. The traditional way of that is assert(3), which checks an expression and aborts the program when the condition is not met. That is a heavy hammer, and not flexible enough in many cases.

So that got me thinking: essentially I wanted a set of macros that I can throw into any function to alert me when something goes wrong. Similar to the BUG_ON in the kernel, or the BUG_WARN in X. But a bit more powerful than that, because the main issue I have with BUG_WARN is that it tells me when a condition fails, but not necessarily why, or what the condition was supposed to check. Two revisions later, Rusty Russell merged my new argcheck CCAN module. It provides a set of macros for error checking that do not abort and, more importantly, also work as conditions. The simplest use-case is:

#include <ccan/argcheck/argcheck.h>

void calculate(int percent) {
   argcheck_int_range(percent, 0, 100);
   // now do something
}

If percent is outside the [0..100] range, a message with the file, function and line number is printed. And, possibly more useful, it will also print the actual value of percent. It gets more interesting when we use symbolic names, and we use the whole things as a condition:

#define MAXIMUM 100

void something(int a) {
     if (!argcheck_int_lt(a, MAXIMUM))
          return;
     // now do something
}

int main(void) {
    something(500);
    return 0;
}

If a doesn't meet the condition, the error message is:

test.c:73 something(): condition "(a < MAXIMUM)" (500 < 100) failed

So not only does it give us the actual values, it also provides the symbolic names (if there are any). Which, for debugging purposes, makes the whole thing quite useful. And we don't need to double-evaluate, we can use the macro as condition. argcheck takes care to only evaluates its arguments once, so there shouldn't be any side-effects. This also goes for disabling argcheck. If ARGCHECK_DISABLE_LOGGING is defined when including argcheck.h no messages are logged but the conditions still exist and the code runs as before. The same goes for a user-defined argcheck_log function (in case you need to override the default fprintf(stderr)).

I've added quite a few macros for various use-cases, including:

• argcheck_flag_set(a, flag) - false if the flag isn't set, but also checks that the flag is not 0
• argcheck_str_not_zero_len(str) - false if the string is NULL or the empty string
• argcheck_str_max_len(str, len) - false if the string is NULL, or longer than len. Works great to warn about truncated string copies.
• argcheck_ptr_null(ptr) - false if the pointer is not NULL. Works great to spot uninitialized variables.

Any comments or feedback let me know, I'm eager to improve this to make it even more useful (it already saved my hide a few times)

evtest is dead. long live evemu

No, don't worry, evtest isn't actually dead. I'll keep maintaining it (it's not a lot of work, after all). But I'll discourage its use and in the future you should be using evemu instead.

evtest is a tool that prints out a human-readable description of an evdev kernel device and its events. evemu does the same, but it also records the events in a format that can be parsed and re-played easily on another machine, making it possible to reproduce bugs easily.

evemu was originally written by Henrik Rydberg, had a stint on launchpad and moved to freedesktop.org earlier this year. Since then, Benjamin Tissoires and I have been working quite a bit on it, trying to make improve on the usability. We're still polishing a few things, but since version 1.1 we have a UI that I'm now reasonably happy with it. And Benjamin we just released 1.2.

evemu has two modes: recording mode and replaying mode. In recording mode, evemu records the bits that the kernel exports for a device and writes them into a file. This includes both the device description and the events (if any) from the device. In replaying mode, evemu creates a virtual (uinput) device that looks exactly the same as the original file [1] and replays the events in the same order and timeframe. The result is that if you record your device and send it to me, chances are I can reproduce the bug without having the hardware. And of course, for the nastier bugs having a recording that reliably reproduces something is great for testing.

Let's look at it in more detail. The first step for a reporter is evemu-record [2].

$> sudo evemu-record
Available devices:
/dev/input/event0: Lid Switch
/dev/input/event1: Sleep Button
/dev/input/event2: Power Button
/dev/input/event3: AT Translated Set 2 keyboard
/dev/input/event4: SynPS/2 Synaptics TouchPad
/dev/input/event5: PIXART USB OPTICAL MOUSE
/dev/input/event6: Microsoft Microsoft® Digital Media Keyboard
/dev/input/event7: Video Bus
/dev/input/event8: TPPS/2 IBM TrackPoint
/dev/input/event9: Wacom ISDv4 E6 Pen
/dev/input/event10: Wacom ISDv4 E6 Finger
/dev/input/event11: Microsoft Microsoft® Digital Media Keyboard
/dev/input/event12: Integrated Camera
/dev/input/event13: ThinkPad Extra Buttons
/dev/input/event14: HDA Intel PCH HDMI/DP,pcm=8
/dev/input/event15: HDA Intel PCH HDMI/DP,pcm=7
/dev/input/event16: HDA Intel PCH HDMI/DP,pcm=3
Select the device event number [0-16]: 

This is obviously the list of devices on my machine, selecting a number will record that device (e.g. 4 will start recording the touchpad). Recording the output produces a bunch of comments describing the device in human-readable form, followed by the bits that we use within evemu.

# EVEMU 1.2
# Input device name: "SynPS/2 Synaptics TouchPad"
# Input device ID: bus 0x11 vendor 0x02 product 0x07 version 0x1b1
# Supported events:
#   Event type 0 (EV_SYN)
#     Event code 0 (SYN_REPORT)
#     Event code 1 (SYN_CONFIG)
#     Event code 3 (SYN_DROPPED)
#   Event type 1 (EV_KEY)
#     Event code 272 (BTN_LEFT)
#     Event code 325 (BTN_TOOL_FINGER)
#     Event code 328 (BTN_TOOL_QUINTTAP)
#     Event code 330 (BTN_TOUCH)
#     Event code 333 (BTN_TOOL_DOUBLETAP)
#     Event code 334 (BTN_TOOL_TRIPLETAP)
#     Event code 335 (BTN_TOOL_QUADTAP)
#   Event type 3 (EV_ABS)
#     Event code 0 (ABS_X)
#       Value   2680
#       Min     1472
#       Max     5472
#       Fuzz       0
#       Flat       0
#       Resolution 75
#     Event code 1 (ABS_Y)
#       Value   4550
#       Min     1408
#       Max     4448
#       Fuzz       0
#       Flat       0
#       Resolution 129
#     Event code 24 (ABS_PRESSURE)
#       Value      0
#       Min        0
#       Max      255
#       Fuzz       0
#       Flat       0
#       Resolution 0

[...]

N: SynPS/2 Synaptics TouchPad
I: 0011 0002 0007 01b1
P: 05 00 00 00 00 00 00 00
B: 00 0b 00 00 00 00 00 00 00
B: 01 00 00 00 00 00 00 00 00
B: 01 00 00 00 00 00 00 00 00
B: 01 00 00 00 00 00 00 00 00
B: 01 00 00 00 00 00 00 00 00
B: 01 00 00 01 00 00 00 00 00
B: 01 20 e5 00 00 00 00 00 00
B: 01 00 00 00 00 00 00 00 00
B: 01 00 00 00 00 00 00 00 00
B: 01 00 00 00 00 00 00 00 00
B: 01 00 00 00 00 00 00 00 00
B: 01 00 00 00 00 00 00 00 00
B: 01 00 00 00 00 00 00 00 00
B: 02 00 00 00 00 00 00 00 00
B: 03 03 00 00 11 00 80 60 06
B: 04 00 00 00 00 00 00 00 00
B: 05 00 00 00 00 00 00 00 00
B: 11 00 00 00 00 00 00 00 00
B: 12 00 00 00 00 00 00 00 00
B: 15 00 00 00 00 00 00 00 00
B: 15 00 00 00 00 00 00 00 00
A: 00 1472 5472 0 0 75
A: 01 1408 4448 0 0 129
[...]

And that is followed by events that look something like this:

E: 1382571966.639767 0000 0000 0000 # ------------ SYN_REPORT (0) ----------
E: 1382571966.650972 0003 0036 4044 # EV_ABS / ABS_MT_POSITION_Y    4044
E: 1382571966.650972 0003 003a 0041 # EV_ABS / ABS_MT_PRESSURE      41
E: 1382571966.650972 0003 0001 4044 # EV_ABS / ABS_Y                4044
E: 1382571966.650972 0003 0018 0041 # EV_ABS / ABS_PRESSURE         41
E: 1382571966.650972 0000 0000 0000 # ------------ SYN_REPORT (0) ----------
E: 1382571966.663486 0003 0039 -001 # EV_ABS / ABS_MT_TRACKING_ID   -1
E: 1382571966.663486 0001 014a 0000 # EV_KEY / BTN_TOUCH            0
E: 1382571966.663486 0003 0018 0000 # EV_ABS / ABS_PRESSURE         0
E: 1382571966.663486 0001 0145 0000 # EV_KEY / BTN_TOOL_FINGER      0
E: 1382571966.663486 0000 0000 0000 # ------------ SYN_REPORT (0) ----------

With the bits we need to replay on the left, and comments for humans on the right.

The comments with the human-readable description are new in version 1.1.0 which makes it finally useful as a replacement for evtest. Previously I had to either replay a device and look at the replay with evtest to make sense of something. But then again, previously the evemu repo was on launchpad, so having to use two tools was less painful than working with bzr...

To replay a device, you'll need two tools: evemu-device and evemu-play. The former creates a device based on the device description. The latter pumps the events into the newly created device.

$> sudo evemu-device my-device-recording.txt
CyPS/2 Cypress Trackpad: /dev/input/event17
$> sudo evemu-play /dev/input/event17 < my-device-recording.txt

And that's it. You've created a new device and that device now spits out the events as recorded. evemu-play will replay the events so that the timing is as close as possible to the original recording, meaning that you can reproduce most bugs.

Speaking of bugs: if you have an input-related bug that is triggered by a given event sequence, record your device. Make sure the recording reproduces the bug and attach the file to the bugreport. I almost always ask for an evemu recording anyway, so having that ready just speeds up the process.

[1] uinput doesn't let you set some fields but that is fine for 95% of the use cases
[2] There used to be evemu-describe, but that's a symlink to evemu-record now

Neues deutsches Tastaturlayout (Changes to the German keyboard layout)

Ich hab's zuerst nicht bemerkt, darum kommt diese Warnung etwas spät. Dennoch: seit Fedora 19 (xkeyboard-config 2.8) gibt es eine kleine Änderung im deutschen Tastaturlayout.

Die Tilde taste ("~"), zu finden neben der Entertaste, betätigt mit AltGr und "+" war ein sogenannter dead key. Solche dead keys ermöglichen es mit mehreren Eingaben Buchstaben wie etwa "ñ" zu erzeugen (tilde + n). dead key hieß allerdings auch dass, um eine tatsächliche Tilde einzugeben, "~", die Taste zweimal betätigt werden musste. Dieses Layout war anders als die DIN und das Standardlayout in Windows und wurde daher geändert. Siehe (fdo bug 9753). Die neue Belegung ist Tilde, ohne dead key Funktionalität.

Falls das alte Layout bevorzugt wird, muss das Layout auf Deutsch(veraltet) gestellt werden. Dies kann entweder über setxkbmap -layout "de(legacy)" bzw. setxkbmap -layout "de" -variant "legacy" geändert werden, bzw in GNOME über eine etwas versteckte Option. Im "Region und Sprache" Dialog den "+" Button betätigen, danach die drei vertikalen Punkte anklicken. Das öffnet das Suchfenster, wo dann nach Deutsch(veraltet) gesucht werden kann. Diesen Eintrag wählen und fertig.

Actually typed this in English first and then felt a bit silly. Anyway, here's the English version:

Ok, I'm a bit late here because I didn't notice and apparently most Fedora users with a German keyboard layout didn't either. Nevertheless, a PSA: the keyboard has changed upstream in version 2.8 which is the one we're shipping in Fedora 19 and thus later versions as well.

The tilde "~" key, on AltGr and the + key, left of the Enter key, used to be a dead key. Dead keys allow multi-key combinations. In this specific case the dead key tilde followed by n would produce ñ. It also means that to type an actual "~" the tilde key had to be typed twice. This layout was against the DIN norm and differed from the default Windows layout too, so it was changed to a normal tilde (see fdo bug 9753, opened in 2007!).

If you prefer the old workings, you'll need to switch to the German(legacy) variant. If you configure with setxkbmap, simply use setxkbmap -layout "de(legacy)" or setxkbmap -layout "de" -variant "legacy". If you're using GNOME, the configuration is a little hidden. Fire up the "Region & Language" dialog in the control center ("Region und Sprache"), hit the little + button and then the three vertical dots which open up the search field.

Search for "legacy" and you'll see German(legacy) pop up. Select it and you're good to go. If you have your desktop in German, the entry to search for "Deutsch (veraltet)".

The X.Org Foundation and the 501(c)(3) status

X.Org is not a single thing. One the one hand, X.Org is a collection of projects that provide a window system stack, most notably the X server, its drivers and Xlib. Some other tools are part of the X.Org software set (see the X11R7.7 katamari release). Additionally, there are projects that are more or less associated with X.Org but aren't necessarily X.Org (depends on who you ask, and it doesn't really matter anyway for this post). These projects include Mesa and Wayland. Let's call this the "Software".

On the other hand, X.Org is the X.Org Foundation, a non-profit corporation to help advance X.Org and related projects. Let's call this the "Foundation". Disclaimer upfront: I'm the current Secretary for the X.Org Foundation.

The Software and the Foundation are not the same thing, but they do share the name. The Foundation has zero technical influence over the development of the Software, it cannot direct roadmaps, it cannot decide on releases, it can't decide on technical features. The Foundation does have money it can provide to developers of the Software though. This is done indirectly, e.g. through financing developer conferences (XDC) and providing travel sponsorship for attendees. The Foundation backs the Endless Vacation of Code (think Google Summer of Code without specific start dates).

Generally, what happens with the Foundation doesn't really matter to the Software and that is by design. If the Foundation goes away, the worst thing that would happen to the Software is that funding for conferences would have to come from elsewhere. Think of the Foundation as being the rich auntie for the developers. You can ask her for money to travel, but she doesn't participate in day-to-day development work.

Recently, you may have read that the Foundation lost 501(c)(3) status (temporarily anyway). For the non-US readers, 501(c)(3) is US legalese for a non-profit organisation which allows for some donations to be tax-deductible. The loss of the 501(c)(3) status caused the usual internet outrage and insults towards the Foundation and the Software, so let's look at that in a bit more detail.

The X.Org Foundation structure

There is a link between the members of the Foundation and the Software: to become a member, you have to "i) be actively involved in the activities relating to the technologies of X.Org, as set forth in the Membership Agreement". So as long as you're hacking on X or you're otherwise affiliated, you can join and membership is free.

The day-to-day work of the Foundation are largely handled by the Board of Directors, 8 elected members in two-year terms, 4 of which are up for election every year. One treasurer, one secretary, pretty much standard stuff. The board has regular meetings, but if they get your blood pressure up you really need to see a doctor.

Gaining, losing and re-gaining 501(c)(3) status

The reason we even have that status is historical. Back a few years ago the old X Consortium got regular chunks money from large corporations (Sun, HP, to name a few). After the change to X.Org, setting that up for tax-deductible donations seemed like a good idea. Fast-forward a bit and with the help of the great people at the SFLC X.Org got 501(c)(3) status in 2012 (retroactively applied to 2009).

Something else happened while fast-forwarding: we never actually solicited or took any donations. The financial crisis had hit, so there wasn't that much money around and anyway we had enough money to run conferences for years to come. The closest thing to a donation we got was to allow interested parties to host beer bashes at our conferences.

So since 2005, we've been slow-burning through money and at the current rate we still have enough money for a couple of years of conferences and travel sponsorship. 501(c)(3) limits the ways how to spend the money (see Wikipedia), so we pretty much only spend it on conferences and travel.

So the summary: the Foundation is a 501(c)(3) that doesn't collect donations. But if we did, they'd be tax-deductible. As long as you're in the US, that is. Non-US people pretty much have nothing to gain either way since we're not registered in any other country and 501(c)(3) doesn't apply to other (any?) countries. (Correct me if I'm wrong. Also: don't listen to me for tax advice)

A couple of months ago 2013 we lost 501(c)(3) status. The reason we lost it: we didn't file the required tax forms. As said above, no donations means no income, so we never actually owed any taxes and we hadn't filed tax forms for years before we go the 501(c)(3) status. So the IRS removing our status was a bit of a surprise. I should also note that the Foundation is one step removed from from the IRS communication: the IRS contacts the SFLC who then contacts us.

Anyway, once again the great people at the SFLC helped us and we got the 501(c)(3) re-instated. So we're back to normal, not collecting tax-deductible donations. With the additional benefit of having learned to double-check tax forms from now on.

Summary

We had 501(c)(3) status but didn't collect donations. Then we lost it, and got it back, all the time not actually collecting donations. There was no money owed to the IRS. No doubt: It's embarrassing, it shouldn't have happened. But in reality it had zero effect.

So pack up the pitchforks, but keep them well oiled, I'm sure there's a reason to get them out soon again.

Final note: you may have heard that the Foundation is planning to join SPI. This is correct but still ongoing and I'll talk about that when it's actually done.