Sunday, June 7, 2009

XI2 Recipes, Part 1

XI2 is now merged into master. Over the next couple of days, I will post various recipes on how to deal with the new functionality. The examples here are merely snippets, full example programs to summarize each part are available here.

Update 13.07.09: Adjusted adjusted to the new cookie event API

In this first part, I will cover general things, initialisation and event selection.

Why XI2?


One of the major reasons for XI2 was the merge of MPX into the server. The current X Input Extension version 1.5 is quite limiting and extending it to fully support MPX has been tough. Programming against it is mostly annoying, so a replacement was sought that makes it easy to program against multiple devices and is flexible enough to cover future use-cases.

The XI2 protocol is fairly conservative, adding only a few requests and events but it also leaves room for more. XI2 and it's APIs are somewhat closer to the core protocol paradigms. The big differences to XI from a client's perspective are: Calls take a deviceid as parameter, there is no need to open devices and event types and masks are constant (more of this below).

Right now, the only bindings for XI2 are Xlib bindings. Nonetheless, I encourage you to start playing with XI2 and think how applications may use it. By testing it now, you can help identify problems and missing bits with the current requests before version 2.0 is released. And of course, fixing issues before a release benefits everyone.

MPX


MPX allows the use of multiple cusors and keyboard foci simultaneously. This again leads to pretty funky user interfaces - bi-manual, multi-user, you name it. It also throws a number of assumptions about current GUIs out of the window but I'll get to that some other time.

MPX introduces an explicit master/slave device hierarchy. The easiest way to remember which is which is: a physical device is a slave device, and a cursor or keyboard focus is a master device.

Slave devices are attached to one master and each time a slave generates an event, this event is routed through the master device to the client. Master devices always come in twos (one cursor, one keyboard focus) and these two are paired.

So a common setup may be a laptop with four master devices (i.e. two cursors and two keyboard foci), the touchpad controls and built-in keyboard control the first pair of master devices, and a USB wireless combo controls the second pair of master devices. The standard setup is to have one pair of master devices, and all devices are attached to this pair's master pointer or master keyboard. Which, incidentally (or not :), is exactly the same setup as we've had since server 1.4. It also means that MPX only takes effect if you create a new pair of master devices, otherwise it's invisible.

I will cover more of MPX in a follow-up post. From most clients' perspective, master device are the devices that matter, only configuration tools and some other specialised apps need to worry about slave devices.

XI2 Initialisation



The typical XI2 program starts something like this:

/* Connect to the X server */
Display *dpy = XOpenDisplay(NULL);

/* XInput Extension available? */
int opcode, event, error;
if (!XQueryExtension(dpy, "XInputExtension", &opcode, &event, &error)) {
printf("X Input extension not available.\n");
return -1;
}

/* Which version of XI2? We support 2.0 */
int major = 2, minor = 0;
if (XIQueryVersion(dpy, &major, &minor) == BadRequest) {
printf("XI2 not available. Server supports %d.%d\n", major, minor);
return -1;
}


First the client connects to the X server, then asks whether the extension is available. XQueryExtension not only tells us whether the X Input Extension is supported, but it also returns the opcode of the extension. This opcode is needed for event parsing (all XI2 events use this opcode). This opcode is set when the server started, so you cannot rely on it being constant across server sessions.

Finally, we announce that we support XI 2.0 and the server returns the version it supports. Although XIQueryVersion is a pure XI2 call, it is implemented so that it will not result in a BadRequest error if you run it against a server that doesn't support XI2 (this part is implemented in Xlib, so if you use xcb this behaviour is not the same).

XIQueryVersion not only returns the supported version, the server will also store the version your client supports. As XI2 progresses it becomes important that you use this call as the server may treat the client differently depending on the supported version.

Selecting for events


After initialising the GUI, a client usually needs to select for events. This can be achieved with XISelectEvents.


XIEventMask eventmask;
unsigned char mask[1] = { 0 }; /* the actual mask */

eventmask.deviceid = 2;
eventmask.mask_len = sizeof(mask); /* always in bytes */
eventmask.mask = mask;
/* now set the mask */
XISetMask(mask, XI_ButtonPress);
XISetMask(mask, XI_Motion);
XISetMask(mask, XI_KeyPress);

/* select on the window */
XISelectEvents(display, window, &eventmask, 1);


An XIEventMask defines the mask for one device. A mask is defined as (1 << event type) and the matching bits must be set in the eventmask.mask field. The size of eventmask.mask can be arbitrary as long as it has enough bits for the masks you need to set. The XI_LASTEVENT define specifies the highest event type in the current XI2 protocol version, so you can use this to determine the mask size. In this example, we only need 6 bits, so a 1 byte mask is enough.

XISelectEvents takes multiple event masks, so you can submit many of these (e.g. one for each device) in one go.

As shown above, each XIEventMask takes a deviceid. This is either the numeric ID for a device or the special ID of XIAllDevices or XIAllMasterDevices. If you select an event mask for XIAllDevices, all devices will send the selected events. XIAllMasterDevices does the same, but only for master devices.

Event masks for XIAllDevices and XIAllMasterDevices are in addition to device-specific event masks. For example, if you select for button press on XIAllDevices, button release on XIAllMasterDevices and motion events on device 2, device 2 will report all three types of events to the client (the effective mask is a bitwise OR).

Furthermore, XIAllDevices and XIAllMasterDevices always apply, even if the device has been added after the client has selected for events. So you only need to issue the event mask once, regardless of how many devices are currently connected and how many will be connected in the future.

The check for events is quite simple:

XEvent event;
XNextEvent(display, &event);
if (ev.xcookie.type == GenericEvent &&
ev.xcookie.extension == opcode &&
XGetEventData(dpy, &ev.xcookie))
{
switch(ev.xcookie.evtype)
{
case XI_ButtonPress:
case XI_Motion:
case XI_KeyPress:
do_something(ev.xcookie.data);
break;
}
}
XFreeEventData(dpy, &ev.xcookie);


A more detailed analysis of the data in each event will be described in a later post.
Note that you need to check against the opcode of the X Input Extension as returned by XQueryExtension(3) (as discussed before).

That's it. With this knowledge, you can already go and write simple programs that listen for events from any device. In the next part, I will cover how to list the input devices and listen for changes in the master/slave device hierarchy.

Friday, June 5, 2009

git patches from tarballs

Generating patches with git is easy if you clone upstream. Many users don't run their software from repositories. They work from either distribution packages or tarballs. So a number of times I've heard something along the lines of "Sorry, it's not a git patch because I'm working from the tarball". It is quite easy though to create git patches from your tarballs. Simply run the following command in the extracted directory:


git init && git add --ignore-errors .; git commit -m "`basename $PWD`"


Explanation:
  • "git init" initialises a new git repository.

  • "git add ." adds all existing files to the repo. The --ignore-errors is there so git skips over files that can't be added, I've had that happen in a few tarballs that had their permissions busted, etc. If there are errors, you obviously need to check whether they affect files you want to hack on. If not, ignore the errors. (btw. you want to run git add before compiling everything, having all object files in the git index is painful)

  • "git commit" commits all newly added files with the name of the current directory as commit message.



Now the directory is basically the same as upstream when they released the tarball, without the history. Either way, you can just hack, commit, rebase, etc. and then create a patch with git-format-patch and submit it to upstream. Assuming that upstream hasn't diverged too much from the tarball, chances are the maintainers can just apply the patch as-is.

Disclaimer: I learned this workflow from the Fedora X11 packages where this method is used to apply upstream patches to the tarballs. So the credit goes to ajax (or maybe someone else).

[edits]
"git init-db" replaced with "git init".

Tuesday, June 2, 2009

Button mapping in X

X differs between device buttons and logical buttons. The former are handled in the device/driver, the latter are what applies to clients. By default, these are mapped in a 1:1 fashion, with device button 1 (D1) being mapped to logical button 1 (L1), D2 to L2, D3 to L3, etc.

Logical buttons are what the client sees in the end. The first logical button is L1 and there are some standard behaviours:
L1 .. left click
L2 .. middle click
L3 .. right click
L4 .. vert scroll wheel up
L5 .. vert scroll wheel down
L6 .. horiz scroll wheel left
L7 .. horiz scroll wheel right

These are de-facto standard in virtually all clients, with middle click usually resulting in a paste from the current selection. Additionally, some clients (e.g. firefox) map L8 and L9 to back/forward.
All these standard actions are by convention only, there's no defined standard (that I know of) and a client may choose to interpret L4 as left clicks (hint: this is not a smart thing to do).

The meaning of the logical buttons is always the same but the user can re-map device buttons to logical buttons. For example, a left-handed mouse uses a button mapping of D1:L3 and D3:L1 (a mapping of 3 2 1 4 5 6 7 ...). Now, when a button 3 is pressed, the server maps this button to logical button 1 and the client executes a left-click.

In-driver physical button mapping


There's one more stage of button mapping, and that stage happens in the driver. The driver communicates with the server over a defined API and the buttons passed across this API are treated as the device buttons. The driver may chose to map the physical button to a different device button to accommodate for certain hardware features or deficiencies. This mapping is driver-specific and may not exist for all drivers. Evdev has it mainly to accommodate for broken hardware, and synaptics has it for tap-to-click behaviour.

As an example, an evdev button mapping may ask for physical button 1 (P1) being mapped to device button 3. When P1 is pressed on the device, evdev passes D3 to the server which may then map it to L1 if a right-handed mouse is configured.

This method is more commonly used in synaptics for the tap-to-click behaviour. This is a purely in-driver emulation of a button, so the TapButton1/2/3 settings define what device button will be sent on a one/two/three finger tap. By default, TapButton1 is mapped to D1, but one could set it to D4 and scroll by tapping. This mapping is handy for the right-handed synaptics quirk described later.

The button map chain


Starting with server 1.6, devices have a chain of button mappings. Since both the physical device and the core pointer show up as separate devices (run xinput --list -short), they may have different button mappings. Virtually all clients only see the core pointer's events and any device that controls the core pointer is subject to the core pointer's mappings. The button mapping for physical devices that control the core pointer is thus as follows:


physical to device mapping → device to logical mapping → core pointer device to logical → client.


No contemporary configuration tool should change the core pointer's mapping. These tools should be XI aware and modify the configuration for each device.

In the following examples, '→' is used for a standard X:X mapping, and '»' for a X:Y button mapping. Assume Mouse M that controls the core pointer CP. Both have the standard 1 2 3 mapping with no in-driver mapping. The path is thus:


M(P1) → M(D1) → M(L1) → CP(D1) → CP(L1) → client.


If a 3 2 1 left-handed mapping on the core pointer is set up, the path is:

M(P1) → M(D1) → M(L1) → CP(D1) » CP(L3) → client.

Thus, all devices that send events through the core pointer are now left-handed.

On the other hand, if only a single device needs to be left-handed, the core pointer maintains it 1 2 3 mapping and the mouse M gets a 3 2 1 mapping:

M(P1) → M(D1) » M(L3) → CP(D3) → CP(L3) → client.

This is the setup configuration tools should choose if the user requires a left-handed mouse.

If a the same mapping is applied to both CP and M, then this mapping is neutralised:

M(P1) → M(D1) » M(L3) → CP(D3) » CP(L1) → client.

Although both M and CP have a left-handed setup, M effectively acts as a right-handed mouse. Even worse, an application that listens for XI events (e.g. the gimp) would see M as a left-handed mouse while all other applications would see it as a right-handed one. This is the reason why the core pointer mapping should be left alone.

Now, for fun, let's also assume that M has a hardware problem and physical button 4 is actually the left mouse button. So an in-driver mapping from 4 to 1 is needed.

M(P4) » M(D1) » M(L3) → CP(D3) » CP(L1) → client.


These examples above are the most common scenarios the average user may encounter.

Scrollwheel and buttons


As mentioned above, scrollwheels are mapped to the logical buttons 4,5 and 6,7. This has historical reasons, the core protocol only allows for two axes and by the time scrollwheels came about, the protocol was already set in stone. So now they're mapped in-driver to logical buttons instead and most applications and toolkits interpret it correctly. This is not ideal, as mapping wheel events to button events looses information (such as missing out on pointer acceleration for smoother scrolling).

The right-handed synaptics quirk


One common problem is how to configure a right-handed synaptics pad while leaving tap-to-click behaviour in the default configuration. As mentioned above, tap-to-click is an in-driver mapping. Right-handed behaviour is (usually) a global mapping applied by the desktop environment and may even be applied to the core pointer (especially with older, non-XI aware configuration tools). The easiest solution is to map tapping in-driver to an unused button number and then map this button back to the logical buttons as desired. For example, TapButton1 on synaptics device S could be mapped to 10, and D10 mapped to L1.


tap 1 » S(D10) → S(L10) → CP(D10) » CP(L1) → client


Thus, even if S is set up with a 3 2 1 mapping for the first three buttons, tapping still works in a "right-handed" fashion.

Note that in order to map two device buttons to the same logical button, X Input 1.5 is needed (server 1.6).

The tools


For a device button mapping:
XSetDeviceButtonMapping(3)
xinput --set-button-map 1 2 3 4 5 ...

For the core pointer mapping:
XSetPointerMapping(3)
xmodmap -e "pointer = 1 2 3"

For the synaptics mapping:
synclient TapButton1=1

For an evdev button mapping:
Option "ButtonMapping" "1 2 3 ..."


[edits]
mclasen pointed out that the middle button is "primary paste", not "clipboard paste". Text amended to be neutral regarding the paste type.

Thursday, May 28, 2009

XI2 merge coming up

I've sent a pull warning to xorg-devel, next week at this time XI2 should be part of master.

Saturday, May 16, 2009

My workflow

After posting about the X.Org supermodule I was asked for more details about how to get everything up and running. So here's my usual workflow. I'm open for improvements in the (moderated) comments.

Prerequisites


My workflow currently includes two patches to git, the second of which is still under review.
Install git from git and apply these patches. They add rebasing support to git submodule update and make the process a lot easier.

Set up a bunch of environment variables that you'll need each time you build:

$> export PKG_CONFIG_PATH=/opt/xorg/lib/pkgconfig
$> export PATH=/opt/xorg/bin:$PATH
$> export LD_LIBRARY_PATH=/opt/xorg/lib
$> export ACLOCAL="aclocal -I /opt/xorg/share/aclocal"


and some more that are just handy:

$> export CFLAGS="-Wall -O0 -ggdb"
$> export CC="ccache gcc"
$> export MAKEFLAGS="-j3"


Building X.Org from git in 5 steps



$> git clone git://people.freedesktop.org/~whot/xorg.git
$> cd xorg
$> git submodule init
$> git submodule update
$> ./util/modular/build.sh -f built.modules /opt/xorg

The supermodule is set up for automatic rebasing and you'll end up with a tree running each module on master. The last command builds everything in the right order and - with the "-f" flag - echos the modules being built into the built.modules file. If it fails (usually due to missing packages) you can resume from the last to-be-built component (the last one in built.modules).


$> ./util/modular/build.sh -f built.modules -r `tail -n 1 built.modules` /opt/xorg


Once that is done, you're left with an X tree in /opt/xorg, most important of which is the binary in /opt/xorg/bin/Xorg.

Working with the tree


I tend to have three or more branches in most repos. The branches that matter are "master", "queue" and "devel". Then I have additional branches for features that result in a patch series (e.g. "xi2").

"master" is always as close to upstream master as I can get. Anything that lands on master will likely be rebased and pushed upstream. Day-to-day bugfixing also happens on master.

"queue" is for patches I sent to the xorg-devel list. The workflow here is usally development on some other branch, then git-format-patch + email, then cherry-picking from the other branch to queue. Patches in "queue" get cherry-picked to master and pushed, and once master is pushed, "queue" is rebased onto master.

"devel" usually happens when I realize that the patch series on master is more than it should be. This is when I branch master into devel, reset master to the previous state and continue on devel. devel is heavily rebased and sometimes doesn't lead anywhere. If it does, intermediate patches are cherry-picked onto master and pushed when they're ready. devel is deleted as soon as I finish with it.

Feature branches (e.g. "xi2") tend to be the same as devel but with a specific feature in mind. Anything that isn't related to it (bugs that I find in code around that feature) is cherry-picked to master and queue, and the feature branch then rebased.

So a single patch may wander from xi2 to devel (when xi2 is branched for some reason) to queue to master before being pushed.

I've been using this workflow for months now, and one of the main reasons why it works fine for me is tig. Tig shows other branches heads and tags in the history list, so by rebasing often I always have a visual marker where the new patches start. Tig also makes cherry-picking easy, so I use it more often than git pull.

The other thing that is incredibly helpful is the zsh git prompt I got from here and modified a bit. My current output reads:

:: whot@dingo:~/xorg/xserver (xi2*+)>


Where xi2 is the branch name, * shows I have changes not added to the index, + shows I have changes added to the index that will be committed. I cannot recommend this prompt for the kernel though, it takes to long. For the repos I work with it's fine.

A word about backups


Since most of the work is local, it's important to have a backup, especially for devel branches that live longer than a day or two. I added a "backup" remote and force-push the branch I've been working on at the end of each day to this remote. In the worst case, I can just clone from there and resume where I left off.

$> git remote add backup user@host:~/repository.git # only do this once
$> git push -f backup branchname

Thursday, April 23, 2009

The big fat X.Org supermodule

I've been using a git supermodule for nearly 2 months now and it works great. So here it is, do with it what you will.

git://people.freedesktop.org/~whot/xorg.git

It includes all the modules I care about to get a server running.

The advantages of git submodule are simply that I have a known working tree I can easily share between my test machines. Whenever I update a component, I can test it and easily revert back to the previous working version if needed.

Advantages for you: if you pull from that tree, you're running a version that at least compiles and runs on my hardware. That doesn't mean it's bug-free of course, but it's a start if you want to get into X server testing.

Disadvantages: I update input stuff often, other stuff when needed. So if you care about the latest and greatest graphics patches, you will need to maintain your own tree. Such is life.

Updating is easy: git pull and git submodule update.

The git submodule interface is missing one important feature to make this workflow better (automatic rebasing), but that'll hopefully be fixed in a future git version.

Monday, April 20, 2009

git-format-patch for a single commit

One thing that always annoyed me was the weirdness of getting a patch from a single commit somewhen back in the history. Turns out I was just ignorant and reading the man page actually helps, git-format-patch accepts a -<n> option, where <n> is the number of patches you need since the commit (inclusive).

So, because I'm a big fan of tig, add this to your $HOME/.tigrc:

bind generic E !git format-patch -1 %(commit)


Start tig, mark the patch you want, hit "E" to get a nice patch file. Doesn't get much simpler than that.

[update Jun 05 2012]
As SEJeff and Michael point out in the comments
git show $sha
shows a single commit. The output of git show is different to git format-patch though (it cannot be applied via git-am).