git patches from tarballs

Generating patches with git is easy if you clone upstream. Many users don't run their software from repositories. They work from either distribution packages or tarballs. So a number of times I've heard something along the lines of "Sorry, it's not a git patch because I'm working from the tarball". It is quite easy though to create git patches from your tarballs. Simply run the following command in the extracted directory:

git init && git add --ignore-errors .; git commit -m "`basename $PWD`"

Explanation:

• "git init" initialises a new git repository.


• "git add ." adds all existing files to the repo. The --ignore-errors is there so git skips over files that can't be added, I've had that happen in a few tarballs that had their permissions busted, etc. If there are errors, you obviously need to check whether they affect files you want to hack on. If not, ignore the errors. (btw. you want to run git add before compiling everything, having all object files in the git index is painful)


• "git commit" commits all newly added files with the name of the current directory as commit message.

Now the directory is basically the same as upstream when they released the tarball, without the history. Either way, you can just hack, commit, rebase, etc. and then create a patch with git-format-patch and submit it to upstream. Assuming that upstream hasn't diverged too much from the tarball, chances are the maintainers can just apply the patch as-is.

Disclaimer: I learned this workflow from the Fedora X11 packages where this method is used to apply upstream patches to the tarballs. So the credit goes to ajax (or maybe someone else).

[edits]
"git init-db" replaced with "git init".

Button mapping in X

X differs between device buttons and logical buttons. The former are handled in the device/driver, the latter are what applies to clients. By default, these are mapped in a 1:1 fashion, with device button 1 (D1) being mapped to logical button 1 (L1), D2 to L2, D3 to L3, etc.

Logical buttons are what the client sees in the end. The first logical button is L1 and there are some standard behaviours:
L1 .. left click
L2 .. middle click
L3 .. right click
L4 .. vert scroll wheel up
L5 .. vert scroll wheel down
L6 .. horiz scroll wheel left
L7 .. horiz scroll wheel right

These are de-facto standard in virtually all clients, with middle click usually resulting in a paste from the current selection. Additionally, some clients (e.g. firefox) map L8 and L9 to back/forward.
All these standard actions are by convention only, there's no defined standard (that I know of) and a client may choose to interpret L4 as left clicks (hint: this is not a smart thing to do).

The meaning of the logical buttons is always the same but the user can re-map device buttons to logical buttons. For example, a left-handed mouse uses a button mapping of D1:L3 and D3:L1 (a mapping of 3 2 1 4 5 6 7 ...). Now, when a button 3 is pressed, the server maps this button to logical button 1 and the client executes a left-click.

In-driver physical button mapping

There's one more stage of button mapping, and that stage happens in the driver. The driver communicates with the server over a defined API and the buttons passed across this API are treated as the device buttons. The driver may chose to map the physical button to a different device button to accommodate for certain hardware features or deficiencies. This mapping is driver-specific and may not exist for all drivers. Evdev has it mainly to accommodate for broken hardware, and synaptics has it for tap-to-click behaviour.

As an example, an evdev button mapping may ask for physical button 1 (P1) being mapped to device button 3. When P1 is pressed on the device, evdev passes D3 to the server which may then map it to L1 if a right-handed mouse is configured.

This method is more commonly used in synaptics for the tap-to-click behaviour. This is a purely in-driver emulation of a button, so the TapButton1/2/3 settings define what device button will be sent on a one/two/three finger tap. By default, TapButton1 is mapped to D1, but one could set it to D4 and scroll by tapping. This mapping is handy for the right-handed synaptics quirk described later.

The button map chain

Starting with server 1.6, devices have a chain of button mappings. Since both the physical device and the core pointer show up as separate devices (run xinput --list -short), they may have different button mappings. Virtually all clients only see the core pointer's events and any device that controls the core pointer is subject to the core pointer's mappings. The button mapping for physical devices that control the core pointer is thus as follows:

physical to device mapping → device to logical mapping → core pointer device to logical → client.

No contemporary configuration tool should change the core pointer's mapping. These tools should be XI aware and modify the configuration for each device.

In the following examples, '→' is used for a standard X:X mapping, and '»' for a X:Y button mapping. Assume Mouse M that controls the core pointer CP. Both have the standard 1 2 3 mapping with no in-driver mapping. The path is thus:

M(P1) → M(D1) → M(L1) → CP(D1) → CP(L1) → client.

If a 3 2 1 left-handed mapping on the core pointer is set up, the path is:

M(P1) → M(D1) → M(L1) → CP(D1) » CP(L3) → client.

Thus, all devices that send events through the core pointer are now left-handed.

On the other hand, if only a single device needs to be left-handed, the core pointer maintains it 1 2 3 mapping and the mouse M gets a 3 2 1 mapping:

M(P1) → M(D1) » M(L3) → CP(D3) → CP(L3) → client.

This is the setup configuration tools should choose if the user requires a left-handed mouse.

If a the same mapping is applied to both CP and M, then this mapping is neutralised:

M(P1) → M(D1) » M(L3) → CP(D3) » CP(L1) → client.

Although both M and CP have a left-handed setup, M effectively acts as a right-handed mouse. Even worse, an application that listens for XI events (e.g. the gimp) would see M as a left-handed mouse while all other applications would see it as a right-handed one. This is the reason why the core pointer mapping should be left alone.

Now, for fun, let's also assume that M has a hardware problem and physical button 4 is actually the left mouse button. So an in-driver mapping from 4 to 1 is needed.

M(P4) » M(D1) » M(L3) → CP(D3) » CP(L1) → client.

These examples above are the most common scenarios the average user may encounter.

Scrollwheel and buttons

As mentioned above, scrollwheels are mapped to the logical buttons 4,5 and 6,7. This has historical reasons, the core protocol only allows for two axes and by the time scrollwheels came about, the protocol was already set in stone. So now they're mapped in-driver to logical buttons instead and most applications and toolkits interpret it correctly. This is not ideal, as mapping wheel events to button events looses information (such as missing out on pointer acceleration for smoother scrolling).

The right-handed synaptics quirk

One common problem is how to configure a right-handed synaptics pad while leaving tap-to-click behaviour in the default configuration. As mentioned above, tap-to-click is an in-driver mapping. Right-handed behaviour is (usually) a global mapping applied by the desktop environment and may even be applied to the core pointer (especially with older, non-XI aware configuration tools). The easiest solution is to map tapping in-driver to an unused button number and then map this button back to the logical buttons as desired. For example, TapButton1 on synaptics device S could be mapped to 10, and D10 mapped to L1.

tap 1 » S(D10) → S(L10) → CP(D10) » CP(L1) → client

Thus, even if S is set up with a 3 2 1 mapping for the first three buttons, tapping still works in a "right-handed" fashion.

Note that in order to map two device buttons to the same logical button, X Input 1.5 is needed (server 1.6).

The tools

For a device button mapping:
XSetDeviceButtonMapping(3)
xinput --set-button-map 1 2 3 4 5 ...

For the core pointer mapping:
XSetPointerMapping(3)
xmodmap -e "pointer = 1 2 3"

For the synaptics mapping:
synclient TapButton1=1

For an evdev button mapping:
Option "ButtonMapping" "1 2 3 ..."


[edits]
mclasen pointed out that the middle button is "primary paste", not "clipboard paste". Text amended to be neutral regarding the paste type.

XI2 merge coming up

I've sent a pull warning to xorg-devel, next week at this time XI2 should be part of master.

My workflow

After posting about the X.Org supermodule I was asked for more details about how to get everything up and running. So here's my usual workflow. I'm open for improvements in the (moderated) comments.

Prerequisites

My workflow currently includes two patches to git, the second of which is still under review.
Install git from git and apply these patches. They add rebasing support to git submodule update and make the process a lot easier.

Set up a bunch of environment variables that you'll need each time you build:

$> export PKG_CONFIG_PATH=/opt/xorg/lib/pkgconfig
$> export PATH=/opt/xorg/bin:$PATH
$> export LD_LIBRARY_PATH=/opt/xorg/lib
$> export ACLOCAL="aclocal -I /opt/xorg/share/aclocal"

and some more that are just handy:

$> export CFLAGS="-Wall -O0 -ggdb"
$> export CC="ccache gcc"
$> export MAKEFLAGS="-j3"

Building X.Org from git in 5 steps

$> git clone git://people.freedesktop.org/~whot/xorg.git
$> cd xorg
$> git submodule init
$> git submodule update
$> ./util/modular/build.sh -f built.modules /opt/xorg

The supermodule is set up for automatic rebasing and you'll end up with a tree running each module on master. The last command builds everything in the right order and - with the "-f" flag - echos the modules being built into the built.modules file. If it fails (usually due to missing packages) you can resume from the last to-be-built component (the last one in built.modules).

$> ./util/modular/build.sh -f built.modules -r `tail -n 1 built.modules` /opt/xorg

Once that is done, you're left with an X tree in /opt/xorg, most important of which is the binary in /opt/xorg/bin/Xorg.

Working with the tree

I tend to have three or more branches in most repos. The branches that matter are "master", "queue" and "devel". Then I have additional branches for features that result in a patch series (e.g. "xi2").

"master" is always as close to upstream master as I can get. Anything that lands on master will likely be rebased and pushed upstream. Day-to-day bugfixing also happens on master.

"queue" is for patches I sent to the xorg-devel list. The workflow here is usally development on some other branch, then git-format-patch + email, then cherry-picking from the other branch to queue. Patches in "queue" get cherry-picked to master and pushed, and once master is pushed, "queue" is rebased onto master.

"devel" usually happens when I realize that the patch series on master is more than it should be. This is when I branch master into devel, reset master to the previous state and continue on devel. devel is heavily rebased and sometimes doesn't lead anywhere. If it does, intermediate patches are cherry-picked onto master and pushed when they're ready. devel is deleted as soon as I finish with it.

Feature branches (e.g. "xi2") tend to be the same as devel but with a specific feature in mind. Anything that isn't related to it (bugs that I find in code around that feature) is cherry-picked to master and queue, and the feature branch then rebased.

So a single patch may wander from xi2 to devel (when xi2 is branched for some reason) to queue to master before being pushed.

I've been using this workflow for months now, and one of the main reasons why it works fine for me is tig. Tig shows other branches heads and tags in the history list, so by rebasing often I always have a visual marker where the new patches start. Tig also makes cherry-picking easy, so I use it more often than git pull.

The other thing that is incredibly helpful is the zsh git prompt I got from here and modified a bit. My current output reads:

:: whot@dingo:~/xorg/xserver (xi2*+)> 

Where xi2 is the branch name, * shows I have changes not added to the index, + shows I have changes added to the index that will be committed. I cannot recommend this prompt for the kernel though, it takes to long. For the repos I work with it's fine.

A word about backups

Since most of the work is local, it's important to have a backup, especially for devel branches that live longer than a day or two. I added a "backup" remote and force-push the branch I've been working on at the end of each day to this remote. In the worst case, I can just clone from there and resume where I left off.

$> git remote add backup user@host:~/repository.git # only do this once
$> git push -f backup branchname

The big fat X.Org supermodule

I've been using a git supermodule for nearly 2 months now and it works great. So here it is, do with it what you will.

git://people.freedesktop.org/~whot/xorg.git

It includes all the modules I care about to get a server running.

The advantages of git submodule are simply that I have a known working tree I can easily share between my test machines. Whenever I update a component, I can test it and easily revert back to the previous working version if needed.

Advantages for you: if you pull from that tree, you're running a version that at least compiles and runs on my hardware. That doesn't mean it's bug-free of course, but it's a start if you want to get into X server testing.

Disadvantages: I update input stuff often, other stuff when needed. So if you care about the latest and greatest graphics patches, you will need to maintain your own tree. Such is life.

Updating is easy: git pull and git submodule update.

The git submodule interface is missing one important feature to make this workflow better (automatic rebasing), but that'll hopefully be fixed in a future git version.

git-format-patch for a single commit

One thing that always annoyed me was the weirdness of getting a patch from a single commit somewhen back in the history. Turns out I was just ignorant and reading the man page actually helps, git-format-patch accepts a -<n> option, where <n> is the number of patches you need since the commit (inclusive).

So, because I'm a big fan of tig, add this to your $HOME/.tigrc:

bind generic E !git format-patch -1 %(commit)

Start tig, mark the patch you want, hit "E" to get a nice patch file. Doesn't get much simpler than that.

[update Jun 05 2012]
As SEJeff and Michael point out in the comments

git show $sha

shows a single commit. The output of git show is different to git format-patch though (it cannot be applied via git-am).

Zapping the server

Some amount of bad blood was spilled last week (read LWN's writeup for a better summary than I'll provide here).

In short, zapping through Control-Alt-Backspace was disabled and some people are not happy. It is a useful combination and I (need to) use it regularly. Should it be enabled by default? No.

Let's think about this: Assume that the server never supported zapping in the past. Now we add a feature that immediately and without asking terminates your session, shuts down all applications, logs you out, brings down your wireless network in the process, shuts down your VPN and generally makes the computer giggle at you. Of course, we documented this shortcut in the Xorg man page.

I guess we would see the same flame-wars. Potentially even by the same people.

The correct thing to do for a UI feature like this - like with any UI feature - is to consider who will benefit from such a change and make it accessible to those. Experienced users will benefit from zapping. So make it accessible to them. Those who want to use the computer but not have to know about it's internals should not be able to accidentally trigger it. Not the other way round.

Anyway. The correct solution was rather easy, and thankfully, Julien Cristau has written the patch for xkeyboard-config and there's a patch for the X server too. Which, amongst from other things, shows that time spent reading and replying to flame-wars is spent less usefully as time spent writing patches.

The following command enables server zapping.

setxkbmap -option "terminate:ctrl_alt_bksp"

I'll leave it as an exercise to the reader to figure out how to click the matching checkbox in the keyboard configuration tools.

So for the future: If you are unhappy with a particular change, take the issue to the appropriate mailing list and/or persons, in polite tone and based on real arguments (made-up facts only count when you're on TV) and it may just get fixed. It's pretty much a no-brainer, but maybe even such a minimum requirement is too high. In the end, Julien's patch was the reason why the whole issue got sorted out, not the flame-wars.

Rawhide users:
There's a dependency chain in keyboard configuration that made me weep inside a little. Update to fedora-setup-keyboard-0.3-4, xkeyboard-config-1.5-5 and xorg-x11-server-1.6.0-18, then restart HAL and restart X. Zapping will work on a plain X server and in gdm, but only if enabled in the gnome session.

[update]
Option DontZap is restored with this change to it's previous functionality - it disables zapping altogether (if enabled). Even if DontZap is disabled (the default), zapping only works if it is present in the xkb map.