ClickPad support in the synaptics driver

[update 20 Apr: clarify middle finger emulation]
One of the features added to the upcoming xf86-input-synaptics version 1.6 is support for clickpad-style devices. This post outlines what clickpads are, how they are supported and how you go about enabling the new features.

What are clickpads

The name ClickPad comes from the Synaptics product of the same name. It describes a touchpad that does not have physical buttons. Instead, the whole touchpad works as a button. Devices like this have been around for a while, most notably the touchpads found in Apple laptops and the Dell Mini 10 series. The challenges for us were to handle the data streams correctly. Most of this work was done by Chase Douglas.

Clickpads give us the position for each finger, and a button 1 event when the pad is pressed. The design of the hardware however means that whenever you click the pad, you always add one more finger to the mix. Decoupling that finger from the ones that actually matter is the challenge. And integrating it with all the other whacky features that the synaptics driver currently has.

Clickpad Support

Clickpad support requires server 1.12 [0]. It heavily relies on multitouch support.
Central to the new feature is the "ClickPad" option and property. It is enabled if the kernel sets the INPUT_PROP_BUTTONPAD property on the device, otherwise you can enable it in an xorg.conf.d snippet or at runtime with either xinput or synclient:

xinput set-prop "device name" "Synaptics ClickPad" 1
synclient ClickPad=1

This toggles a few driver behaviours to make the clickpad much more usable. Most notably, you can use one finger to press the button and another finger to move the cursor around.

Word of warning here: if you enable clickpad support manually at runtime, you will also have to manually disable traditional middle mouse button emulation (synclient EmulateMidButtonTime=0). For autoconfigured devices or xorg.conf.d configured devices this is done automatically.

The second big feature for clickpads is support for software buttons. Since the device only gives us left button clicks, we expose an option to allow for right and middle button clicks. By default, we ship this xorg.conf.d snippet:

Section "InputClass"
        Identifier "Default clickpad buttons"
        MatchDriver "synaptics"
        Option "SoftButtonAreas" "50% 0 82% 0 0 0 0 0"
EndSection

The order of soft button edges is left, right, top, bottom for the right button, then left, right, top, bottom for the middle button. So the above snippet sets the right half of the bottom 18% to be a right button, with no middle button present. A coordinate of 0 means "stretch to edge of touchpad". [1]

Traditional middle mouse button emulation is disabled by default and we don't handle it for clickpads. Traditional middle mouse button emulation refers to the method of generating a button 2 press when both button 1 and button 3 are pressed simultaneously. Mostly because, well, I'm not sure how you would even trigger middle mouse button emulation for clickpads given that all buttons except the first are already emulated anyway. You can still emulate middle mouse button emulations through clickfingers (see below), tapping, or the soft button areas as described above.

Tapping works as on any other touchpad.

ClickFinger functionality works, albeit slightly different. In the default ClickFinger setting, if you have 2 fingers on the touchpad and you press the left button, you will see a ClickFinger2 action performed (right button by default). On clickpads we guess the number of fingers by proximity (since you need one finger to actually press the button). Fingers closer together than 30% of the touchpad's size [2] will count towards ClickFinger actions, others will be skipped. So in the default use-case, where you have two fingers down on the touchpad in the middle and you use a third to click the button, you will still get a ClickFinger2 action. Likewise, if you press with two fingers down, you will also get a ClickFinger2 action.

All other functions are not affected by clickpad support and behave as before.

• [0] Ubuntu 12.04 will ship a 1.11/1.12 mix, that will work as well
• [1] "edge of touchpads" should be 100% but it isn't due to a years-old mishandling of the coordinate ranges. So 0 is better to use here.
• [2] All touchpads lie about their size, so we just hardcode 30% since anything we infer is going to be wrong anyways

New layout template

Short summary: layout of the blog changed. Long summary: One of the issues I ran into repeatedly was that any code postings would overrun the rather narrow column width. Now I've done the lazy thing and applied a new standard template that appears to be wider. So in the future, code snippets are hopefully easier to read.

Chrome's multitask mode - an April fool's 2 years too late?

Ok, I was away on holidays for the last week so I missed everything. Thanks to Simon Thum, I found the Chrome team's April fools joke: Chrome's Multitask Mode. Allegedly a mode for Chrome that allows to use multiple pointers simultaneously.

Great, except that it's just an April fool's joke, they didn't actually implement it. Even though, well, it's been a standard feature in the X.Org X server since September 2009. Yep, right. Fedora 12 had it, for example, and every version since. It's ubiquitous on the GNU/Linux desktop. GTK3 has support for it.

Being able to use two hands is quite useful, research papers on the benefits go back well over 20 years. Use cases are generally as dominant hand/non-dominant hand methods (e.g. an artist holding the palette with one hand while painting with the other one) or as equal bimanual interaction (aligning a rectangular mask around an object). All this isn't new, and as I said above this is all a standard feature in X since 2009. You really just [1] need to add it in your application. So Chrome's April fool's joke is pretty much a joke about not implementing a feature. Which, well, uhm... ok. Haha.

The video is a bit outlandish, with a guy playing golf and a shooter simultaneously. However, listening to just the audio on the video largely makes sense (except the switching off your computer part). Using two mice became natural for me quite quickly. I even conducted a user-study about users using two browser windows simultaneously to research and gather information about a specific topic. Yep, they could do it, including typing up the results in a decidedly single-user Abiword without major conflicts.

Now I'm waiting for next year's joke, maybe it's about how we drive cars with, wait for it, steering wheels.

[1] I say "just", but implementing it is really hard. It opens a can of worms about assumptions in user interface paradigms that aren't quite that simple to deal with. From a technical point of view, it's a "just" though...

ssh and shared connections

My job requires me to ssh into various boxes often, with several connections to the target host. Some people use screen on the target host but I work better if I have multiple terminal windows. But re-connecting to the same host can be annoying, a connection does take time and it should be instantaneous. Luckily, SSHv2 allows to share a connection, making reconnection a lot faster. Also, if you have password-authenticated connection instead of a key-based one you won't have to type the password for each new connection (but really, you should be using keys anyway). The few lines you'll need in your $HOME/.ssh/config:

ControlMaster auto
ControlPath ~/.ssh/sockets/ssh_mux_%h_%p_%r
ControlPersist 60

All three are extensively described in the ssh_config(5) man page, but here's a summary:

• ControlMaster auto will create a new ssh connection when no matching socket exists. Otherwise, it will just use the existing connection.
• ControlPath is simply the path to the control socket, with %h, %p and %r being replaced with target host, port and username to keep the socket name unique. Having this in a user-specific location instead of /tmp is generally a good idea.
• ControlPersist defines how long the master connection should be kept open after exit. You can specify "yes" for indefinite or a number of idle seconds. If you reconnect within that idle time, it will again re-use the existing connection. Note that if you do not have ControlPersist and you quit the master connection, you will terminate all other connections too! ControlPersist was added in OpenSSH 5.6.

You can provide these options globally or inside a Host section of the config, depending on your needs. A few final notes: since you essentially only have one connection now, you can only forward one X11 display, one ssh agent, etc. at a time. If you need a separate connection for a otherwise shared host, use "ssh -S none". Also, if you're doing heavy data transfer on laggy connections you're probably better off having separate connections.

Multitouch in X - Multitouch-touchpads

This post is part of a series on multi-touch support in the X.Org X server.

1. Multitouch in X - Getting events
2. Multitouch in X - Pointer emulation
3. Multitouch in X - Touch grab handling
4. Multitouch in X - Multitouch-touchpads

In this post, I'll outline the handling of touch events for multitouch-capable touchpads. Multi-touch touchpads that are supported are those that provide position information for more than one finger. The current version of the synaptics X driver does some tricks to pretend two-finger interaction on single-finger touchpads - such devices are not applicable here.

Touchpads are primarily pointer devices and any multi-touch interaction is usually a gesture. In the protocol, such devices are of the type XIDependentDevice and the server does adjust touch event delivery. I've already hinted at this here but this time I'll give a more detailed explanation.

Touch event delivery

Unlike for direct touch devices such as touchscreens, dependent devices have a different picking mechanism in the server. We assume that all gestures are semantically associated with the cursor position. For example, for scrolling, you would move the cursor on top of the window to be scrolled, then you would start scrolling. The server thus adjusts event delivery accordingly. Whereas for direct touch devices the touch events are delivered to whichever window is at the position of the touch, touch events from dependent devices are always delivered to the window underneath the pointer (grab semantics are adjusted to follow the same rules). So if you start a gesture in the top-left corner of the touchpad, the window underneath the cursor gets the events with the top-left coordinates. Note that the event and root coordinates always reflect the pointer position.

The average multi-touch touchpad has two modes of operation: single-finger operation where the purpose is to move the visible cursor and multi-finger operation which is usually interpreted into a gesture on the client. These two modes are important, as they too affect event delivery. The protocol specifies that any interaction with the device that serves to move the visible cursor only should not generate touch events, and that touch events will start once that interaction becomes a true multi-touch interaction. This leaves the drivers a little bit of leeway, but the current implementation in the synaptics driver is the following:

1. A user places one finger on the touchpad and moves. The client will receive regular pointer events.
2. The user places a second finger on the touchpad. The client will now receive a TouchBegin event for the first and the second touch, at their respective current positions in device coordinate range.
3. Movement of either finger now will generate touch events, but no pointer events.
4. Any other fingers will generate touch events only.
5. When one of two remaining fingers on the touchpoint ceases the touch, a TouchEnd is sent for both the terminating touch and the remaining touch. The remaining finger will revert to sending pointer events.

Legacy in-driver gestures

As you are likely aware, the synaptics driver currently supports several pseudo gestures such as tap-to-click or two-finger scrolling. These gestures are interpreted in the driver, thus the server and client never see the raw data for them.

With proper multi-touch support these gestures are now somewhat misplaced. On the one hand, we want the clients to interpret multitouch, on the other hand we want the gestures to be handled in the same manner in all applications. (Coincidentally, this is also a problem that we need to solve for Wayland).

We toyed with ideas of marking emulated events so clients can filter but since we do need to be compatible to the core and XI 1.x behaviours, we only found one solution: any in-driver gestures that alter event deliver must not generate touch events. Thus, if the driver is set to handle two-finger scrolling, the clients will only see the pointer events and scroll events, they will not see touch events from two-fingers. To get two-finger scrolling handled by the client, the in-driver gesture must be disabled. The obvious side-effect of that is that you then cannot scroll in applications that don't support the gestures. Oh well, it's the price we have to pay for having integrated gesture support in the wrong place.

XKB breaking grabs - CVE-2012-0064

Given that there is a copious amount of misinformation being spread, here is a summary of CVE-2012-0064, straight from a horse's mouth.

Outline of the issue

The bug allows users to work around screen locking (e.g. gnome-screensaver) by hitting Control+Alt+keypad multiply or Control+Alt+keypad divide. This terminates the input grab the screensaver has and thus allows a user to interact with the desktop, skipping the password entry.

Affected versions

Affected is anyone running X server 1.11 or later (or release candidates thereof). So if "Xorg -version" shows something else on your box, stop worrying. I doubt any distribution would have back-ported the patches.

In Fedora/Red Hat land - the only distributions affected are Fedora 16 and current Fedora Rawhide. Both have been fixed, the F16 update is avaialble here. Note that the update is to xkeyboard-config, not to the server itself.

Fedora 15 is not affected. RHEL 4, 5, 6 and thus CentOS 4, 5, 6 and other derivatives are not affected. I believe that most other distributions have now pushed out updates as well, if you want to link to the respective updates, please do so in the comments.

Sergey has also pushed out xkeyboard-config 2.5 today with the fix included.

History of the feature

The X protocol does not allow the server to break grabs. Once a client has a grab, the server must wait for that client to release the grab, terminate, or the grab window to become unviewable. This is an issue when debugging applications - if your client has a keyboard grab, you cannot use the debugger since all key events will go to the client being debugged. To avoid this issue, the X server has had two combinations to break grabs: Control+Alt+Keypad multiply and Control+Alt+Keypad divide. They forced grab termination inside the X server and although against the protocol it made debugging possible. The option required explicit enabling in the xorg.conf.

These options were removed in server 1.4 and disabled since. Which made debugging hard, so last year we merged a patch to bring them back, together with some other features. They are triggered by XKB actions (as they used to be). The plan was to remove the XKB actions from the default keymap so that the action is available on request but not enabled by default. This is where a miscommunication happened, the removal from the default keymap never happened. So server 1.11 and vanilla xkeyboard-config ship with both the actions available and present in the current keymap. As a result, any user can break a grab from any application and thus get around screen locking.

Outline of the fix

To shoot yourself in the foot, you need two items: a gun and a trigger. We have removed the trigger. The fix we've now pushed into xkeyboard-config removes the actions from the default keymap and into an XKB option instead. So the fix does not remove the gun, but it requires the user to screw the trigger in themselves before trying to hurt themselves. In a default configuration, it is thus no longer possible to break the grab of your screensaver.

To re-enable grab debugging, run setxkbmap with "-option grab:break_actions" or enable "Allow breaking grabs with keyboard actions (warning: security risk)" in the "Miscellaneous compatibility options" in your keyboard layout configuration tool of choice.

Multitouch in X - Touch grab handling

This post is part of a series on multi-touch support in the X.Org X server.

1. Multitouch in X - Getting events
2. Multitouch in X - Pointer emulation
3. Multitouch in X - Touch grab handling
4. Multitouch in X - Multitouch-touchpads

In this post, I'll outline how grabs on touch events work. This post assumes basic knowledge of the XI2 Xlib interfaces.

Passive grabs

The libXi interface has one new passive grab call: XIGrabTouchBegin, which works pretty much like the existing passive grab APIs. As with event selection, you must set all three event masks XI_TouchBegin, XI_TouchUpdate and XI_TouchEnd or a BadValue error occurs. Once a passive grab activates in response to a touch, the client must choose to either accept or reject a touch. Details on that below.

Grabs activate on a TouchBegin event and due to the nature of multitouch, multiple touch grabs may be active at any time - some of them for different clients.

Active grabs

Active grabs do not have a new call, they are handled through the event masks of the existing XIGrabDevice(3) call. If a client has an active touch grab on the device, it is automatically the owner of the touch sequence (ownership is described below). If a client has an active pointer or keyboard grab on the device, it is the owner of the touch sequence for pointer emulated touch events only. Other touch events are unaffected by the grab and are processed normally.

Acceptance and rejection

Pointer grabs provide exclusive access to the device, but to some degree a client can opt to replay the event it received on the next client. We expect that touch sequences will often trigger gesture recognition, and a client may realise after a few events that it doesn't actually want that touch sequence. So we expanded the replay semantics. clients with a touch grab must choose to either accept or reject a touch.

Accepting a touch signals to the server that the touch sequence is meant for this client and no-one else. The server then exclusively delivers to that client until the terminating TouchEnd.

Rejecting a touch sequence signals that the touch sequence is not meant for this client. Once a client rejects a touch sequence, the server sends the TouchEnd event to that client (if the touch is still active) and replays the full touch sequence [1] on the next grab or window. We use the term owner of a touch sequence to talk about the current recipient.

The order of events for two clients Cg and Cw, with Cg having a grab and Cw having a regular touch event selection on a window, is thus:

TouchBegin to Cg    → 
TouchUpdate to Cg   → 
TouchUpdate to Cg   → 
                    ← Cg rejects touch
                    ← Cw becomes new owner
TouchEnd+ to Cg     →
TouchBegin* to Cw   → 
TouchUpdate* to Cw  → 
TouchUpdate* to Cw  → 
#### physical touch ends #### 
TouchEnd to Cw      →

Events with + mark an event created by the server, * mark events replayed by the server

For nested grabs, this sequence simply repeats for each client until either a grabbing client accepts the touch or the client with the event selection becomes the owner.

In the above case, the touch ended after Cg rejected the touch. If the touch ends before the current owner accepted or rejected it, the owner gets the TouchEnd event and the touch is left handing until the owner accepts or rejects it. If accepted, that's it. If rejected, the new owner gets the full sequence in one go, including the TouchEnd event. The sequence is thus:

TouchBegin to Cg    → 
TouchUpdate to Cg   → 
TouchUpdate to Cg   → 
#### physical touch ends #### 
TouchEnd to Cg      →
                    ← Cg rejects touch
                    ← Cw becomes new owner
TouchBegin* to Cw   → 
TouchUpdate* to Cw  → 
TouchUpdate* to Cw  → 
TouchEnd* to Cw     →

Touch ownership handling

One additional event type that XI 2.2 introduces is the XI_TouchOwnership event. Clients selecting for this event signal that they need to receive touch events before they're the owner of the touch sequence. This event type can be selected on both grabs and event selections.

First up: there are specific use-cases where you need this. If you don't fall into them, you're better off just skipping on ownership events, they make everything more complicated. And whether you need ownership events depends not only on you, but also the stack you're running under. On normal touch event selection, touch events are only delivered to the current owner of the touch. With multiple grabs, the delivery is sequential and delivery of touch events may be delayed.

Clients selecting for touch ownership events get the events as they occur, even if they are not the current owner. The XI_TouchOwnership event is delivered if and when they become the current owner. The last part is important: if you select for ownership events, you may receive touch events but you may not become the owner of that sequence. So while you can start reacting to that sequence, anything your app does must be undo-able in case the e.g. window manager claims the touch sequence.

If we look at the same sequence as above with two clients selecting for ownership, the sequence looks like this:

TouchBegin to Cg     → 
TouchBegin to Cw     → 
TouchOwnership to Cg →
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
                     ← Cg rejects touch
                     ← Cw becomes new owner
TouchEnd+ to Cg      →
TouchOwnership to Cw →
#### physical touch ends #### 
TouchEnd to Cw      →

Note: TouchOwnership events do not correspond to any physical event, they are always generated by the server

If a touch ends before the owner accepts, the current owner gets the TouchEnd, all others get a TouchUpdate event instead. That TouchUpdate has a flag XITouchPendingEnd set, signalling that no more actual events will arrive from this touch but the touch is still waiting for owner acceptance.

TouchBegin to Cg     → 
TouchBegin to Cw     → 
TouchOwnership to Cg →
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
#### physical touch ends #### 
TouchEnd to Cg       →
TouchUpdate to Cw    →  (XITouchPendingEnd flag set)
                     ← Cg rejects touch
                     ← Cw becomes new owner
TouchOwnership to Cw →
TouchEnd to Cw       →

In both cases, we dealt with a rejecting owner. For an accepting owner, the sequences look like this:

TouchBegin to Cg     → 
TouchBegin to Cw     → 
TouchOwnership to Cg →
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
                     ← Cg accepts touch
TouchEnd+ to Cw      →
TouchUpdate to Cg    → 
#### physical touch ends #### 
TouchEnd to Cg      →

or

TouchBegin to Cg     → 
TouchBegin to Cw     → 
TouchOwnership to Cg →
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
#### physical touch ends #### 
TouchEnd to Cg       →
TouchUpdate to Cw    →  (XITouchPendingEnd flag set)
                     ← Cg accepts touch
TouchEnd* to Cw      →

In the case of multiple grabs, the same strategy applies in order of grab activation. Ownership events may be selected by some clients but not others. In that case, each client is treated as requested, so the event sequence the server deals with may actually look like this:

TouchBegin to C1     → 
TouchBegin to C3     → 
TouchOwnership to C1 →
TouchUpdate to C1    → 
TouchUpdate to C3    → 
TouchUpdate to C1    → 
TouchUpdate to C3    → 
                     ← C1 rejects touch
                     ← C2 becomes new owner
TouchEnd+ to C1      →
TouchBegin* to C2    → 
TouchUpdate* to C2   → 
TouchUpdate* to C2   → 
                     ← C2 rejects touch
                     ← C3 becomes new owner
TouchEnd+ to C2      →
TouchOwnership to C3 →
#### physical touch ends #### 
TouchEnd to C3       →

[1] obviously we need to store these events so "full sequence" really means all events until the buffer was full