There are many good reasons for bugs not getting addressed. One of them is the design argument ("this system isn't designed for this feature") or the dependency argument ("this bug is in this library we link against"). Both are valid explanations for why a software project does not address a bug directly. And of course the ever-present reasons "lack of time" and "not enough manpower". However, quite often the defence for a bug in some software is along the lines of "Windows can't do this either".
If you explain a deficiency in your software through the deficiencies of competing software, you still have a problem.
I don't care whether other implementations are buggy when I'm using some software. I care about what I'm using because there's a compelling reason why I chose this one. And if it doesn't do the job, it needs to be fixed.
Always strive for your software to be excellent and look to other software for inspiration, not excuses.
Tuesday, January 18, 2011
Monday, November 22, 2010
A High-Level Overview of Grabs
One of the most common topics coming up in X input is grabs. This is a high-level overview of input device grabs and how they affect input event processing. Note that the examples here are simplified, grabs are probably the most complicated topic in input processing.
By default, X input events are delivered to all clients that registered for a specific event on a window. The basic premise of a grab is that it affects event delivery to deliver events exclusively to one client only.
There are three types of grabs, two classes of grabs, and two modes for a grab. The three types are:
The two classes are core grabs and device grabs and the two modes are synchronous and asynchronous.
A grab comes in a combination of type + class + mode, so a grab may be an "active synchronous device grab".
Let's look at the difference between types first.
An active grab is a direct result of a client request. The client requests "Grab this device and deliver its events exclusively to me". Future events are then delivered to this client until the client "ungrabs" the device again.
From a UI point-of-view, active grabs are commonly used when events must go to one specific application, regardless of the pointer or keyboard focus. One example are popup menus, where you want the next click to either activate a field or remove the window when clicking outside of the popup menu.
A passive grab is a promise by the X server to grab the device next time a button or key is pressed. The client essentially requests "grab the button when it is on window W and Alt and Shift are down". This request is stored and in the future when the pointer moves into window W, the specified modifiers are down and the user presses the button, the passive grab is activated (so now we have an "activated passive grab") and events are delivered exclusively to the client that requested the passive grab. Once the button or key is released, the passive grab is released again (and events are sent to all applicable clients). Unlike active grabs, passive grabs will activate repeatedly whenever the conditions are met. Ungrabbing a passive grab means to remove the conditions that trigger the activation of a passive grab.
XI2 added enter/leave and focus in/out passive grabs in addition to the already common button and key grabs. So a passive grab may activate when a pointer enters the window now.
These are a special type of passive grabs that are not under the direct influence of clients. Whenever a button press event is delivered normally to a client (i.e. no passive grab activated), this client gets an implicit passive grab on the device to ensure that future events, most notably the release event, is also delivered to the same client. Implicit passive grabs only come in synchronous mode, more on that later.
In the core X protocol, only pointer and keyboard grabs are defined. With the addition of the X Input Extension, the device grabs were added as well. The only real difference is that with the latter a client explicitly specifies which device to grab. In the case of a core grab, the client simply says "grab the pointer" or "grab the keyboard", in the case of a device grab, the client must specify a valid device to grab. The class of a grab affects the events being sent to the client. A client with a core grab cannot receive X Input events from this device and vice versa.
With the addition of MPX and XI2 to the server, core grabs now work according to the ClientPointer principle. Since one of the requirements for MPX/XI2 was to not interfere with core applications, the behaviour of the two classes of grabs differs slightly.
The basic assumption here is that core applications do not know about the existence of multiple devices, but XI-aware applications do. Other than that, the two classes behave mostly identical.
Grabs, with the exception of implicit passive grabs, may come as synchronous or asynchronous grabs. Asynchronous are simple: the event stream is processed by the X server and sent to the grabbing client immediately. During synchronous grabs, the server will queue up events internally and ask the client how to continue for each event. On a synchronous passive button grab for example, the client will receive the first event (the button press) but no more. It can then decide what to do and instruct the server how to proceed with the events. One common instruction is to "thaw" the device, i.e. release the event stream and continue in asynchronous mode. The other common instruction is to "replay" the event. This is a common operation for window managers which put a grab on mouse buttons as it allows them to reshuffle windows before the event is seen by an application. The list of actions is essentially this:
If a client requests an active grab, you are at the mercy of this client. If the client decides to not ungrab (for whatever reason, usually a bug), then you will not be able to use the device on any other application. This is what's commonly referred to as "stuck grab". The only way to get out of this at the moment is to find the offending client and kill it. All grabs for a client are automatically terminated when a client exits.
A passive grab may be converted into an active grab. If a client already has a passive grab and requests an active grab on the same device, this active passive grab is converted to an active grab and thus does not terminate automatically anymore. The client then has to send an ungrab request to the server. This is a common operation for popup menus, a client has a synchronised passive grab that is converted into an asynchronous active grab once the passive grab activates. It only works passive → active though, an active grab cannot be converted into a passive grab.
All grabs are based on to a window and the event coordinates in the subsequent events are relative to that window's origin. During a grab, the pointer or keyboard is regarded as inside this window (even when outside the window's boundaries) and thus enter/leave and focus in/out events are sent as if the device had left the previous window. These event will have a flag set to notify other clients that the leave or focus out event was caused by a grab.
If you think all this is confusing, look at the code that implements this :)
Also note that this post skips on a number of details and contains some amount of handwaving and sticking fingers in your ears and going "lalalala".
[edit] fix typo, on synchronous button grabs the client receives only the first event.
By default, X input events are delivered to all clients that registered for a specific event on a window. The basic premise of a grab is that it affects event delivery to deliver events exclusively to one client only.
There are three types of grabs, two classes of grabs, and two modes for a grab. The three types are:
- active grabs
- passive grabs
- implicit passive grab
The two classes are core grabs and device grabs and the two modes are synchronous and asynchronous.
A grab comes in a combination of type + class + mode, so a grab may be an "active synchronous device grab".
Grab Types
Let's look at the difference between types first.
Active Grabs
An active grab is a direct result of a client request. The client requests "Grab this device and deliver its events exclusively to me". Future events are then delivered to this client until the client "ungrabs" the device again.
From a UI point-of-view, active grabs are commonly used when events must go to one specific application, regardless of the pointer or keyboard focus. One example are popup menus, where you want the next click to either activate a field or remove the window when clicking outside of the popup menu.
Passive Grabs
A passive grab is a promise by the X server to grab the device next time a button or key is pressed. The client essentially requests "grab the button when it is on window W and Alt and Shift are down". This request is stored and in the future when the pointer moves into window W, the specified modifiers are down and the user presses the button, the passive grab is activated (so now we have an "activated passive grab") and events are delivered exclusively to the client that requested the passive grab. Once the button or key is released, the passive grab is released again (and events are sent to all applicable clients). Unlike active grabs, passive grabs will activate repeatedly whenever the conditions are met. Ungrabbing a passive grab means to remove the conditions that trigger the activation of a passive grab.
XI2 added enter/leave and focus in/out passive grabs in addition to the already common button and key grabs. So a passive grab may activate when a pointer enters the window now.
Implicit Passive Grabs
These are a special type of passive grabs that are not under the direct influence of clients. Whenever a button press event is delivered normally to a client (i.e. no passive grab activated), this client gets an implicit passive grab on the device to ensure that future events, most notably the release event, is also delivered to the same client. Implicit passive grabs only come in synchronous mode, more on that later.
Grab Classes
In the core X protocol, only pointer and keyboard grabs are defined. With the addition of the X Input Extension, the device grabs were added as well. The only real difference is that with the latter a client explicitly specifies which device to grab. In the case of a core grab, the client simply says "grab the pointer" or "grab the keyboard", in the case of a device grab, the client must specify a valid device to grab. The class of a grab affects the events being sent to the client. A client with a core grab cannot receive X Input events from this device and vice versa.
With the addition of MPX and XI2 to the server, core grabs now work according to the ClientPointer principle. Since one of the requirements for MPX/XI2 was to not interfere with core applications, the behaviour of the two classes of grabs differs slightly.
- A core grab is a promise that the device will only send events to the grabbing client, and that no other device may interact with the client while the grab is active.
- A device grab is a promise that the device will only send events to the grabbing client but other devices may still send events to this client too.
The basic assumption here is that core applications do not know about the existence of multiple devices, but XI-aware applications do. Other than that, the two classes behave mostly identical.
Grab Modes
Grabs, with the exception of implicit passive grabs, may come as synchronous or asynchronous grabs. Asynchronous are simple: the event stream is processed by the X server and sent to the grabbing client immediately. During synchronous grabs, the server will queue up events internally and ask the client how to continue for each event. On a synchronous passive button grab for example, the client will receive the first event (the button press) but no more. It can then decide what to do and instruct the server how to proceed with the events. One common instruction is to "thaw" the device, i.e. release the event stream and continue in asynchronous mode. The other common instruction is to "replay" the event. This is a common operation for window managers which put a grab on mouse buttons as it allows them to reshuffle windows before the event is seen by an application. The list of actions is essentially this:
- Button press occurs on window W.
- Synchronised passive grab for window manager WM activates.
- WM receives button event.
- WM brings window W to the foreground.
- WM requests to "replay" the event.
- X server deactivates passive grab.
- The same button press event is now sent to W and the clients listening for button events on W.
Misc Other Info
If a client requests an active grab, you are at the mercy of this client. If the client decides to not ungrab (for whatever reason, usually a bug), then you will not be able to use the device on any other application. This is what's commonly referred to as "stuck grab". The only way to get out of this at the moment is to find the offending client and kill it. All grabs for a client are automatically terminated when a client exits.
A passive grab may be converted into an active grab. If a client already has a passive grab and requests an active grab on the same device, this active passive grab is converted to an active grab and thus does not terminate automatically anymore. The client then has to send an ungrab request to the server. This is a common operation for popup menus, a client has a synchronised passive grab that is converted into an asynchronous active grab once the passive grab activates. It only works passive → active though, an active grab cannot be converted into a passive grab.
All grabs are based on to a window and the event coordinates in the subsequent events are relative to that window's origin. During a grab, the pointer or keyboard is regarded as inside this window (even when outside the window's boundaries) and thus enter/leave and focus in/out events are sent as if the device had left the previous window. These event will have a flag set to notify other clients that the leave or focus out event was caused by a grab.
If you think all this is confusing, look at the code that implements this :)
Also note that this post skips on a number of details and contains some amount of handwaving and sticking fingers in your ears and going "lalalala".
[edit] fix typo, on synchronous button grabs the client receives only the first event.
Thursday, November 18, 2010
How to ignore configuration errors
Two common errors turning up in the log files are "(EE) ioctl EVIOCGNAME failed: Inappropriate ioctl for device" and "(--) SynPS/2 Synaptics TouchPad: no supported touchpad found".
One example for the former is:
One example for the latter:
In both cases, you'll notice that the device gets added twice. Once with a /dev/input/eventX device path, and once with a /dev/input/mouseX path. The first one succeeds and that results in your device being added to the X server's device list. The second one fails because both evdev and synaptics try to use an ioctl that is only available on event devices. The errors do not indicate a driver bug but a misconfiguration. The local configuration needs to be adjusted to avoid non-event devices being assigned to evdev or synaptics.
Look at the synaptics xorg.conf.d snippet we ship in Fedora:
That last line is important. It only tries to assign the synaptics driver to event devices, not to mouse devices. Once you've added that line (or poked your distro maintainer to do it for you), the errors will go away. Alternatively, you could just ignore the errors and not file a bug for it :)
Note that the synaptics upstream snippet does not contain this line. Synaptics supports more than just Linux so we cannot add this upstream.
One example for the former is:
[ 59.837] (II) config/udev: Adding input device ADS7846 Touchscreen (/dev/input/event3)
[ 59.837] (**) ADS7846 Touchscreen: Applying InputClass "default"
[ 59.841] (**) ADS7846 Touchscreen: Applying InputClass "evdev touchscreen "
[ 59.842] (**) ADS7846 Touchscreen: always reports core events
[ 59.842] (**) ADS7846 Touchscreen: Device: "/dev/input/event3"
[ 59.842] (--) ADS7846 Touchscreen: Found absolute axes
[ 59.842] (--) ADS7846 Touchscreen: Found x and y absolute axes
[ 59.842] (--) ADS7846 Touchscreen: Found absolute touchscreen
[ 59.843] (II) ADS7846 Touchscreen: Configuring as touchscreen
[ 59.843] (**) ADS7846 Touchscreen: YAxisMapping: buttons 4 and 5
[ 59.843] (**) ADS7846 Touchscreen: EmulateWheelButton: 4, EmulateWheelIner0
[ 59.845] (II) XINPUT: Adding extended input device "ADS7846 Touchscreen" ()
[ 59.851] (II) ADS7846 Touchscreen: initialized for absolute axes.
--------------------- divider added for clarity ----------------
[ 59.854] (II) config/udev: Adding input device ADS7846 Touchscreen (/dev/input/mouse0)
[ 59.854] (**) ADS7846 Touchscreen: Applying InputClass "default"
[ 59.855] (**) ADS7846 Touchscreen: always reports core events
[ 59.855] (**) ADS7846 Touchscreen: Device: "/dev/input/mouse0"
[ 59.855] (EE) ioctl EVIOCGNAME failed: Inappropriate ioctl for device
[ 59.856] (II) UnloadModule: "evdev"
[ 59.856] (EE) PreInit returned NULL for "ADS7846 Touchscreen"
One example for the latter:
[ 494.925] (II) config/udev: Adding input device SynPS/2 Synaptics TouchPad (/dev/input/event6)
[ 494.925] (**) SynPS/2 Synaptics TouchPad: Applying InputClass "evdev touchpad catchall"
[ 494.925] (**) SynPS/2 Synaptics TouchPad: Applying InputClass "touchpad catchall"
[ 494.925] (II) LoadModule: "synaptics"
[ 494.927] (II) Loading /usr/lib/xorg/modules/input/synaptics_drv.so
[ 494.927] (II) Module synaptics: vendor="X.Org Foundation"
[ 494.927] compiled for 1.9.0, module version = 1.3.99
[ 494.928] Module class: X.Org XInput Driver
[ 494.928] ABI class: X.Org XInput driver, version 11.0
[ 494.928] (II) Synaptics touchpad driver version 1.3.99
[ 494.928] (**) Option "Device" "/dev/input/event6"
[ 495.282] (--) SynPS/2 Synaptics TouchPad: x-axis range 1472 - 5682
[ 495.282] (--) SynPS/2 Synaptics TouchPad: y-axis range 1408 - 5076
[ 495.282] (--) SynPS/2 Synaptics TouchPad: pressure range 0 - 255
[ 495.282] (--) SynPS/2 Synaptics TouchPad: finger width range 0 - 0
[ 495.283] (--) SynPS/2 Synaptics TouchPad: buttons: left right
[ 495.283] (--) SynPS/2 Synaptics TouchPad: invalid finger width range. defaulting to 0 - 16
[ 495.522] (--) SynPS/2 Synaptics TouchPad: touchpad found
[ 495.522] (**) SynPS/2 Synaptics TouchPad: always reports core events
[ 495.682] (II) XINPUT: Adding extended input device "SynPS/2 Synaptics TouchPad" (type: TOUCHPAD)
[ 495.683] (**) SynPS/2 Synaptics TouchPad: (accel) MinSpeed is now constant deceleration 2.5
[ 495.683] (**) SynPS/2 Synaptics TouchPad: MaxSpeed is now 1.75
[ 495.683] (**) SynPS/2 Synaptics TouchPad: AccelFactor is now 0.036
[ 495.683] (**) SynPS/2 Synaptics TouchPad: (accel) keeping acceleration scheme 1
[ 495.683] (**) SynPS/2 Synaptics TouchPad: (accel) acceleration profile 1
[ 495.683] (**) SynPS/2 Synaptics TouchPad: (accel) acceleration factor: 2.000
[ 495.683] (**) SynPS/2 Synaptics TouchPad: (accel) acceleration threshold: 4
[ 495.842] (--) SynPS/2 Synaptics TouchPad: touchpad found
--------------------- divider added for clarity ----------------
[ 495.843] (II) config/udev: Adding input device SynPS/2 Synaptics TouchPad (/dev/input/mouse0)
[ 495.843] (**) SynPS/2 Synaptics TouchPad: Applying InputClass "touchpad catchall"
[ 495.844] (II) Synaptics touchpad driver version 1.3.99
[ 495.844] (**) Option "Device" "/dev/input/mouse0"
[ 495.922] (--) SynPS/2 Synaptics TouchPad: invalid x-axis range. defaulting to 1615 - 5685
[ 495.922] (--) SynPS/2 Synaptics TouchPad: invalid y-axis range. defaulting to 1729 - 4171
[ 495.922] (--) SynPS/2 Synaptics TouchPad: invalid pressure range. defaulting to 0 - 256
[ 495.922] (--) SynPS/2 Synaptics TouchPad: invalid finger width range. defaulting to 0 - 16
[ 495.943] (EE) Query no Synaptics: 6003C8
[ 495.943] (--) SynPS/2 Synaptics TouchPad: no supported touchpad found
[ 495.943] (EE) SynPS/2 Synaptics TouchPad Unable to query/initialize Synaptics hardware.
[ 495.944] (EE) PreInit returned NULL for "SynPS/2 Synaptics TouchPad"
In both cases, you'll notice that the device gets added twice. Once with a /dev/input/eventX device path, and once with a /dev/input/mouseX path. The first one succeeds and that results in your device being added to the X server's device list. The second one fails because both evdev and synaptics try to use an ioctl that is only available on event devices. The errors do not indicate a driver bug but a misconfiguration. The local configuration needs to be adjusted to avoid non-event devices being assigned to evdev or synaptics.
Look at the synaptics xorg.conf.d snippet we ship in Fedora:
Section "InputClass"
Identifier "touchpad catchall"
Driver "synaptics"
MatchIsTouchpad "on"
MatchDevicePath "/dev/input/event*"
EndSection
That last line is important. It only tries to assign the synaptics driver to event devices, not to mouse devices. Once you've added that line (or poked your distro maintainer to do it for you), the errors will go away. Alternatively, you could just ignore the errors and not file a bug for it :)
Note that the synaptics upstream snippet does not contain this line. Synaptics supports more than just Linux so we cannot add this upstream.
Monday, November 15, 2010
XKB mouse emulation removed from default keymap
XKB has a feature referred to as "MouseKeys" or in the case of xkeyboard-config "PointerKeys". Once enabled, the keypad controls the pointer movement and can be used to emulate button events. The default keymap hat the PointerKeys toggle on Shift+NumLock.
Like so many other things, it is a low level feature that has virtually zero desktop integration. The only way to find out whether it is on or not is to hit a key and see the mouse cursor move. That of course requires you to already know about the feature.
If you've accidentally enabled it the only thing you may notice is that your keypad stopped working. Worse, with a nasty bug we had in servers 1.7 and later (fixed now), the mouse buttons would get stuck when PointerKeys was used intermixed with physical mouse button presses. I've received a fair number of bug reports and other complaints about this feature that I finally decided to remove it from the default options. Sergey has pushed the patch to the xkeyboard-config repository over the weekend, so expect that in xkeyboard-config 2.1 and later, PointerKeys will need to be explicitly enabled.
You can do so with the XKB option "keypad:pointerkeys", or by ticking the "Toggle PointerKeys with Shift + NumLock" checkbox in your desktop keyboard configuration UI.
Like the changed zapping behaviour and the middle button emulation default (both of which have completely failed to bring the world to an end, despite claims to the contrary), this is only a changed default. The feature is still there and can be re-enabled on-the-fly or permanently if needed.
Like so many other things, it is a low level feature that has virtually zero desktop integration. The only way to find out whether it is on or not is to hit a key and see the mouse cursor move. That of course requires you to already know about the feature.
If you've accidentally enabled it the only thing you may notice is that your keypad stopped working. Worse, with a nasty bug we had in servers 1.7 and later (fixed now), the mouse buttons would get stuck when PointerKeys was used intermixed with physical mouse button presses. I've received a fair number of bug reports and other complaints about this feature that I finally decided to remove it from the default options. Sergey has pushed the patch to the xkeyboard-config repository over the weekend, so expect that in xkeyboard-config 2.1 and later, PointerKeys will need to be explicitly enabled.
You can do so with the XKB option "keypad:pointerkeys", or by ticking the "Toggle PointerKeys with Shift + NumLock" checkbox in your desktop keyboard configuration UI.
Like the changed zapping behaviour and the middle button emulation default (both of which have completely failed to bring the world to an end, despite claims to the contrary), this is only a changed default. The feature is still there and can be re-enabled on-the-fly or permanently if needed.
Thursday, November 4, 2010
Adding Reviewed-by tags to patches
I merge a lot of patches and (thankfully!) I get a lot of Reviewed-by, Acked-by, etc. comments on both my patches and others that I end up merging. Copying those is painful and quiet annoying. Somewhen last year, I stumbled across snipMate which allows you to add autocompletion snippets to vim. My $HOME/.vim/snippets/gitcommit file now looks like this:
and the names of the common patch reviewers. So whenever I comment on a patch or I amend a commit after receiving reviews, just typing "rev<space>name alias<space>" is enough to add the Reviewed-by: line.
snipMate is quite powerful but so far I haven't used it much beyond the above. But even just that saved me numerous hours.
snippet ack
Acked-by: ${1}
snippet rev
Reviewed-by: ${1}
snippet sob
Signed-off-by: ${1}
snippet tested
Tested-by: ${1}
snippet me
Peter Hutterer <peter.hutterer@...>
and the names of the common patch reviewers. So whenever I comment on a patch or I amend a commit after receiving reviews, just typing "rev<space>name alias<space>" is enough to add the Reviewed-by: line.
snipMate is quite powerful but so far I haven't used it much beyond the above. But even just that saved me numerous hours.
Thursday, October 28, 2010
XDS 2010 videos
Matt Dew announced that he put the videos from XDS 2010 on youtube. If you're interested about the various topics we talked about, peruse it at your convenience.
For those that don't know him, Matt has been tackling the challenge of collecting X-related documentation (see his talk) for a while now. Someone give that man a medal, or alternatively I guess a suitable beverage will do.
For those that don't know him, Matt has been tackling the challenge of collecting X-related documentation (see his talk) for a while now. Someone give that man a medal, or alternatively I guess a suitable beverage will do.
Saturday, October 2, 2010
Thoughts on Linux multitouch
Note: I am speaking for myself here, in my role as input maintainer for X.Org. The opinions here are my own and may not be the same as those of my employer, other X.Org developers or anyone else alive or not.
Two weeks ago, I was in Toulouse, France, at a multitouch workshop organised by Stèphane Chatty. After the workshop, in the same week was XDS. The workshop had a nice mix of people, Benjamin Tissoires whom I credit with kicking off much of the multitouch work with his evdev hacks, Pengfei, a PhD student of Stèphane, and Chase Douglas from Canonical, involved with their multitouch efforts. Ping Cheng from Wacom, Gowri Ries and Pascal Auriel from Stantum represented the hardware side. And Zeno Albisser and Denis Dzyubenko from Nokia for the toolkit side (Qt). We talked multitouch for two days and I think we got a lot done - not in code but in concepts and general design questions. The current state is essentially that both hardware and software guys are waiting on us, the X server, to integrate multitouch.
The workshop also happened shortly after the Ubuntu multitouch announcement and ended up being an excellent setting to discuss what was, is, and will be happening in regards to multitouch support on Linux. This post is a long one, take your time and don't just skim through. I will focus more on concepts and approaches, not on hard technical stuff. That is for a later post, mainly because I've been on vacation since and I have not yet caught up with new patches.
Let's start from the basics. Traditional input devices such as mice provide one physical device to control one virtual input point (the cursor or the keyboard focus). At it's very basic, multitouch is the ability to use multiple input points on a single physical device. This has some implications on technical issues but many more implications on the UI design. It is also the main reason why multi-pointer (which we support since server 1.7) is not a replacement for multitouch.
Multitouch comes in roughly two different forms: direct touch and what I call dependent touch. Direct touch is simple, a touch point directly relates to the position on the device, i.e. a touch in the upper right corner of the touch device will usually result in some action in the upper right corner of the screen. Dependent-touch devices are e.g. multi-touch aware touchpads but also devices like the Apple Magic Mouse. In their case, the touch is usually interpreted relative to the pointer position of that same device.
The usage of such devices differs vastly too. A direct-touch device will likely be used as such, with plenty of direct object manipulations at their respective position. This again has UI implications: a common design challenge for direct-touch user interfaces is to ensure that all objects are within reach of the user. By the way, from a purely technical point of view, the "touch" part of direct-touch is irrelevant.
A dependent-touch device has a stronger dependency on gestures; a simple touch on such a device doesn't necessarily have a meaning. Touch capabilities on these devices provides an augmentation of traditional input actions, similar to extra buttons or knobs. As usual, the world isn't black and white only and this is only a rough description with plenty of corner cases.
The hardest thing about multitouch is the difference between the data we get and the context the user assumes. Touch devices see touches appear and disappear but cannot (yet) identify appearing and disappearing touches and their context across time. I'll give you one example of the mismatch between the data we get and the context a user may assume: In a drawing application, the following events happen: First, a touch appears on the colour selector and selects the colour red, then the touch disappears. Second, a touch appears on the colour selector and selects the colour green, then the touch disappears. Then two touchpoints appear on and move across the canvas.
Possible scenarios that cause the above datastream are:
Which one it is is impossible to know from the data stream. Hence the difficulty of creating good multitouch interfaces, the data we get only carries a fragment of the information that the user assumes from the context. It is largely a UI design challenge to create a UI where the mismatch does not lead to problems.
One interesting comment from Stèphane was that he and the Intuilab were building more multi-touch UIs a few years ago than he is now. Back then, the interfaces were for specialists who could afford training and thus complex interactions. These days, multitouch is mostly for ad-hoc interaction without any previous training on the UI. Thus, the UI must be "intuitive" and easy to understand. The result of that is that modern multitouch is more often than not just two-finger touch. This also matches the (little) experience I've had with the iPhone so far.
Another thing I noticed again on our tour through the Intuilab is that multitouch interfaces are mainly aimed for consuming content. Some interfaces to assemble existing content are used (photos, slides, etc.) but real content creation seems rare and still relies heavily on settings with more traditional input devices.
A tour through the ENAC LII showed another interesting change - a system for ATC training that was using touchscreens exclusively a few years ago has since been switched to use Wacom Cintiq tablets with styli. As with any device, multitouch devices are no silver bullet, they are great for some tasks but not a complete replacement for all other devices. Using them appropriately is the key to success. Bill Buxton's often-cited mantra of "Everything is best for something and worst for something else" applies here as well.
Why is it taking us so long when there's plenty of multitouch offerings out there already? The simple answer is: we are not working on the same problem.
If we look at commercial products that provide multitouch, Apple's iPhones and iPads are often the first ones that come to mind. These provide multitouch but in a very restrictive setting: one multi-touch aware application running in full-screen. Doing this is suprisingly easy from a technical point of view, all you need is a new API that you write all new applications against. It is of course still hard to make it a good API and design good user interfaces for the new applications, but that is not a purely technical problem anymore. Apple's products also provide multitouch in a new setting, an evironment that's closer to an appliance than a traditional desktop. They have a defined set of features, different form factors, and many of the user expectations we have on the traditional desktop do not exist. For example, hardly anyone expects Word or OpenOffice to run as-is on an iPhone.
The main problems we face with integrating multitouch support into the X server is the need for the traditional desktop. Multitouch must work across multiple windowed application windows, with some pointer emulation to be able to use legacy applications on a screen. I have yet to see a commercial solution that provides this, even the Microsoft Surface applications I've played with so far only emulate this within very restrictive settings. (Disclaimer: I do not own an iPhone, iPad or Surface table so there is the chance that I've missed some new development). We cannot simply ask all developers to rewrite their applications as multi-touch aware and completely ditch the existing ones. Note that e.g. Android can do this, for the same reasons that Apple could.
Having multitouch work across any application window, with fluid transitions between pointer/keyboard interfaces and multitouch interfaces is tricky. And of course, we need to stay compatible with the core protocol and its extensions. Funnily enough, many of the issues we face with multitouch are similar to the ones I faced years ago during the MPX development phase. This is one reason why I am more confident but also quite cautious about it, I already know how it can go wrong.
The basic requirements we have are:
Replace multitouch with multiple users and you have exactly the same requirements I had for MPX (and XI2). Why exactly these four requirements I'll explain separately one day, but the short answer is: zero transition cost between multitouch and non-multitouch interfaces.
Whenever multitouch is mentioned, gestures are brought up quickly too. Gestures are prevalent in this context because they appear to be a natural extension of touch input. They are, but only to a very limited extent. The set of natural gestures is surprisingly small and goes barely beyond the well-known tap, pinch, rotate and swipe. As an exercise, think of the natural gestures for copy and paste. Then see how many of your friends come up with exactly the same gesture.
Moreover, gesture discussions often focus on multi-finger gestures. In the not-too-distant future off-the-shelf hardware will support arbitrary object shapes and that's when hand and arm gestures will become relevant too.
Gestures do not rule out touch. Especially on direct-touch devices it is likely that a UI will use gestures as shortcuts to complex operations but the vast majority of interactions just use plain touch input for object selection, moving objects, and invoking menu commands. So it is important to think of gestures as an abstraction of some touch input but it is by no means the sole vehicle of deliverying touch input. From a technical point of view, this means that gesture recognition must not be in the way of touch recognition.
Gesture detection introduces a delay and that is unavoidable. What can be avoided is the effect of this delay on the UI. With good gesture design the delay is irrelevant. Take the example of a system with two gestures: a one finger hold gesture and a four finger hold gesture (i.e. an action is triggered when one or four fingers are put onto a position). Assume that the timeout for the gesture is 2 seconds. When a user puts a single finger on the device, the system now has to wait 2 seconds before it can react. If within 2 seconds the user does not put three more fingers down, the single-finger action can be taken. Decreasing the timeout lands the UI designer at the blurry line between a laggy UI and too-hard-to-trigger gestures.
To solve this issue, the system must be able to recognize which gesture is performed before the user expects feedback. Consider the gesture set of tapping one finger, moving one finger and a four finger hold gesture. Tapping and moving are easy to detect and they don't suffer from an artificial delay. The only time the timeout is needed is when the user puts a finger down but doesn't move or release it immediately. Now the system has to wait for more fingers. However, since a single finger not moving is not a defined gesture, the user expects no feedback and the timeout is irrelevant.
Integrating gesture support is thus not a matter of simply defining a few gestures. It's a matter of selecting the right ones that do not expose the user to a laggy experience. The FOSS software stack is notoriously segregated and thus true gesture integration must be done across window managers, desktop environments and applications. Coincidentally, this is my main worry with the Unity gesture support, I feel the gestures have a potential to overlap in some cases and thus be detrimental to the user experience. I hope I'm wrong here.
When it comes to input event hanlding, the X server's job is to convert a set of hardware events into a set of events meaningful in a GUI. This includes converting the hardware information into screen-relative information and picking (selecting where to send the event to). I've already outlined the four requirements for multitouch above.
Originally, I thought multitouch could be integrated into our current event system but a few months back I gave up on that. Whenever I tried to sort out the corner cases, there were some situations where the core protocol would be unresolvable. The current aim is that we add new events and they will live side-by-side to pointer and keyboard events, not necessarily influencing those. This means that clients must add support for multitouch to be useable, even XI2 enabled clients will not provide that functionality without modifications (pointer emulation for single-finger touches is still considered as required feature though).
Beyond that, things are still in flux. At the time of this writing, there are two protocol proposals that are largely overlapping. Daniel Stone's version has taken a few features from mine as I have taken some of his but the basics approach is the same (and we came up with it independently, which is a good sign). Daniel's proposal is also backed by an implementation though I have yet to look at these patches. Either way, what will be happening is a new set of events and a few requests to detect which devices have multitouch. Passive grabs on touch points will be possible but that still needs to be sorted out. Once it is, I'll post more details.
The summary here is that things are moving, at least in my case largely due to Canonical's announcement which lit a fire under some southern body parts and finally made me spend more time thinking about multitouch.
One thing that is likely of interest to many is how all this will integrate with Ubuntu's multitouch support. Ubuntu has committed to strong gesture support. This is commendable but quite limiting in my opinion as true multitouch is much more than gestures. The current UI seems aimed more at dependent-touch devices like the Apple Magic Trackpad where gestures agument traditional input, less so at a fully multitouch-aware desktop. Fair enough, so far there aren't a lot of multitouch applications out there but for system-level design we need to keep the medium and long term in mind too.
The current integration is a hack (and the developers don't try to hide this), but most good things start as hacks so there's not much to say here. I am certianly impressed that they are willing to deliver a UI based on what is still a moving target. The few issues I had were with the design, so let's focus on that.
The originally proposed X Gesture Extension tried to get a recogniser into the server (not a 100% accurate statement, but close enough). It only allowed for one recogniser in the system. I do not think that we'll ever manage to find the one true gesture recognizer, we haven't managed to find the one true window manager or toolkit either. I talked at length to Chase about this and the revised architecture diagrams Chase presented at the workshop and XDS do not call for gesture recognition in the server anymore.
With the new approach the recognition happens purely client-side and is done by a library or a daemon. This allows for multiple different gesture recognisers be active at the same time, a scenario that is quite likely to happen (I do envision GTK, Qt, Mozilla, etc. all wanting their own system). Whether that's a good thing for the UI is another matter, consistency for gestures is important and especially Ping Cheng was not happy at the prospect of having multiple, possibly inconsistent, systems. We need to find some common ground here between desktop environments and toolkits. Geis tries to address this, though it is in a quite early stage. The main issue with geis is a political one. The implementation requires copyright assignment and puts up quite a barrier to entry. The utouch team tries to get around this by separating the interface specification and the implementation, but that's just calling for a fork or duplicated efforts before there's anything to fork or duplicate. So while I think geis is the right approach and much needed, I fear the political decisions will hold it back unnecessarily. My suggestion is to ditch the copyright assignment, because something as central as a generic gesture interface should be developed together with toolkit developers, X.Org developers, application developers, etc. And that's just easier when contributions are as simple as possible.
As for the UI itself - any UI decisions already made will unlikely be an issue from a technical perspective. Gestures are an important part of multitouch support and we will make it possible to provide gestures. All the technical parts are still in flux, but the current abstractions Ubuntu has put in place are likely good enough to adjust to a new baseline. So while I cannot speak for Ubuntu at all, my guess is that in future versions the bottom bits of the stack will be swapped out but the top bits (the user-visible ones) will largely remain the same.
The above should provide a general overview on some of the issues with multitouch and challenges we face. I've skipped over a few and only briefly touched on others for brevity, it's quite easy to write full novels about all this.
I will post more technical details about the kernel's multitouch API and how we plan to use it in X.Org for our protocol extensions once the latter is at least partially nailed down. For now, the concepts described above hopefully framed your mind in terms of multitouch and some of the concepts we're looking at.
Two weeks ago, I was in Toulouse, France, at a multitouch workshop organised by Stèphane Chatty. After the workshop, in the same week was XDS. The workshop had a nice mix of people, Benjamin Tissoires whom I credit with kicking off much of the multitouch work with his evdev hacks, Pengfei, a PhD student of Stèphane, and Chase Douglas from Canonical, involved with their multitouch efforts. Ping Cheng from Wacom, Gowri Ries and Pascal Auriel from Stantum represented the hardware side. And Zeno Albisser and Denis Dzyubenko from Nokia for the toolkit side (Qt). We talked multitouch for two days and I think we got a lot done - not in code but in concepts and general design questions. The current state is essentially that both hardware and software guys are waiting on us, the X server, to integrate multitouch.
The workshop also happened shortly after the Ubuntu multitouch announcement and ended up being an excellent setting to discuss what was, is, and will be happening in regards to multitouch support on Linux. This post is a long one, take your time and don't just skim through. I will focus more on concepts and approaches, not on hard technical stuff. That is for a later post, mainly because I've been on vacation since and I have not yet caught up with new patches.
Multitouch basics
Let's start from the basics. Traditional input devices such as mice provide one physical device to control one virtual input point (the cursor or the keyboard focus). At it's very basic, multitouch is the ability to use multiple input points on a single physical device. This has some implications on technical issues but many more implications on the UI design. It is also the main reason why multi-pointer (which we support since server 1.7) is not a replacement for multitouch.
Multitouch comes in roughly two different forms: direct touch and what I call dependent touch. Direct touch is simple, a touch point directly relates to the position on the device, i.e. a touch in the upper right corner of the touch device will usually result in some action in the upper right corner of the screen. Dependent-touch devices are e.g. multi-touch aware touchpads but also devices like the Apple Magic Mouse. In their case, the touch is usually interpreted relative to the pointer position of that same device.
The usage of such devices differs vastly too. A direct-touch device will likely be used as such, with plenty of direct object manipulations at their respective position. This again has UI implications: a common design challenge for direct-touch user interfaces is to ensure that all objects are within reach of the user. By the way, from a purely technical point of view, the "touch" part of direct-touch is irrelevant.
A dependent-touch device has a stronger dependency on gestures; a simple touch on such a device doesn't necessarily have a meaning. Touch capabilities on these devices provides an augmentation of traditional input actions, similar to extra buttons or knobs. As usual, the world isn't black and white only and this is only a rough description with plenty of corner cases.
The lack of context
The hardest thing about multitouch is the difference between the data we get and the context the user assumes. Touch devices see touches appear and disappear but cannot (yet) identify appearing and disappearing touches and their context across time. I'll give you one example of the mismatch between the data we get and the context a user may assume: In a drawing application, the following events happen: First, a touch appears on the colour selector and selects the colour red, then the touch disappears. Second, a touch appears on the colour selector and selects the colour green, then the touch disappears. Then two touchpoints appear on and move across the canvas.
Possible scenarios that cause the above datastream are:
- The user has erroneously selected red, then corrected to colour green, now is painting with two fingers in green.
- The user has selected red with one finger, green with another finger and now wants two paint with two different colours.
- The user has erroneously selected red with one finger, corrected to green and now wants to paint in green and the colour the other finger already had assigned to.
- Two users selected two different colours and now paint simultaneously.
Which one it is is impossible to know from the data stream. Hence the difficulty of creating good multitouch interfaces, the data we get only carries a fragment of the information that the user assumes from the context. It is largely a UI design challenge to create a UI where the mismatch does not lead to problems.
Multitouch evolutions
One interesting comment from Stèphane was that he and the Intuilab were building more multi-touch UIs a few years ago than he is now. Back then, the interfaces were for specialists who could afford training and thus complex interactions. These days, multitouch is mostly for ad-hoc interaction without any previous training on the UI. Thus, the UI must be "intuitive" and easy to understand. The result of that is that modern multitouch is more often than not just two-finger touch. This also matches the (little) experience I've had with the iPhone so far.
Another thing I noticed again on our tour through the Intuilab is that multitouch interfaces are mainly aimed for consuming content. Some interfaces to assemble existing content are used (photos, slides, etc.) but real content creation seems rare and still relies heavily on settings with more traditional input devices.
A tour through the ENAC LII showed another interesting change - a system for ATC training that was using touchscreens exclusively a few years ago has since been switched to use Wacom Cintiq tablets with styli. As with any device, multitouch devices are no silver bullet, they are great for some tasks but not a complete replacement for all other devices. Using them appropriately is the key to success. Bill Buxton's often-cited mantra of "Everything is best for something and worst for something else" applies here as well.
Apple, Microsoft and other multitouch products
Why is it taking us so long when there's plenty of multitouch offerings out there already? The simple answer is: we are not working on the same problem.
If we look at commercial products that provide multitouch, Apple's iPhones and iPads are often the first ones that come to mind. These provide multitouch but in a very restrictive setting: one multi-touch aware application running in full-screen. Doing this is suprisingly easy from a technical point of view, all you need is a new API that you write all new applications against. It is of course still hard to make it a good API and design good user interfaces for the new applications, but that is not a purely technical problem anymore. Apple's products also provide multitouch in a new setting, an evironment that's closer to an appliance than a traditional desktop. They have a defined set of features, different form factors, and many of the user expectations we have on the traditional desktop do not exist. For example, hardly anyone expects Word or OpenOffice to run as-is on an iPhone.
The main problems we face with integrating multitouch support into the X server is the need for the traditional desktop. Multitouch must work across multiple windowed application windows, with some pointer emulation to be able to use legacy applications on a screen. I have yet to see a commercial solution that provides this, even the Microsoft Surface applications I've played with so far only emulate this within very restrictive settings. (Disclaimer: I do not own an iPhone, iPad or Surface table so there is the chance that I've missed some new development). We cannot simply ask all developers to rewrite their applications as multi-touch aware and completely ditch the existing ones. Note that e.g. Android can do this, for the same reasons that Apple could.
Having multitouch work across any application window, with fluid transitions between pointer/keyboard interfaces and multitouch interfaces is tricky. And of course, we need to stay compatible with the core protocol and its extensions. Funnily enough, many of the issues we face with multitouch are similar to the ones I faced years ago during the MPX development phase. This is one reason why I am more confident but also quite cautious about it, I already know how it can go wrong.
The basic requirements we have are:
- Multitouch in any application that supports it
- Multitouch in any number of applications simultanously
- Pointer emulation for non-multitouch applications
- Simultaneous execution of multitouch and pointer/keyboard applications
Replace multitouch with multiple users and you have exactly the same requirements I had for MPX (and XI2). Why exactly these four requirements I'll explain separately one day, but the short answer is: zero transition cost between multitouch and non-multitouch interfaces.
Gestures
Whenever multitouch is mentioned, gestures are brought up quickly too. Gestures are prevalent in this context because they appear to be a natural extension of touch input. They are, but only to a very limited extent. The set of natural gestures is surprisingly small and goes barely beyond the well-known tap, pinch, rotate and swipe. As an exercise, think of the natural gestures for copy and paste. Then see how many of your friends come up with exactly the same gesture.
Moreover, gesture discussions often focus on multi-finger gestures. In the not-too-distant future off-the-shelf hardware will support arbitrary object shapes and that's when hand and arm gestures will become relevant too.
Gestures do not rule out touch. Especially on direct-touch devices it is likely that a UI will use gestures as shortcuts to complex operations but the vast majority of interactions just use plain touch input for object selection, moving objects, and invoking menu commands. So it is important to think of gestures as an abstraction of some touch input but it is by no means the sole vehicle of deliverying touch input. From a technical point of view, this means that gesture recognition must not be in the way of touch recognition.
Gesture detection introduces a delay and that is unavoidable. What can be avoided is the effect of this delay on the UI. With good gesture design the delay is irrelevant. Take the example of a system with two gestures: a one finger hold gesture and a four finger hold gesture (i.e. an action is triggered when one or four fingers are put onto a position). Assume that the timeout for the gesture is 2 seconds. When a user puts a single finger on the device, the system now has to wait 2 seconds before it can react. If within 2 seconds the user does not put three more fingers down, the single-finger action can be taken. Decreasing the timeout lands the UI designer at the blurry line between a laggy UI and too-hard-to-trigger gestures.
To solve this issue, the system must be able to recognize which gesture is performed before the user expects feedback. Consider the gesture set of tapping one finger, moving one finger and a four finger hold gesture. Tapping and moving are easy to detect and they don't suffer from an artificial delay. The only time the timeout is needed is when the user puts a finger down but doesn't move or release it immediately. Now the system has to wait for more fingers. However, since a single finger not moving is not a defined gesture, the user expects no feedback and the timeout is irrelevant.
Integrating gesture support is thus not a matter of simply defining a few gestures. It's a matter of selecting the right ones that do not expose the user to a laggy experience. The FOSS software stack is notoriously segregated and thus true gesture integration must be done across window managers, desktop environments and applications. Coincidentally, this is my main worry with the Unity gesture support, I feel the gestures have a potential to overlap in some cases and thus be detrimental to the user experience. I hope I'm wrong here.
X.Org multitouch support
When it comes to input event hanlding, the X server's job is to convert a set of hardware events into a set of events meaningful in a GUI. This includes converting the hardware information into screen-relative information and picking (selecting where to send the event to). I've already outlined the four requirements for multitouch above.
Originally, I thought multitouch could be integrated into our current event system but a few months back I gave up on that. Whenever I tried to sort out the corner cases, there were some situations where the core protocol would be unresolvable. The current aim is that we add new events and they will live side-by-side to pointer and keyboard events, not necessarily influencing those. This means that clients must add support for multitouch to be useable, even XI2 enabled clients will not provide that functionality without modifications (pointer emulation for single-finger touches is still considered as required feature though).
Beyond that, things are still in flux. At the time of this writing, there are two protocol proposals that are largely overlapping. Daniel Stone's version has taken a few features from mine as I have taken some of his but the basics approach is the same (and we came up with it independently, which is a good sign). Daniel's proposal is also backed by an implementation though I have yet to look at these patches. Either way, what will be happening is a new set of events and a few requests to detect which devices have multitouch. Passive grabs on touch points will be possible but that still needs to be sorted out. Once it is, I'll post more details.
The summary here is that things are moving, at least in my case largely due to Canonical's announcement which lit a fire under some southern body parts and finally made me spend more time thinking about multitouch.
Ubuntu's multitouch support
One thing that is likely of interest to many is how all this will integrate with Ubuntu's multitouch support. Ubuntu has committed to strong gesture support. This is commendable but quite limiting in my opinion as true multitouch is much more than gestures. The current UI seems aimed more at dependent-touch devices like the Apple Magic Trackpad where gestures agument traditional input, less so at a fully multitouch-aware desktop. Fair enough, so far there aren't a lot of multitouch applications out there but for system-level design we need to keep the medium and long term in mind too.
The current integration is a hack (and the developers don't try to hide this), but most good things start as hacks so there's not much to say here. I am certianly impressed that they are willing to deliver a UI based on what is still a moving target. The few issues I had were with the design, so let's focus on that.
The originally proposed X Gesture Extension tried to get a recogniser into the server (not a 100% accurate statement, but close enough). It only allowed for one recogniser in the system. I do not think that we'll ever manage to find the one true gesture recognizer, we haven't managed to find the one true window manager or toolkit either. I talked at length to Chase about this and the revised architecture diagrams Chase presented at the workshop and XDS do not call for gesture recognition in the server anymore.
With the new approach the recognition happens purely client-side and is done by a library or a daemon. This allows for multiple different gesture recognisers be active at the same time, a scenario that is quite likely to happen (I do envision GTK, Qt, Mozilla, etc. all wanting their own system). Whether that's a good thing for the UI is another matter, consistency for gestures is important and especially Ping Cheng was not happy at the prospect of having multiple, possibly inconsistent, systems. We need to find some common ground here between desktop environments and toolkits. Geis tries to address this, though it is in a quite early stage. The main issue with geis is a political one. The implementation requires copyright assignment and puts up quite a barrier to entry. The utouch team tries to get around this by separating the interface specification and the implementation, but that's just calling for a fork or duplicated efforts before there's anything to fork or duplicate. So while I think geis is the right approach and much needed, I fear the political decisions will hold it back unnecessarily. My suggestion is to ditch the copyright assignment, because something as central as a generic gesture interface should be developed together with toolkit developers, X.Org developers, application developers, etc. And that's just easier when contributions are as simple as possible.
As for the UI itself - any UI decisions already made will unlikely be an issue from a technical perspective. Gestures are an important part of multitouch support and we will make it possible to provide gestures. All the technical parts are still in flux, but the current abstractions Ubuntu has put in place are likely good enough to adjust to a new baseline. So while I cannot speak for Ubuntu at all, my guess is that in future versions the bottom bits of the stack will be swapped out but the top bits (the user-visible ones) will largely remain the same.
Conclusion
The above should provide a general overview on some of the issues with multitouch and challenges we face. I've skipped over a few and only briefly touched on others for brevity, it's quite easy to write full novels about all this.
I will post more technical details about the kernel's multitouch API and how we plan to use it in X.Org for our protocol extensions once the latter is at least partially nailed down. For now, the concepts described above hopefully framed your mind in terms of multitouch and some of the concepts we're looking at.
Subscribe to:
Posts (Atom)