Matt Dew announced that he put the videos from XDS 2010 on youtube. If you're interested about the various topics we talked about, peruse it at your convenience.
For those that don't know him, Matt has been tackling the challenge of collecting X-related documentation (see his talk) for a while now. Someone give that man a medal, or alternatively I guess a suitable beverage will do.
Thursday, October 28, 2010
Saturday, October 2, 2010
Thoughts on Linux multitouch
Note: I am speaking for myself here, in my role as input maintainer for X.Org. The opinions here are my own and may not be the same as those of my employer, other X.Org developers or anyone else alive or not.
Two weeks ago, I was in Toulouse, France, at a multitouch workshop organised by Stèphane Chatty. After the workshop, in the same week was XDS. The workshop had a nice mix of people, Benjamin Tissoires whom I credit with kicking off much of the multitouch work with his evdev hacks, Pengfei, a PhD student of Stèphane, and Chase Douglas from Canonical, involved with their multitouch efforts. Ping Cheng from Wacom, Gowri Ries and Pascal Auriel from Stantum represented the hardware side. And Zeno Albisser and Denis Dzyubenko from Nokia for the toolkit side (Qt). We talked multitouch for two days and I think we got a lot done - not in code but in concepts and general design questions. The current state is essentially that both hardware and software guys are waiting on us, the X server, to integrate multitouch.
The workshop also happened shortly after the Ubuntu multitouch announcement and ended up being an excellent setting to discuss what was, is, and will be happening in regards to multitouch support on Linux. This post is a long one, take your time and don't just skim through. I will focus more on concepts and approaches, not on hard technical stuff. That is for a later post, mainly because I've been on vacation since and I have not yet caught up with new patches.
Let's start from the basics. Traditional input devices such as mice provide one physical device to control one virtual input point (the cursor or the keyboard focus). At it's very basic, multitouch is the ability to use multiple input points on a single physical device. This has some implications on technical issues but many more implications on the UI design. It is also the main reason why multi-pointer (which we support since server 1.7) is not a replacement for multitouch.
Multitouch comes in roughly two different forms: direct touch and what I call dependent touch. Direct touch is simple, a touch point directly relates to the position on the device, i.e. a touch in the upper right corner of the touch device will usually result in some action in the upper right corner of the screen. Dependent-touch devices are e.g. multi-touch aware touchpads but also devices like the Apple Magic Mouse. In their case, the touch is usually interpreted relative to the pointer position of that same device.
The usage of such devices differs vastly too. A direct-touch device will likely be used as such, with plenty of direct object manipulations at their respective position. This again has UI implications: a common design challenge for direct-touch user interfaces is to ensure that all objects are within reach of the user. By the way, from a purely technical point of view, the "touch" part of direct-touch is irrelevant.
A dependent-touch device has a stronger dependency on gestures; a simple touch on such a device doesn't necessarily have a meaning. Touch capabilities on these devices provides an augmentation of traditional input actions, similar to extra buttons or knobs. As usual, the world isn't black and white only and this is only a rough description with plenty of corner cases.
The hardest thing about multitouch is the difference between the data we get and the context the user assumes. Touch devices see touches appear and disappear but cannot (yet) identify appearing and disappearing touches and their context across time. I'll give you one example of the mismatch between the data we get and the context a user may assume: In a drawing application, the following events happen: First, a touch appears on the colour selector and selects the colour red, then the touch disappears. Second, a touch appears on the colour selector and selects the colour green, then the touch disappears. Then two touchpoints appear on and move across the canvas.
Possible scenarios that cause the above datastream are:
Which one it is is impossible to know from the data stream. Hence the difficulty of creating good multitouch interfaces, the data we get only carries a fragment of the information that the user assumes from the context. It is largely a UI design challenge to create a UI where the mismatch does not lead to problems.
One interesting comment from Stèphane was that he and the Intuilab were building more multi-touch UIs a few years ago than he is now. Back then, the interfaces were for specialists who could afford training and thus complex interactions. These days, multitouch is mostly for ad-hoc interaction without any previous training on the UI. Thus, the UI must be "intuitive" and easy to understand. The result of that is that modern multitouch is more often than not just two-finger touch. This also matches the (little) experience I've had with the iPhone so far.
Another thing I noticed again on our tour through the Intuilab is that multitouch interfaces are mainly aimed for consuming content. Some interfaces to assemble existing content are used (photos, slides, etc.) but real content creation seems rare and still relies heavily on settings with more traditional input devices.
A tour through the ENAC LII showed another interesting change - a system for ATC training that was using touchscreens exclusively a few years ago has since been switched to use Wacom Cintiq tablets with styli. As with any device, multitouch devices are no silver bullet, they are great for some tasks but not a complete replacement for all other devices. Using them appropriately is the key to success. Bill Buxton's often-cited mantra of "Everything is best for something and worst for something else" applies here as well.
Why is it taking us so long when there's plenty of multitouch offerings out there already? The simple answer is: we are not working on the same problem.
If we look at commercial products that provide multitouch, Apple's iPhones and iPads are often the first ones that come to mind. These provide multitouch but in a very restrictive setting: one multi-touch aware application running in full-screen. Doing this is suprisingly easy from a technical point of view, all you need is a new API that you write all new applications against. It is of course still hard to make it a good API and design good user interfaces for the new applications, but that is not a purely technical problem anymore. Apple's products also provide multitouch in a new setting, an evironment that's closer to an appliance than a traditional desktop. They have a defined set of features, different form factors, and many of the user expectations we have on the traditional desktop do not exist. For example, hardly anyone expects Word or OpenOffice to run as-is on an iPhone.
The main problems we face with integrating multitouch support into the X server is the need for the traditional desktop. Multitouch must work across multiple windowed application windows, with some pointer emulation to be able to use legacy applications on a screen. I have yet to see a commercial solution that provides this, even the Microsoft Surface applications I've played with so far only emulate this within very restrictive settings. (Disclaimer: I do not own an iPhone, iPad or Surface table so there is the chance that I've missed some new development). We cannot simply ask all developers to rewrite their applications as multi-touch aware and completely ditch the existing ones. Note that e.g. Android can do this, for the same reasons that Apple could.
Having multitouch work across any application window, with fluid transitions between pointer/keyboard interfaces and multitouch interfaces is tricky. And of course, we need to stay compatible with the core protocol and its extensions. Funnily enough, many of the issues we face with multitouch are similar to the ones I faced years ago during the MPX development phase. This is one reason why I am more confident but also quite cautious about it, I already know how it can go wrong.
The basic requirements we have are:
Replace multitouch with multiple users and you have exactly the same requirements I had for MPX (and XI2). Why exactly these four requirements I'll explain separately one day, but the short answer is: zero transition cost between multitouch and non-multitouch interfaces.
Whenever multitouch is mentioned, gestures are brought up quickly too. Gestures are prevalent in this context because they appear to be a natural extension of touch input. They are, but only to a very limited extent. The set of natural gestures is surprisingly small and goes barely beyond the well-known tap, pinch, rotate and swipe. As an exercise, think of the natural gestures for copy and paste. Then see how many of your friends come up with exactly the same gesture.
Moreover, gesture discussions often focus on multi-finger gestures. In the not-too-distant future off-the-shelf hardware will support arbitrary object shapes and that's when hand and arm gestures will become relevant too.
Gestures do not rule out touch. Especially on direct-touch devices it is likely that a UI will use gestures as shortcuts to complex operations but the vast majority of interactions just use plain touch input for object selection, moving objects, and invoking menu commands. So it is important to think of gestures as an abstraction of some touch input but it is by no means the sole vehicle of deliverying touch input. From a technical point of view, this means that gesture recognition must not be in the way of touch recognition.
Gesture detection introduces a delay and that is unavoidable. What can be avoided is the effect of this delay on the UI. With good gesture design the delay is irrelevant. Take the example of a system with two gestures: a one finger hold gesture and a four finger hold gesture (i.e. an action is triggered when one or four fingers are put onto a position). Assume that the timeout for the gesture is 2 seconds. When a user puts a single finger on the device, the system now has to wait 2 seconds before it can react. If within 2 seconds the user does not put three more fingers down, the single-finger action can be taken. Decreasing the timeout lands the UI designer at the blurry line between a laggy UI and too-hard-to-trigger gestures.
To solve this issue, the system must be able to recognize which gesture is performed before the user expects feedback. Consider the gesture set of tapping one finger, moving one finger and a four finger hold gesture. Tapping and moving are easy to detect and they don't suffer from an artificial delay. The only time the timeout is needed is when the user puts a finger down but doesn't move or release it immediately. Now the system has to wait for more fingers. However, since a single finger not moving is not a defined gesture, the user expects no feedback and the timeout is irrelevant.
Integrating gesture support is thus not a matter of simply defining a few gestures. It's a matter of selecting the right ones that do not expose the user to a laggy experience. The FOSS software stack is notoriously segregated and thus true gesture integration must be done across window managers, desktop environments and applications. Coincidentally, this is my main worry with the Unity gesture support, I feel the gestures have a potential to overlap in some cases and thus be detrimental to the user experience. I hope I'm wrong here.
When it comes to input event hanlding, the X server's job is to convert a set of hardware events into a set of events meaningful in a GUI. This includes converting the hardware information into screen-relative information and picking (selecting where to send the event to). I've already outlined the four requirements for multitouch above.
Originally, I thought multitouch could be integrated into our current event system but a few months back I gave up on that. Whenever I tried to sort out the corner cases, there were some situations where the core protocol would be unresolvable. The current aim is that we add new events and they will live side-by-side to pointer and keyboard events, not necessarily influencing those. This means that clients must add support for multitouch to be useable, even XI2 enabled clients will not provide that functionality without modifications (pointer emulation for single-finger touches is still considered as required feature though).
Beyond that, things are still in flux. At the time of this writing, there are two protocol proposals that are largely overlapping. Daniel Stone's version has taken a few features from mine as I have taken some of his but the basics approach is the same (and we came up with it independently, which is a good sign). Daniel's proposal is also backed by an implementation though I have yet to look at these patches. Either way, what will be happening is a new set of events and a few requests to detect which devices have multitouch. Passive grabs on touch points will be possible but that still needs to be sorted out. Once it is, I'll post more details.
The summary here is that things are moving, at least in my case largely due to Canonical's announcement which lit a fire under some southern body parts and finally made me spend more time thinking about multitouch.
One thing that is likely of interest to many is how all this will integrate with Ubuntu's multitouch support. Ubuntu has committed to strong gesture support. This is commendable but quite limiting in my opinion as true multitouch is much more than gestures. The current UI seems aimed more at dependent-touch devices like the Apple Magic Trackpad where gestures agument traditional input, less so at a fully multitouch-aware desktop. Fair enough, so far there aren't a lot of multitouch applications out there but for system-level design we need to keep the medium and long term in mind too.
The current integration is a hack (and the developers don't try to hide this), but most good things start as hacks so there's not much to say here. I am certianly impressed that they are willing to deliver a UI based on what is still a moving target. The few issues I had were with the design, so let's focus on that.
The originally proposed X Gesture Extension tried to get a recogniser into the server (not a 100% accurate statement, but close enough). It only allowed for one recogniser in the system. I do not think that we'll ever manage to find the one true gesture recognizer, we haven't managed to find the one true window manager or toolkit either. I talked at length to Chase about this and the revised architecture diagrams Chase presented at the workshop and XDS do not call for gesture recognition in the server anymore.
With the new approach the recognition happens purely client-side and is done by a library or a daemon. This allows for multiple different gesture recognisers be active at the same time, a scenario that is quite likely to happen (I do envision GTK, Qt, Mozilla, etc. all wanting their own system). Whether that's a good thing for the UI is another matter, consistency for gestures is important and especially Ping Cheng was not happy at the prospect of having multiple, possibly inconsistent, systems. We need to find some common ground here between desktop environments and toolkits. Geis tries to address this, though it is in a quite early stage. The main issue with geis is a political one. The implementation requires copyright assignment and puts up quite a barrier to entry. The utouch team tries to get around this by separating the interface specification and the implementation, but that's just calling for a fork or duplicated efforts before there's anything to fork or duplicate. So while I think geis is the right approach and much needed, I fear the political decisions will hold it back unnecessarily. My suggestion is to ditch the copyright assignment, because something as central as a generic gesture interface should be developed together with toolkit developers, X.Org developers, application developers, etc. And that's just easier when contributions are as simple as possible.
As for the UI itself - any UI decisions already made will unlikely be an issue from a technical perspective. Gestures are an important part of multitouch support and we will make it possible to provide gestures. All the technical parts are still in flux, but the current abstractions Ubuntu has put in place are likely good enough to adjust to a new baseline. So while I cannot speak for Ubuntu at all, my guess is that in future versions the bottom bits of the stack will be swapped out but the top bits (the user-visible ones) will largely remain the same.
The above should provide a general overview on some of the issues with multitouch and challenges we face. I've skipped over a few and only briefly touched on others for brevity, it's quite easy to write full novels about all this.
I will post more technical details about the kernel's multitouch API and how we plan to use it in X.Org for our protocol extensions once the latter is at least partially nailed down. For now, the concepts described above hopefully framed your mind in terms of multitouch and some of the concepts we're looking at.
Two weeks ago, I was in Toulouse, France, at a multitouch workshop organised by Stèphane Chatty. After the workshop, in the same week was XDS. The workshop had a nice mix of people, Benjamin Tissoires whom I credit with kicking off much of the multitouch work with his evdev hacks, Pengfei, a PhD student of Stèphane, and Chase Douglas from Canonical, involved with their multitouch efforts. Ping Cheng from Wacom, Gowri Ries and Pascal Auriel from Stantum represented the hardware side. And Zeno Albisser and Denis Dzyubenko from Nokia for the toolkit side (Qt). We talked multitouch for two days and I think we got a lot done - not in code but in concepts and general design questions. The current state is essentially that both hardware and software guys are waiting on us, the X server, to integrate multitouch.
The workshop also happened shortly after the Ubuntu multitouch announcement and ended up being an excellent setting to discuss what was, is, and will be happening in regards to multitouch support on Linux. This post is a long one, take your time and don't just skim through. I will focus more on concepts and approaches, not on hard technical stuff. That is for a later post, mainly because I've been on vacation since and I have not yet caught up with new patches.
Multitouch basics
Let's start from the basics. Traditional input devices such as mice provide one physical device to control one virtual input point (the cursor or the keyboard focus). At it's very basic, multitouch is the ability to use multiple input points on a single physical device. This has some implications on technical issues but many more implications on the UI design. It is also the main reason why multi-pointer (which we support since server 1.7) is not a replacement for multitouch.
Multitouch comes in roughly two different forms: direct touch and what I call dependent touch. Direct touch is simple, a touch point directly relates to the position on the device, i.e. a touch in the upper right corner of the touch device will usually result in some action in the upper right corner of the screen. Dependent-touch devices are e.g. multi-touch aware touchpads but also devices like the Apple Magic Mouse. In their case, the touch is usually interpreted relative to the pointer position of that same device.
The usage of such devices differs vastly too. A direct-touch device will likely be used as such, with plenty of direct object manipulations at their respective position. This again has UI implications: a common design challenge for direct-touch user interfaces is to ensure that all objects are within reach of the user. By the way, from a purely technical point of view, the "touch" part of direct-touch is irrelevant.
A dependent-touch device has a stronger dependency on gestures; a simple touch on such a device doesn't necessarily have a meaning. Touch capabilities on these devices provides an augmentation of traditional input actions, similar to extra buttons or knobs. As usual, the world isn't black and white only and this is only a rough description with plenty of corner cases.
The lack of context
The hardest thing about multitouch is the difference between the data we get and the context the user assumes. Touch devices see touches appear and disappear but cannot (yet) identify appearing and disappearing touches and their context across time. I'll give you one example of the mismatch between the data we get and the context a user may assume: In a drawing application, the following events happen: First, a touch appears on the colour selector and selects the colour red, then the touch disappears. Second, a touch appears on the colour selector and selects the colour green, then the touch disappears. Then two touchpoints appear on and move across the canvas.
Possible scenarios that cause the above datastream are:
- The user has erroneously selected red, then corrected to colour green, now is painting with two fingers in green.
- The user has selected red with one finger, green with another finger and now wants two paint with two different colours.
- The user has erroneously selected red with one finger, corrected to green and now wants to paint in green and the colour the other finger already had assigned to.
- Two users selected two different colours and now paint simultaneously.
Which one it is is impossible to know from the data stream. Hence the difficulty of creating good multitouch interfaces, the data we get only carries a fragment of the information that the user assumes from the context. It is largely a UI design challenge to create a UI where the mismatch does not lead to problems.
Multitouch evolutions
One interesting comment from Stèphane was that he and the Intuilab were building more multi-touch UIs a few years ago than he is now. Back then, the interfaces were for specialists who could afford training and thus complex interactions. These days, multitouch is mostly for ad-hoc interaction without any previous training on the UI. Thus, the UI must be "intuitive" and easy to understand. The result of that is that modern multitouch is more often than not just two-finger touch. This also matches the (little) experience I've had with the iPhone so far.
Another thing I noticed again on our tour through the Intuilab is that multitouch interfaces are mainly aimed for consuming content. Some interfaces to assemble existing content are used (photos, slides, etc.) but real content creation seems rare and still relies heavily on settings with more traditional input devices.
A tour through the ENAC LII showed another interesting change - a system for ATC training that was using touchscreens exclusively a few years ago has since been switched to use Wacom Cintiq tablets with styli. As with any device, multitouch devices are no silver bullet, they are great for some tasks but not a complete replacement for all other devices. Using them appropriately is the key to success. Bill Buxton's often-cited mantra of "Everything is best for something and worst for something else" applies here as well.
Apple, Microsoft and other multitouch products
Why is it taking us so long when there's plenty of multitouch offerings out there already? The simple answer is: we are not working on the same problem.
If we look at commercial products that provide multitouch, Apple's iPhones and iPads are often the first ones that come to mind. These provide multitouch but in a very restrictive setting: one multi-touch aware application running in full-screen. Doing this is suprisingly easy from a technical point of view, all you need is a new API that you write all new applications against. It is of course still hard to make it a good API and design good user interfaces for the new applications, but that is not a purely technical problem anymore. Apple's products also provide multitouch in a new setting, an evironment that's closer to an appliance than a traditional desktop. They have a defined set of features, different form factors, and many of the user expectations we have on the traditional desktop do not exist. For example, hardly anyone expects Word or OpenOffice to run as-is on an iPhone.
The main problems we face with integrating multitouch support into the X server is the need for the traditional desktop. Multitouch must work across multiple windowed application windows, with some pointer emulation to be able to use legacy applications on a screen. I have yet to see a commercial solution that provides this, even the Microsoft Surface applications I've played with so far only emulate this within very restrictive settings. (Disclaimer: I do not own an iPhone, iPad or Surface table so there is the chance that I've missed some new development). We cannot simply ask all developers to rewrite their applications as multi-touch aware and completely ditch the existing ones. Note that e.g. Android can do this, for the same reasons that Apple could.
Having multitouch work across any application window, with fluid transitions between pointer/keyboard interfaces and multitouch interfaces is tricky. And of course, we need to stay compatible with the core protocol and its extensions. Funnily enough, many of the issues we face with multitouch are similar to the ones I faced years ago during the MPX development phase. This is one reason why I am more confident but also quite cautious about it, I already know how it can go wrong.
The basic requirements we have are:
- Multitouch in any application that supports it
- Multitouch in any number of applications simultanously
- Pointer emulation for non-multitouch applications
- Simultaneous execution of multitouch and pointer/keyboard applications
Replace multitouch with multiple users and you have exactly the same requirements I had for MPX (and XI2). Why exactly these four requirements I'll explain separately one day, but the short answer is: zero transition cost between multitouch and non-multitouch interfaces.
Gestures
Whenever multitouch is mentioned, gestures are brought up quickly too. Gestures are prevalent in this context because they appear to be a natural extension of touch input. They are, but only to a very limited extent. The set of natural gestures is surprisingly small and goes barely beyond the well-known tap, pinch, rotate and swipe. As an exercise, think of the natural gestures for copy and paste. Then see how many of your friends come up with exactly the same gesture.
Moreover, gesture discussions often focus on multi-finger gestures. In the not-too-distant future off-the-shelf hardware will support arbitrary object shapes and that's when hand and arm gestures will become relevant too.
Gestures do not rule out touch. Especially on direct-touch devices it is likely that a UI will use gestures as shortcuts to complex operations but the vast majority of interactions just use plain touch input for object selection, moving objects, and invoking menu commands. So it is important to think of gestures as an abstraction of some touch input but it is by no means the sole vehicle of deliverying touch input. From a technical point of view, this means that gesture recognition must not be in the way of touch recognition.
Gesture detection introduces a delay and that is unavoidable. What can be avoided is the effect of this delay on the UI. With good gesture design the delay is irrelevant. Take the example of a system with two gestures: a one finger hold gesture and a four finger hold gesture (i.e. an action is triggered when one or four fingers are put onto a position). Assume that the timeout for the gesture is 2 seconds. When a user puts a single finger on the device, the system now has to wait 2 seconds before it can react. If within 2 seconds the user does not put three more fingers down, the single-finger action can be taken. Decreasing the timeout lands the UI designer at the blurry line between a laggy UI and too-hard-to-trigger gestures.
To solve this issue, the system must be able to recognize which gesture is performed before the user expects feedback. Consider the gesture set of tapping one finger, moving one finger and a four finger hold gesture. Tapping and moving are easy to detect and they don't suffer from an artificial delay. The only time the timeout is needed is when the user puts a finger down but doesn't move or release it immediately. Now the system has to wait for more fingers. However, since a single finger not moving is not a defined gesture, the user expects no feedback and the timeout is irrelevant.
Integrating gesture support is thus not a matter of simply defining a few gestures. It's a matter of selecting the right ones that do not expose the user to a laggy experience. The FOSS software stack is notoriously segregated and thus true gesture integration must be done across window managers, desktop environments and applications. Coincidentally, this is my main worry with the Unity gesture support, I feel the gestures have a potential to overlap in some cases and thus be detrimental to the user experience. I hope I'm wrong here.
X.Org multitouch support
When it comes to input event hanlding, the X server's job is to convert a set of hardware events into a set of events meaningful in a GUI. This includes converting the hardware information into screen-relative information and picking (selecting where to send the event to). I've already outlined the four requirements for multitouch above.
Originally, I thought multitouch could be integrated into our current event system but a few months back I gave up on that. Whenever I tried to sort out the corner cases, there were some situations where the core protocol would be unresolvable. The current aim is that we add new events and they will live side-by-side to pointer and keyboard events, not necessarily influencing those. This means that clients must add support for multitouch to be useable, even XI2 enabled clients will not provide that functionality without modifications (pointer emulation for single-finger touches is still considered as required feature though).
Beyond that, things are still in flux. At the time of this writing, there are two protocol proposals that are largely overlapping. Daniel Stone's version has taken a few features from mine as I have taken some of his but the basics approach is the same (and we came up with it independently, which is a good sign). Daniel's proposal is also backed by an implementation though I have yet to look at these patches. Either way, what will be happening is a new set of events and a few requests to detect which devices have multitouch. Passive grabs on touch points will be possible but that still needs to be sorted out. Once it is, I'll post more details.
The summary here is that things are moving, at least in my case largely due to Canonical's announcement which lit a fire under some southern body parts and finally made me spend more time thinking about multitouch.
Ubuntu's multitouch support
One thing that is likely of interest to many is how all this will integrate with Ubuntu's multitouch support. Ubuntu has committed to strong gesture support. This is commendable but quite limiting in my opinion as true multitouch is much more than gestures. The current UI seems aimed more at dependent-touch devices like the Apple Magic Trackpad where gestures agument traditional input, less so at a fully multitouch-aware desktop. Fair enough, so far there aren't a lot of multitouch applications out there but for system-level design we need to keep the medium and long term in mind too.
The current integration is a hack (and the developers don't try to hide this), but most good things start as hacks so there's not much to say here. I am certianly impressed that they are willing to deliver a UI based on what is still a moving target. The few issues I had were with the design, so let's focus on that.
The originally proposed X Gesture Extension tried to get a recogniser into the server (not a 100% accurate statement, but close enough). It only allowed for one recogniser in the system. I do not think that we'll ever manage to find the one true gesture recognizer, we haven't managed to find the one true window manager or toolkit either. I talked at length to Chase about this and the revised architecture diagrams Chase presented at the workshop and XDS do not call for gesture recognition in the server anymore.
With the new approach the recognition happens purely client-side and is done by a library or a daemon. This allows for multiple different gesture recognisers be active at the same time, a scenario that is quite likely to happen (I do envision GTK, Qt, Mozilla, etc. all wanting their own system). Whether that's a good thing for the UI is another matter, consistency for gestures is important and especially Ping Cheng was not happy at the prospect of having multiple, possibly inconsistent, systems. We need to find some common ground here between desktop environments and toolkits. Geis tries to address this, though it is in a quite early stage. The main issue with geis is a political one. The implementation requires copyright assignment and puts up quite a barrier to entry. The utouch team tries to get around this by separating the interface specification and the implementation, but that's just calling for a fork or duplicated efforts before there's anything to fork or duplicate. So while I think geis is the right approach and much needed, I fear the political decisions will hold it back unnecessarily. My suggestion is to ditch the copyright assignment, because something as central as a generic gesture interface should be developed together with toolkit developers, X.Org developers, application developers, etc. And that's just easier when contributions are as simple as possible.
As for the UI itself - any UI decisions already made will unlikely be an issue from a technical perspective. Gestures are an important part of multitouch support and we will make it possible to provide gestures. All the technical parts are still in flux, but the current abstractions Ubuntu has put in place are likely good enough to adjust to a new baseline. So while I cannot speak for Ubuntu at all, my guess is that in future versions the bottom bits of the stack will be swapped out but the top bits (the user-visible ones) will largely remain the same.
Conclusion
The above should provide a general overview on some of the issues with multitouch and challenges we face. I've skipped over a few and only briefly touched on others for brevity, it's quite easy to write full novels about all this.
I will post more technical details about the kernel's multitouch API and how we plan to use it in X.Org for our protocol extensions once the latter is at least partially nailed down. For now, the concepts described above hopefully framed your mind in terms of multitouch and some of the concepts we're looking at.
Wednesday, September 8, 2010
Presentation in Toulouse
I'll be giving a talk next Tuesday, 14 September in Toulouse, France. If you happen to be around that area, more information is available here:
http://www.toulibre.org/
http://www.linuxfr.org/2010/09/07/27352.html
http://www.toulibre.org/
http://www.linuxfr.org/2010/09/07/27352.html
Friday, September 3, 2010
Wacom support in Linux
This article should have been written a long long time ago. Anyway, here's the gist of Wacom tablet support under Linux and that hopefully clears up the confusion about the various packages.
linuxwacom is two things. It is the project name used for wacom-specific implementations and thus in the URL as well (in this post, I will always use "linuxwacom project" to refer to the project).
It is also the name of the tarball shipped by the linuxwacom project. Or at least the name of one tarball, anyway. Now, this tarball includes virtually everything linuxwacom as a project wants to provide, most notably a kernel driver, an X driver and several utilities. More on these later.
The main (and long-term) contributor to linuxwacom is still Ping Cheng, a Wacom employee. Wacom naturally has a strong focus on its customers and thus linuxwacom goes into some pains supporting systems all the way back to XFree86-4.
However, the X server has moved quite a bit.
In August 2009, I split out the X11 driver from the linuxwacom package and imported it into a git repository called xf86-input-wacom. This driver has seen quite a bit of rework, including dropping of some features (like XFree86-4 support but also old-style serial tablets) and gaining some features (like input device properties). The driver includes a simple tool called xsetwacom that provides some options to manipulate driver settings at runtime. xsetwacom used to be part of the linuxwacom tarball and I've tried to keep xsetwacom's interface as close to the original as possible.
As of a couple of weeks ago, the xf86-input-wacom driver also includes a simple isdv4-serial-debugger to print information about serial tablets. That's it though, xf86-input-wacom does not contain any other tools.
xf86-input-wacom supports serial devices directly and USB devices provided the kernel supports them. In that regard, it is quite similar to evdev or synaptics - both rely on the kernel for actual hardware support. If your kernel doesn't support a particular device, then xf86-input-wacom won't help.
Note that xf86-input-wacom is the replacement for the linuxwacom X driver, that driver will not compile on X servers 1.7 or later. And the name does not imply that it is for Wacom tablets only, we have code to handle some Waltop tablets and recently added a patch for Fujitsu serial devices. More devices will be added as needed.
The linuxwacom tarball ships with several kernel drivers for various kernel versions. Note that these are out-of-tree drivers and thus if you compile them and then update your kernel, you must recompile and re-install the driver. This out-of-tree development is mostly due to Wacom's commitment to its customers.
Personally I would love to see kernel development happening on LKML only but this is not the case right now. Especially for newer models that support touch the upstream kernel requires the use of the MT protocol, something that the X drivers (neither xf86-input-wacom nor the old linuxwacom one) can handle right now. Especially Bamboos are affected by this. AFAIK, no distribution supports them out-of-the box and users must compile the kernel driver from the linuxwacom tarball. Once that driver is installed, xf86-input-wacom works fine with it.
This is a source of much confusion but while I intended to start working at the kernel drivers, I have enough things on my slate and I have yet to actually do so. Luckily, Henrik and others are working on it.
However, when I read "You may be tempted to download the 0.10.8 version if you have an X server >= 1.7. I tried this, but it did not work. Stick with the production driver, it will work." and (after compiling the kernel driver) "Now, plug in your tablet. It SHOULD be recognized and immediately function as a mouse. " on linux.com I cringed. What happens here is that after the kernel driver from the linuxwacom tarball was installed, xf86-input-wacom automatically picked it up. You need both parts, kernel and X driver.
All these are still available in the linuxwacom tarball. wacdump provides similar functionality to evtest so I don't see the need for maintaining two tools that do the same thing. xidump does the same as xinput --test. Hence both tools are not available anymore on newer distributions.
wacomcpl is a major source of headache. It's a TCL wrapper around xsetwacom to provide a basic graphical configuration utility. It doesn't work anymore due to a number of subtle changes in the xsetwacom interface but also because it partially expects options that don't exist anymore (or in other forms) in the xf86-input-wacom driver. I thought about fixing it up but after looking at it I ran away screaming. I've traded some bugs with Bastien and he'll write a GNOME configuration utility. That's currently holding up on the need for an actual UI (and as engineers we both shouldn't do that one ourselves...) but it will hopefully see the light at some point in the near future. Meanwhile, no GUI configuration tool.
The linuxwacom project is still largely controlled by Wacom, with the exception of xf86-input-wacom that's maintained by me but is heavily contributed to by Ping Cheng, both in patches and code review.
Wacom's internal processes are different to the ones we use in distributions and upstream and that can make life hard for us. (Likewise, our processes make life harder for Wacom so the finger-pointing goes both ways :) Ping and I have had plenty of discussions about this and we try to meet somewhere in the middle where possible, disrupting the project completely helps no-one.
xf86-input-wacom is in a reasonably good state now though the cleanup has taken much longer than expected and there are still several items to be removed or rewritten to fit into the new X server behaviours. I expect this work to take another couple of months but much of it is aimed ad making the driver easier to maintain and develop for. For the average user, it won't have as much impact.
As I said above, I'd like to see kernel development happen on LKML in upstream kernels, not in a separate tarball. This is - aside from the GUI tool - the most pressing issue right now.
As usual, the best way of helping out is to test the code that is available. I'm not a designer so if the tablet doesn't crash the server that's pretty close to 100% functional to me. I rely on others to tell me when behaviours change.
If you're a coder, get right in. If your device doesn't work, go for the upstream kernel and try to get it working there. Anything upstream will feed back into distros and thus improve the out-of-the-box experience. If you have feature-requests for xf86-input-wacom, dig in and write it. The developer's mailing list is useful in any case. Finally, there's the wiki that will eventually be a useful one, but I keep getting distracted with other stuff whenever I try to move info over. So any help there will be useful to others looking for documentation.
And as I pointed out above - we're in desparate need for a UI for the GNOME config utility. If you're interested in helping out here, please contact me.
Finally, I want to point out that Ping (and by inference Wacom Inc.) has been excellent and though we sometimes disagree on technical issues there is no question that Wacom is supportive of the project and their contributions are invaluable.
The role of linuxwacom
linuxwacom is two things. It is the project name used for wacom-specific implementations and thus in the URL as well (in this post, I will always use "linuxwacom project" to refer to the project).
It is also the name of the tarball shipped by the linuxwacom project. Or at least the name of one tarball, anyway. Now, this tarball includes virtually everything linuxwacom as a project wants to provide, most notably a kernel driver, an X driver and several utilities. More on these later.
The main (and long-term) contributor to linuxwacom is still Ping Cheng, a Wacom employee. Wacom naturally has a strong focus on its customers and thus linuxwacom goes into some pains supporting systems all the way back to XFree86-4.
However, the X server has moved quite a bit.
xf86-input-wacom
In August 2009, I split out the X11 driver from the linuxwacom package and imported it into a git repository called xf86-input-wacom. This driver has seen quite a bit of rework, including dropping of some features (like XFree86-4 support but also old-style serial tablets) and gaining some features (like input device properties). The driver includes a simple tool called xsetwacom that provides some options to manipulate driver settings at runtime. xsetwacom used to be part of the linuxwacom tarball and I've tried to keep xsetwacom's interface as close to the original as possible.
As of a couple of weeks ago, the xf86-input-wacom driver also includes a simple isdv4-serial-debugger to print information about serial tablets. That's it though, xf86-input-wacom does not contain any other tools.
xf86-input-wacom supports serial devices directly and USB devices provided the kernel supports them. In that regard, it is quite similar to evdev or synaptics - both rely on the kernel for actual hardware support. If your kernel doesn't support a particular device, then xf86-input-wacom won't help.
Note that xf86-input-wacom is the replacement for the linuxwacom X driver, that driver will not compile on X servers 1.7 or later. And the name does not imply that it is for Wacom tablets only, we have code to handle some Waltop tablets and recently added a patch for Fujitsu serial devices. More devices will be added as needed.
kernel drivers
The linuxwacom tarball ships with several kernel drivers for various kernel versions. Note that these are out-of-tree drivers and thus if you compile them and then update your kernel, you must recompile and re-install the driver. This out-of-tree development is mostly due to Wacom's commitment to its customers.
Personally I would love to see kernel development happening on LKML only but this is not the case right now. Especially for newer models that support touch the upstream kernel requires the use of the MT protocol, something that the X drivers (neither xf86-input-wacom nor the old linuxwacom one) can handle right now. Especially Bamboos are affected by this. AFAIK, no distribution supports them out-of-the box and users must compile the kernel driver from the linuxwacom tarball. Once that driver is installed, xf86-input-wacom works fine with it.
This is a source of much confusion but while I intended to start working at the kernel drivers, I have enough things on my slate and I have yet to actually do so. Luckily, Henrik and others are working on it.
However, when I read "You may be tempted to download the 0.10.8 version if you have an X server >= 1.7. I tried this, but it did not work. Stick with the production driver, it will work." and (after compiling the kernel driver) "Now, plug in your tablet. It SHOULD be recognized and immediately function as a mouse. " on linux.com I cringed. What happens here is that after the kernel driver from the linuxwacom tarball was installed, xf86-input-wacom automatically picked it up. You need both parts, kernel and X driver.
wacdump, wacomcpl, and other tools
All these are still available in the linuxwacom tarball. wacdump provides similar functionality to evtest so I don't see the need for maintaining two tools that do the same thing. xidump does the same as xinput --test. Hence both tools are not available anymore on newer distributions.
wacomcpl is a major source of headache. It's a TCL wrapper around xsetwacom to provide a basic graphical configuration utility. It doesn't work anymore due to a number of subtle changes in the xsetwacom interface but also because it partially expects options that don't exist anymore (or in other forms) in the xf86-input-wacom driver. I thought about fixing it up but after looking at it I ran away screaming. I've traded some bugs with Bastien and he'll write a GNOME configuration utility. That's currently holding up on the need for an actual UI (and as engineers we both shouldn't do that one ourselves...) but it will hopefully see the light at some point in the near future. Meanwhile, no GUI configuration tool.
Future progress
The linuxwacom project is still largely controlled by Wacom, with the exception of xf86-input-wacom that's maintained by me but is heavily contributed to by Ping Cheng, both in patches and code review.
Wacom's internal processes are different to the ones we use in distributions and upstream and that can make life hard for us. (Likewise, our processes make life harder for Wacom so the finger-pointing goes both ways :) Ping and I have had plenty of discussions about this and we try to meet somewhere in the middle where possible, disrupting the project completely helps no-one.
xf86-input-wacom is in a reasonably good state now though the cleanup has taken much longer than expected and there are still several items to be removed or rewritten to fit into the new X server behaviours. I expect this work to take another couple of months but much of it is aimed ad making the driver easier to maintain and develop for. For the average user, it won't have as much impact.
As I said above, I'd like to see kernel development happen on LKML in upstream kernels, not in a separate tarball. This is - aside from the GUI tool - the most pressing issue right now.
How to help
As usual, the best way of helping out is to test the code that is available. I'm not a designer so if the tablet doesn't crash the server that's pretty close to 100% functional to me. I rely on others to tell me when behaviours change.
If you're a coder, get right in. If your device doesn't work, go for the upstream kernel and try to get it working there. Anything upstream will feed back into distros and thus improve the out-of-the-box experience. If you have feature-requests for xf86-input-wacom, dig in and write it. The developer's mailing list is useful in any case. Finally, there's the wiki that will eventually be a useful one, but I keep getting distracted with other stuff whenever I try to move info over. So any help there will be useful to others looking for documentation.
And as I pointed out above - we're in desparate need for a UI for the GNOME config utility. If you're interested in helping out here, please contact me.
Finally, I want to point out that Ping (and by inference Wacom Inc.) has been excellent and though we sometimes disagree on technical issues there is no question that Wacom is supportive of the project and their contributions are invaluable.
XTS census
Tiago's X 1.9 Census includes the commits to the X Test Suite for the 1.9 window. IMO, this isn't particularly expressive since XTS follows its own cycle and the server merge windows have no meaning to it.
The short history of XTS is:
The X Test Suite was written back in the day of yonder (possibly more than two yonks ago) and it is designed to perform a number of protocol tests to verify whether the server has changed behaviour. This includes simple things like creating a window and making sure the MapNotify is sent through to more complex interactions with modifiers and grabs.
XTS was still in CVS until 2009 when I wanted to figure out how much I really broke with MPX. So I imported it into git and started autotooling it. XTS is a bit of a beast and at some point I gave up and asked Dan Nicholson for help; his autotooling skills exceed mine by some unmeasurable amount. A few months later and some 80-something commits later Dan came back with a fully autotooled XTS. So building XTS went from requiring a wiki page to git clone, autogen.sh, make.
From then on, we continued working on making it more sane and easier to run, along with misc code cleanups (DECNET support? Really?) and one of my favourite issues: that after 4 hours of runtime one realised that every test had failed because DISPLAY wasn't set. Anyway, I digress.
We found a couple of issues in the X server as well, with 1.7.7, 1.8.1 and 1.9.0 being the first three releases that pass the test suite without crashing.
XTS is still a bit from being really useful, at the moment many of the test results are write-only (though a few test errors have fed back into the server or even libxcb). Anyway, here are the raw numbers from gitdm (that 1 employer is Unknown because I couldn't be bothered setting up a gitdm.config file).
Note: 1024 of my changesets are a semi-automatic rename of files and that skews the statistic.
The point of this blog post? Give credit to Dan for his magnificent work on XTS because he's been carrying the load for quite a while. And of course, many thanks to Aaron, Jon and Pat for their contributions.
The short history of XTS is:
The X Test Suite was written back in the day of yonder (possibly more than two yonks ago) and it is designed to perform a number of protocol tests to verify whether the server has changed behaviour. This includes simple things like creating a window and making sure the MapNotify is sent through to more complex interactions with modifiers and grabs.
XTS was still in CVS until 2009 when I wanted to figure out how much I really broke with MPX. So I imported it into git and started autotooling it. XTS is a bit of a beast and at some point I gave up and asked Dan Nicholson for help; his autotooling skills exceed mine by some unmeasurable amount. A few months later and some 80-something commits later Dan came back with a fully autotooled XTS. So building XTS went from requiring a wiki page to git clone, autogen.sh, make.
From then on, we continued working on making it more sane and easier to run, along with misc code cleanups (DECNET support? Really?) and one of my favourite issues: that after 4 hours of runtime one realised that every test had failed because DISPLAY wasn't set. Anyway, I digress.
We found a couple of issues in the X server as well, with 1.7.7, 1.8.1 and 1.9.0 being the first three releases that pass the test suite without crashing.
XTS is still a bit from being really useful, at the moment many of the test results are write-only (though a few test errors have fed back into the server or even libxcb). Anyway, here are the raw numbers from gitdm (that 1 employer is Unknown because I couldn't be bothered setting up a gitdm.config file).
Note: 1024 of my changesets are a semi-automatic rename of files and that skews the statistic.
Processed 1275 csets from 5 developers
1 employers found
A total of 33380 lines added, 30758 removed (delta 2622)
Developers with the most changesets
Peter Hutterer 1061 (83.2%)
Dan Nicholson 192 (15.1%)
Aaron Plattner 12 (0.9%)
Jon TURNEY 4 (0.3%)
Tinderbox user 4 (0.3%)
Developers with the most changed lines
Dan Nicholson 20545 (50.0%)
Peter Hutterer 10296 (25.0%)
Aaron Plattner 3854 (9.4%)
Tinderbox user 655 (1.6%)
Jon TURNEY 30 (0.1%)
Developers with the most lines removed
Aaron Plattner 2000 (6.5%)
Developers with the most signoffs (total 5)
Dan Nicholson 3 (60.0%)
Peter Hutterer 2 (40.0%)
Developers with the most reviews (total 10)
Dan Nicholson 7 (70.0%)
Peter Hutterer 3 (30.0%)
Developers with the most test credits (total 1)
Pat Kane 1 (100.0%)
Developers who gave the most tested-by credits (total 1)
Dan Nicholson 1 (100.0%)
The point of this blog post? Give credit to Dan for his magnificent work on XTS because he's been carrying the load for quite a while. And of course, many thanks to Aaron, Jon and Pat for their contributions.
Tuesday, August 31, 2010
The antipattern of hate
Code changed to protect the guilty, but the approach is the same. Also, this pseudo-code below is actually more streamlined, more sensible and slightly saner than the original.
If you think of writing code like that, don't. Google for function parameters and try to understand how they work. Some poor sod will eventually have to figure out wtf you're trying to do [1]. I've encountered this bit of code a few weeks back and I am still flabbergasted.
/me hopes this venting makes the anger go away.
[1] as I stated in an early post, any software project is a collaborative project. It has at least two developers [...]. The person you may end up hating for the above may be you.
static char* array[10];
static int other_array[10];
void get_values(int n)
{
for (i = 0; i < n; i++)
other_array[i] = get_val(array[i]);
}
int main (void) {
array[0] = "somestring";
array[1] = "some other string"
array[2] = "lalala"
array[3] = "bananas"
get_values(4);
for (i = 0; i < 4; i++)
blah(other_array[i]);
return 0;
}
If you think of writing code like that, don't. Google for function parameters and try to understand how they work. Some poor sod will eventually have to figure out wtf you're trying to do [1]. I've encountered this bit of code a few weeks back and I am still flabbergasted.
/me hopes this venting makes the anger go away.
[1] as I stated in an early post, any software project is a collaborative project. It has at least two developers [...]. The person you may end up hating for the above may be you.
Friday, August 6, 2010
First draft of multitouch protocol spec published
In case you're not subscribed to xorg-devel, I've sent off the first draft for the multitouch X protocol specification (as part of the X Input Extension 2.1) today. If you have interest in multitouch, please read through it and see if you can poke holes into the current approach.
http://lists.freedesktop.org/archives/xorg-devel/2010-August/011759.html
Note that this is very low-level stuff. The data described by the protocol is what's sent on the wire to the client. Most applications are separated from this data by toolkits (GTK, Qt, etc.) and only few people are unlucky enough to have to care about the protocol itself.
http://lists.freedesktop.org/archives/xorg-devel/2010-August/011759.html
Note that this is very low-level stuff. The data described by the protocol is what's sent on the wire to the client. Most applications are separated from this data by toolkits (GTK, Qt, etc.) and only few people are unlucky enough to have to care about the protocol itself.
Subscribe to:
Posts (Atom)