Showing posts with label xi2. Show all posts
Showing posts with label xi2. Show all posts

Thursday, September 23, 2021

What's new in XI 2.4 - touchpad gestures

After a nine year hiatus, a new version of the X Input Protocol is out. Credit for the work goes to Povilas Kanapickas, who also implemented support for XI 2.4 in the various pieces of the stack [0]. So let's have a look.

X has had touch events since XI 2.2 (2012) but those were only really useful for direct touch devices (read: touchscreens). There were accommodations for indirect touch devices like touchpads but they were never used. The synaptics driver set the required bits for a while but it was dropped in 2015 because ... it was complicated to make use of and no-one seemed to actually use it anyway. Meanwhile, the rest of the world moved on and touchpad gestures are now prevalent. They've been standard in MacOS for ages, in Windows for almost ages and - with recent GNOME releases - now feature prominently on the Linux desktop as well. They have been part of libinput and the Wayland protocol for years (and even recently gained a new set of "hold" gestures). Meanwhile, X was left behind in the dust or mud, depending on your local climate.

XI 2.4 fixes this, it adds pinch and swipe gestures to the XI2 protocol and makes those available to supporting clients [2]. Notably here is that the interpretation of gestures is left to the driver [1]. The server takes the gestures and does the required state handling but otherwise has no decision into what constitutes a gesture. This is of course no different to e.g. 2-finger scrolling on a touchpad where the server just receives scroll events and passes them on accordingly.

XI 2.4 gesture events are quite similar to touch events in that they are processed as a sequence of begin/update/end with both types having their own event types. So the events you will receive are e.g. XIGesturePinchBegin or XIGestureSwipeUpdate. As with touch events, a client must select for all three (begin/update/end) on a window. Only one gesture can exist at any time, so if you are a multi-tasking octopus prepare to be disappointed.

Because gestures are tied to an indirect-touch device, the location they apply at is wherever the cursor is currently positioned. In that, they work similar to button presses, and passive grabs apply as expected too. So long-term the window manager will likely want a passive grab on the root window for swipe gestures while applications will implement pinch-to-zoom as you'd expect.

In terms of API there are no suprises. libXi 1.8 is the version to implement the new features and there we have a new XIGestureClassInfo returned by XIQueryDevice and of course the two events: XIGesturePinchEvent and XIGestureSwipeEvent. Grabbing is done via e.g. XIGrabSwipeGestureBegin, so for those of you with XI2 experience this will all look familiar. For those of you without - it's probably no longer worth investing time into becoming an XI2 expert.

Overall, it's a nice addition to the protocol and it will help getting the X server slightly closer to Wayland for a widely-used feature. Once GTK, mutter and all the other pieces in the stack are in place, it will just work for any (GTK) application that supports gestures under Wayland already. The same will be true for Qt I expect.

X server 21.1 will be out in a few weeks, xf86-input-libinput 1.2.0 is already out and so are xorgproto 2021.5 and libXi 1.8.

[0] In addition to taking on the Xorg release, so clearly there are no limits here
[1] More specifically: it's done by libinput since neither xf86-input-evdev nor xf86-input-synaptics will ever see gestures being implemented
[2] Hold gestures missed out on the various deadlines

Thursday, January 24, 2013

Proposed XI2.3 addition: XIGetSupportedVersion

Update March 7 2013: This addition was not merged into XI 2.3, largely because there is no real need for it. XI 1.x' XGetExtensionVersion() returns the server version without locking in a client version and at this point there was no perceived need for getting the already-requested client version back. I'll leave this here for archival purposes but again, this request was not merged into XI 2.3

Original post below

Posting this here too to get a bit more exposure.

XIQueryVersion(3) is the first XI2 request clients should send to the server. The client announces its supported version and in return receives the server version (which is always less or equal to the client, never higher).

As XI 2.1 - 2.3 progressed, we started using this information in the server. Clients are treated slightly differently depending on their announced version. The current differences are:

  • XIQueryPointer will not set the button 1 mask for pointer-emulated events if the client supports XI 2.2 or newer.
  • XIAllowEvents will allow XIRejectTouch and XIAcceptTouch for clients supporting XI 2.2 or newer.
  • Raw event delivery changes if a client supports XI 2.1 or newer.
The client can issue multiple XIQueryVersion requests, but they need to have the same version numbers to provide for consistent server behaviour.

So far, so good. This works fine as long as the client supports one specific version. However, as toolkits like GTK have come to support XI2, the requirements changed a bit. An application and its toolkit usually look like a single client to the server. However, the client may support XI 2.0, but the toolkit may support XI 2.3. And neither knows of the other's version support. If the client requests XIQueryVersion before the toolkit, the toolkit is locked into the client version. But if the toolkit first requests XIQueryVersion, the client is locked into the version supported by the toolkit. Worst case the client may get a BadValue and quit because it may not be built for this case.

Jasper St. Pierre and Owen Taylor brought this up on #xorg-devel today, and I've send a proposed solution to the mailing list.

A new XIGetSupportedVersion request simply returns the server's major/minor version number. Uncapped, so really what the server supports. And the same request also returns the client version previously announced with XIQueryVersion. Or zero, if the client hasn't called it yet.

This request enables toolkits to query what the client has already set, and of course what the server supports without modifying the client state. The request is currently an RFC, but I do hope we may get this into XI 2.3.

If you're working on XI2-aware clients or toolkits and you have a use-case that requires this or would break by this addition, please speak up now.

Monday, December 10, 2012

What's new in XI 2.3 - Pointer Barrier events and barrier releases

Update 7 March 2013: XI 2.3 is released, amend accordingly

The main feature making up XI 2.3 is Pointer Barrier events. This feature was initiated by Chris Halse-Rogers and then taken up by Jasper St. Pierre and me, until it became part of XI 2.3.

The usual conditions apply: a client must announce XI2.3 support with the XIQueryVersion() request to utilise any of the below.

Pointer barriers

Pointer Barriers are an XFixes v5.0 additions to restrict pointer movement. They were first available in server 1.11 (released Aug 2011). A pointer barrier is created by a client along a specific horizontal or vertical line of pixels. Relative input devices such as mice or touchpads will be constrained by this barrier, preventing the cursor from moving across the barrier.

For example, GNOME3 creates a vertical barrier on the top-left screen edge. In a multi-monitor screen, flicking the pointer up to the Activities menu will constrain the pointer there even if there is a screen to the left of the menu. This makes the menu much easier to hit.

The client can choose for the barrier to be transparent in specific directions. GNOME3's pointer barrier is permissive in the direction of positive X (i.e. left-to-right). Thus, moving from the left screen to the right is unrestricted, even though the pointer is constrained when moving in the other direction.

A simple client that creates a vertical pointer barrier looks like this:

Display *dpy = XOpenDisplay(NULL);
int fixes_opcode, fixes_event_base, fixes_error_base;
PointerBarrier barrier;

if (!XQueryExtension(dpy, "XFIXES",
                     &fixes_opcode,
                     &fixes_event_base,
                     &fixes_error_base))
    return EXIT_FAILURE;

/* vertical barrier from 20/20 to 20/100 */
barrier = XFixesCreatePointerBarrier(dpy, DefaultRootWindow(dpy),
                                     20, 20,
                                     20, 100,
                                     0, /* block in all directions */
                                     0, NULL); /* no per-device barriers */

mainloop();

The above code will set up a barrier for all devices, blocking in all directions.

Pointer barrier events

As of XI 2.3, pointer barrier events are sent to clients when pointer movement is constrained by a barrier, provided that client created pointer barriers, the clients have the matching event mask set. Two new event types were added, XI_BarrierHit and XI_BarrierLeave. Both events are only sent to clients owning a barrier and are only sent to the window used to create the barrier (the root window in the example above).

unsigned char m[XIMaskLen(XI_BarrierLeave)] = {0};
XIEventMask mask;
mask.deviceid = XIAllMasterDevices;
mask.mask_len = XIMaskLen(XI_BarrierLeave);
mask.mask = m;

XISetMask(mask.mask, XI_BarrierHit);
XISetMask(mask.mask, XI_BarrierLeave);
XISelectEvents(dpy, DefaultRootWindow(dpy), &mask, 1);
XSync(dpy, False);

while (1) {
    XEvent ev;
    XNextEvent(dpy, &ev);
    if (ev.type != GenericEvent || ev.xcookie.extension != xi2_opcode)
       continue;

    XGetEventData(dpy, &ev.xcookie);
    XIBarrierEvent *b = ev.xcookie.data;
    if (b->evtype == XI_BarrierHit)
       printf("Pointer hit the barrier\n");
    else if (b->evtype == XI_BarrierLeave)
       printf("Pointer left the barrier\n");
    
    XFreeEventData(dpy, &ev.xcookie);
}

An XI_BarrierHit event is first sent when the pointer movement is first constrained by a barrier. It includes some information such as the device, the barrier and the window. It also includes coordinate data.

XI_BarrierHit events are sent for each movement of the pointer against this barrier, or along it, until the pointer finally moves away from the barrier again. "Moving away" means a move to different position that is not blocked by the barrier.

  • if the barrier is vertical, the pointer moves to a different X axis value, or
  • if the barrier is horizontal, the pointer moves to a different Y axis value, or
  • the pointer moves past the barrier's end point
Once the pointer does move away, a XI_BarrierLeave event is sent. A pointer that moves against a barrier, pushes against it for 3 more events and then pulls back will thus generate 4 XI_BarrierHit events and one XI_BarrierLeave event.

Who gets events? Always the client that created the barrier, and only if the window has the event mask set. If the client has a grab on the device with the grab_window being the barrier window, the barrier events follow the usual grab event mask behaviour:

  • if the grab event mask has XI_BarrierHit set, the event is delivered
  • if the grab event mask does not have XI_BarrierHit set but the window mask does and owner_events is True, the event is delivered
  • if owner_events is False and the grab mask does not have XI_BarrierHit set, no event is sent
The above applies to XI_BarrierLeave events as well.

If the client's grab has a grab_window different to the barrier window, or the device is grabbed by another client, event delivery is as usual. In all cases, if the device is grabbed, the XIBarrierDeviceIsGrabbed flag is set. Clients should use this flag to determine what to do. For example, the barrier that is used to trigger the GNOME overlay should probably not trigger if another client has a grab as it may interfere with drag-and-drop.

Coordinates and time in pointer barrier events

Barrier events contain two sets of x/y coordinates. First, the root coordinates which represent the position of the pointer after being confined by barrier (and screen extents, where applicable). This coordinate is the same you would get from a subsequent XIQueryPointer request.

The second set are the delta coordinates (dx/dy), in screen coordinates, from the last pointer position, had the barrier not constrained it. So you can calculate how much the pointer would have moved and thus derive speed. The dtime field in the event helps you to calculate speed, it provides the time in milliseconds since the last pointer event. The deltas are calculated after taking pointer acceleration into account.

    XIBarrierEvent *b = ev.xcookie.data;
    double dx, dy;
    double speed;
    unsigned int millis;

    dx = b->dx;
    dy = b->dy;
    millis = b->dtime;

    speed = sqrt(dx * dx + dy * dy) / millis * 1000;

    printf("Barrier was hit at %.2f/%.2f at %.2f pixels/sec\n",
           b->root_x, b->root_y,  speed);

Releasing a pointer from barrier constraints

By default, a pointer barrier blocks all movement of relative input devices across a barrier. However, a client can opt to temporarily release the pointer from the barrier constraints with the XIBarrierReleasePointer request.

To do so, the client needs the event ID of the barrier event. Since a pointer may bump against the same barrier multiple times before the client reacts (X is asynchronous, after all), the event ID serves to identify a set of movements against or along the pointer barrier.

An event ID is assigned to the first XI_BarrierHit event, and then it remains the same until the XI_BarrierLeave event. This event is the last event with the current event ID, any future barrier events will have a new event ID. This approach may be familiar to you from dealing with touch events, that use a similar approach (touch IDs start at TouchBegin and are tracked through to the TouchEnd).

To release a pointer and let it pass through the barrier, call XIBarrierReleasePointer().

    XIBarrierEvent *b = ev.xcookie.data;
    ...
    if (speed > 200) {
        printf("Movement exceeds speed limit, allowing pointer to go through\n")
        XIBarrierReleasePointer(dpy, b->deviceid, b->barrier, b->eventid);
        XFlush(dpy);
    }
If, when the request arrives at the server, the pointer is still trapped by the barrier, the barrier is now transparent and the pointer can move through it with the next movement. If the pointer moves away from the barrier after releasing it and later moves against this barrier again, it will be constrained once more (albeit with a different eventid). Likewise, if the pointer has already moved away and against the barrier again before the client reacted, the release request has no effect.

If the release does succeed and the pointer moves through the barrier, the client gets a XI_BarrierLeave event with the XIBarrierPointerReleased flag.

The pointer barrier hit-box

As mentioned above, a XI_BarrierLeave event is sent when the pointer moves away from the barrier. This would usually require a 1 pixel movement (let's ignore subpixel-movement, our hand's aren't that precise). However, during testing we found that 1 px pointer-movement can still happen even when a user tries hart to move along the barrier, or even when pushing against the barrier. Thus, we implemented a hit-box of 2 pixels. Thus, a 1 px movement along the barrier still counts as hitting the barrier, and the pointer is not treated as having left the barrier until it leaves the hit-box.

Testing the current state

The current code is available in the barriers branch of the following repositories:

git://people.freedesktop.org/~whot/inputproto.git 
git://people.freedesktop.org/~whot/libXi.git 
git://people.freedesktop.org/~whot/xserver.git 
An example program to test the feature is available here

Tuesday, June 12, 2012

XI 2.1 protocol design issues

When XI 2.1 smooth scrolling was released, we were glad to finally have a way for fluid scrolling events. Unfortunately, two design issues have since been discovered that make it harder for applications than necessary. Mea culpa, I apologise, these are things that we should have caught in the review process.

Scrolling may overflow the axis range

Scrolling events are now posted through axes marked as scrolling axes. This means they're limited to the 32.32 fixed point value range that valuators can take. That range matters because despite scrolling being relative, coordinates in XIDeviceEvents must always be absolute values. Scrolling down is more common than scrolling up, so the axis will slowly accumulate until it reaches the maximum range. We have to reset them to 0 once we reach INT_MAX to avoid overflow so at some point you may see a scrolling event that jumps from something that's close to 232 to 0. You can at this point assume that this was not a valid scroll event. This is a rather theoretical case, overflowing this range requires a lot of scrolling, even on touchpads with a higher scroll increment.

The first scroll event has no usable scroll information

This is a more significant issue. The smooth scrolling information is part of the valuator data, but axis data is only sent in some events, and then only to some clients. Once a pointer leaves the window, the original client loses track of what happens to the valuators. Once the pointer re-enters the client's window it will receive the new axis value once more scrolling happens. That value is an absolute value but clients need the delta between it and the previous scroll event. That information is only available once the second event is sent to the client.

For example, the axis data may be 3000 when the pointer leaves the window. Then the pointer re-enters the window and the first scroll event sends has a value of 5500. Since we don't know how much the axis increased elsewhere, we cannot act on this scroll event, we have to wait for the next one to calculate a delta.

I confirmed with Carlos that this is indeed what GTK does: reset the scroll valuators on enter and then track the valuators for future events.

Tuesday, January 3, 2012

Multitouch in X - Touch grab handling

This post is part of a series on multi-touch support in the X.Org X server.
  1. Multitouch in X - Getting events
  2. Multitouch in X - Pointer emulation
  3. Multitouch in X - Touch grab handling
  4. Multitouch in X - Multitouch-touchpads
In this post, I'll outline how grabs on touch events work. This post assumes basic knowledge of the XI2 Xlib interfaces.

Passive grabs

The libXi interface has one new passive grab call: XIGrabTouchBegin, which works pretty much like the existing passive grab APIs. As with event selection, you must set all three event masks XI_TouchBegin, XI_TouchUpdate and XI_TouchEnd or a BadValue error occurs. Once a passive grab activates in response to a touch, the client must choose to either accept or reject a touch. Details on that below.

Grabs activate on a TouchBegin event and due to the nature of multitouch, multiple touch grabs may be active at any time - some of them for different clients.

Active grabs

Active grabs do not have a new call, they are handled through the event masks of the existing XIGrabDevice(3) call. If a client has an active touch grab on the device, it is automatically the owner of the touch sequence (ownership is described below). If a client has an active pointer or keyboard grab on the device, it is the owner of the touch sequence for pointer emulated touch events only. Other touch events are unaffected by the grab and are processed normally.

Acceptance and rejection

Pointer grabs provide exclusive access to the device, but to some degree a client can opt to replay the event it received on the next client. We expect that touch sequences will often trigger gesture recognition, and a client may realise after a few events that it doesn't actually want that touch sequence. So we expanded the replay semantics. clients with a touch grab must choose to either accept or reject a touch.

Accepting a touch signals to the server that the touch sequence is meant for this client and no-one else. The server then exclusively delivers to that client until the terminating TouchEnd.

Rejecting a touch sequence signals that the touch sequence is not meant for this client. Once a client rejects a touch sequence, the server sends the TouchEnd event to that client (if the touch is still active) and replays the full touch sequence [1] on the next grab or window. We use the term owner of a touch sequence to talk about the current recipient.

The order of events for two clients Cg and Cw, with Cg having a grab and Cw having a regular touch event selection on a window, is thus:
TouchBegin to Cg    → 
TouchUpdate to Cg   → 
TouchUpdate to Cg   → 
                    ← Cg rejects touch
                    ← Cw becomes new owner
TouchEnd+ to Cg     →
TouchBegin* to Cw   → 
TouchUpdate* to Cw  → 
TouchUpdate* to Cw  → 
#### physical touch ends #### 
TouchEnd to Cw      →
Events with + mark an event created by the server, * mark events replayed by the server

For nested grabs, this sequence simply repeats for each client until either a grabbing client accepts the touch or the client with the event selection becomes the owner.

In the above case, the touch ended after Cg rejected the touch. If the touch ends before the current owner accepted or rejected it, the owner gets the TouchEnd event and the touch is left handing until the owner accepts or rejects it. If accepted, that's it. If rejected, the new owner gets the full sequence in one go, including the TouchEnd event. The sequence is thus:
TouchBegin to Cg    → 
TouchUpdate to Cg   → 
TouchUpdate to Cg   → 
#### physical touch ends #### 
TouchEnd to Cg      →
                    ← Cg rejects touch
                    ← Cw becomes new owner
TouchBegin* to Cw   → 
TouchUpdate* to Cw  → 
TouchUpdate* to Cw  → 
TouchEnd* to Cw     →

Touch ownership handling

One additional event type that XI 2.2 introduces is the XI_TouchOwnership event. Clients selecting for this event signal that they need to receive touch events before they're the owner of the touch sequence. This event type can be selected on both grabs and event selections.

First up: there are specific use-cases where you need this. If you don't fall into them, you're better off just skipping on ownership events, they make everything more complicated. And whether you need ownership events depends not only on you, but also the stack you're running under. On normal touch event selection, touch events are only delivered to the current owner of the touch. With multiple grabs, the delivery is sequential and delivery of touch events may be delayed.

Clients selecting for touch ownership events get the events as they occur, even if they are not the current owner. The XI_TouchOwnership event is delivered if and when they become the current owner. The last part is important: if you select for ownership events, you may receive touch events but you may not become the owner of that sequence. So while you can start reacting to that sequence, anything your app does must be undo-able in case the e.g. window manager claims the touch sequence.

If we look at the same sequence as above with two clients selecting for ownership, the sequence looks like this:
TouchBegin to Cg     → 
TouchBegin to Cw     → 
TouchOwnership to Cg →
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
                     ← Cg rejects touch
                     ← Cw becomes new owner
TouchEnd+ to Cg      →
TouchOwnership to Cw →
#### physical touch ends #### 
TouchEnd to Cw      →
Note: TouchOwnership events do not correspond to any physical event, they are always generated by the server

If a touch ends before the owner accepts, the current owner gets the TouchEnd, all others get a TouchUpdate event instead. That TouchUpdate has a flag XITouchPendingEnd set, signalling that no more actual events will arrive from this touch but the touch is still waiting for owner acceptance.
TouchBegin to Cg     → 
TouchBegin to Cw     → 
TouchOwnership to Cg →
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
#### physical touch ends #### 
TouchEnd to Cg       →
TouchUpdate to Cw    →  (XITouchPendingEnd flag set)
                     ← Cg rejects touch
                     ← Cw becomes new owner
TouchOwnership to Cw →
TouchEnd to Cw       →
In both cases, we dealt with a rejecting owner. For an accepting owner, the sequences look like this:
TouchBegin to Cg     → 
TouchBegin to Cw     → 
TouchOwnership to Cg →
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
                     ← Cg accepts touch
TouchEnd+ to Cw      →
TouchUpdate to Cg    → 
#### physical touch ends #### 
TouchEnd to Cg      →
or
TouchBegin to Cg     → 
TouchBegin to Cw     → 
TouchOwnership to Cg →
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
TouchUpdate to Cg    → 
TouchUpdate to Cw    → 
#### physical touch ends #### 
TouchEnd to Cg       →
TouchUpdate to Cw    →  (XITouchPendingEnd flag set)
                     ← Cg accepts touch
TouchEnd* to Cw      →
In the case of multiple grabs, the same strategy applies in order of grab activation. Ownership events may be selected by some clients but not others. In that case, each client is treated as requested, so the event sequence the server deals with may actually look like this:
TouchBegin to C1     → 
TouchBegin to C3     → 
TouchOwnership to C1 →
TouchUpdate to C1    → 
TouchUpdate to C3    → 
TouchUpdate to C1    → 
TouchUpdate to C3    → 
                     ← C1 rejects touch
                     ← C2 becomes new owner
TouchEnd+ to C1      →
TouchBegin* to C2    → 
TouchUpdate* to C2   → 
TouchUpdate* to C2   → 
                     ← C2 rejects touch
                     ← C3 becomes new owner
TouchEnd+ to C2      →
TouchOwnership to C3 →
#### physical touch ends #### 
TouchEnd to C3       →


[1] obviously we need to store these events so "full sequence" really means all events until the buffer was full

Thursday, December 22, 2011

Multitouch in X - Pointer emulation

This post is part of a series on multi-touch support in the X.Org X server.
  1. Multitouch in X - Getting events
  2. Multitouch in X - Pointer emulation
  3. Multitouch in X - Touch grab handling
  4. Multitouch in X - Multitouch-touchpads
In this post, I'll outline how pointer emulation on touch events works. This post assumes basic knowledge of the XI2 Xlib interfaces.

Why pointer emulation?

One of the base requirements of adding multitouch support to the X server was that traditional, non-multitouch applications can still be used. Multitouch should be a transparent addition, available where needed, not required where not supported.

So we do pointer emulation for multitouch events, and it's actually specified in the protocol how we do it. Mainly so it's reliable and predictable for clients.

What is pointer emulation in X

Pointer emulation simply means that for specific touch sequences, we generate pointer events. The conditions for emulation are that the the touch sequence is eligible for pointer emulation (details below) and that no other client has a touch selection on that window/grab.

The second condition is important: if your client selects for both touch and pointer events on a window, you will never see the emulated pointer events. If you are an XI 2.2 client and you select for pointer but not touch events, you will see pointer events. These events are marked with the XIPointerEmulated so that you know they come from an emulated source.

Emulation on direct-touch devices

For direct-touch devices, we emulate pointer events for a touch sequence provided the touch is the first touch on the device, i.e. no other touch sequences were active for this device when the touch started. The touch sequence is emulated until it ends, even if other touches start and end while that sequence is active.

Emulation on dependent-touch devices

Dependent touch devices do not emulate pointer events. Rather, we send the normal mouse movements from the device as regular pointer events.

Button events and button state

Pointer emulation triggers motion events and, more importantly, button events. The button number for touches is hardcoded to 1 (any more specific handling such as long-click for right buttons should be handled by touch-aware clients instead), so the detail field of an emulated button event is 1 (unless the button is logically mapped).

The button state field on emulated pointer events adjusts for pointer emulation as it would for regular button events. The button state is thus (usually) 0x0 for the emulated ButtonPress and 0x100 for the MotionNotify and ButtonRelease events.

Likewise, any request that returns the button state will have the appropriate state set, even if no emulated event actually got sent.

Grab handling works as for regular pointer events, though the interactions between touch grabs and emulated pointer grabs are somewhat complex. I'll get to that in a later post.

The confusing bit

There is one behaviour about the pointer emulation that may be confusing, even though the specs may seem logical and the behaviour is within the specs.

If you put one finger down, it will emulate pointer events. If you then put another finger down, the first finger will continue to emulate pointer events. If you now lift the first finger (keeping the second down) and put the first finger down again, that finger will not generate events. This is noticable mainly in bi-manual or multi-user interaction.

The reason this doesn't work is simple: to the X server, putting the first finger down just looks like another touchpoint appearing when there is already one present. The server does not know that this is the same finger again, it doesn't know that your intention was to emulate again with that finger. Most of the semantics for such interaction is in your head alone and hard to guess. Guessing it wrong can be quite bad, since that new touchpoint may have been part of a two-finger gesture with the second finger and whoops - instead of scrolling you just closed a window, pasted your password somewhere or killed a kitten. So we err on the side of caution, because, well, think of the kittens.

Multitouch in X - Getting events

This post is part of a series on multi-touch support in the X.Org X server.
  1. Multitouch in X - Getting events
  2. Multitouch in X - Pointer emulation
  3. Multitouch in X - Touch grab handling
  4. Multitouch in X - Multitouch-touchpads
I recommend re-reading Thoughts on Linux multitouch from last year for some higher-level comments.
In this post, I'll outline how to identify touch devices and register for touch events.

This post assumes basic knowledge of the XI2 Xlib interfaces. Code examples should not be scrutinised for language-correctness.

New event types

XI 2.2 defines four new event types: XI_TouchBegin, XI_TouchUpdate, XI_TouchEnd are the standard events that most applications will be using. The fourth event, XI_TouchOwnership is mainly for handling specific situations where reaction speed is at a premium and gesture processing when grabs are active. I won't be covering those in this post.

Identifying touch devices

To use multitouch functionality from a client application, the client must announce support for the X Input Extension version 2.2 through the XIQueryVersion(3) request.
int major = 2, minor = 2;
XIQueryVersion(dpy, &major, &minor);
if (major * 1000 + minor < 2002)
    printf("Server does not support XI 2.2\n");
Once announced, an XIQueryDevice(3) call may return a new class type, the XITouchClass. If this class is present on a device, the device supports multitouch.The class struct itself is defined like this:
typedef struct
{
    int         type;
    int         sourceid;
    int         mode;
    int         num_touches;
} XITouchClassInfo;
The num_touches field specifies the number of simultaneous touches supported by the device. If the number is 0, we simply don't know (likely) or the device supports an unlimited number of touches (less likely). Regardless of the value expect that some devices lie, so it's best to treat this value as a guide only.

The mode field specifies the type of touch devices. We currently define two types and the server behaviour differs depending on the type:
  • XIDirectTouch for direct-input touch devices (e.g. your average touchscreen or tablet).  For this type of device, the touch events will be delivered to the windows at the of the touch point. Again, similar to what you would expect from a tablet interface - you press top left and the application top-left responds.
  • XIDependentTouch for a indirect input devices with multi-touch functionality. Touchpads are the prime example here. Touch events on such devices will be sent to the window underneath the cursor and clients are expected to interpret the touchpoints as (semantically) relative to the cursor position. For example, if your cursor is inside a Firefox window and you touch with two fingers on the top-left corner of the touchpad, Firefox will get those events. It can then decide on how to interpret those touchpoints.
A device that has a TouchClass may send touch events, but these events use the same axes as pointer events. Having said that, a touch device may still send pointer events as well - if the physical device generates both.
Your code to identify touch devices could roughly look like this:
XIDeviceInfo *info;
int nevices;

info = XIQueryDevice(display, XIAllDevices, &ndevices);

for (i = 0; i < ndevices; i++)
{
    XIDeviceInfo *dev = &info[i];
    printf("Device name %d\n", dev->name);
    for (j = 0; j < dev->num_classes; j++)
    {
        XIAnyClassInfo *class = dev->classes[j];
        XITouchClassInfo *t = (XITouchClassInfo*)class;

        if (class->type != XITouchClass)
            continue;

        printf("%s touch device, supporting %d touches.\n",
               (t->mode == XIDirectTouch) ?  "direct" : "dependent",
               t->num_touches);
    }
}

Selecting for touch events

Selecting for touch events on a window is mostly identical to pointer events. A client creates an event mask and submits it with XISelectEvents(3). One exception applies: a client must always select for all three touch events [1], XI_TouchBegin, XI_TouchUpdate, XI_TouchEnd. Selecting for one or two only will result in a BadValue error.

As for button events, only one client may select for touch events on any given window and the event delivery attempts traverse from the bottom-most window in the window tree up to the root window. Where a matching event selection is found, the event is delivered and the traversal stops.

Handling touch events

The three event types [1] are XIDeviceEvents like pointer and keyboard events. So from a client's point of view, in essence all we added was new event types.

The detail field of touch events specifies the touch ID, a unique ID for this particular touch for the lifetime of the touch sequence. Each touch sequence consists of a TouchBegin event, zero or more TouchUpdate events and one TouchEnd event. Since multiple touch sequences may be ongoing at any time, keeping track of the ID is important. The server guarantees that the touch ID is unique per device and that it will not be re-used [2]. Note that while touch IDs increase, they increase by an implementation-defined amount. Don't rely on the next touch ID to be the current ID + 1.

The button state in a touch event is the state of the physical buttons only. A TouchUpdate or TouchEnd event will thus usually have a zero button state. [3]

That's pretty much it, otherwise the handling of touch events is identical to pointer or keyboard events. Touch event handling should be straightforward and the significant deviations from the current protocol are in the grab handling, something I'll handle in a future post.

[1] I know, it's four. Good that you're paying attention.
[2] Technically ID collision may occur. For that to happen, you'd need to hold at least one touch down while triggering enough touches to exhaust a 32 bit ID range. And hope that after the wraparound you will get the same ID. There are better ways to spend your weekend.
[3] pointer emulation changes this, but I'll get to that some other time.

Thursday, December 15, 2011

Multitouch patches posted

After pulling way too many 12+ hour days, I've finally polished the patchset for native multitouch support in the X.Org server into a reasonable state. The full set of patches is now on the list. And I'm still expecting this to get merged for 1.12 (and thus in time for Fedora 17).

The code is available from the multitouch branches of the following repositories:
  git://people.freedesktop.org/~whot/xserver
  git://people.freedesktop.org/~whot/inputproto
  git://people.freedesktop.org/~whot/xf86-input-evdev
  git://people.freedesktop.org/~whot/libXi
Here's a screencast running Fedora 16 with the modified X server and a little multitouch event debugging application.


Below is a short summary of what multitouch in X actually means, but one thing is important: being the windowing system, X provides multitouch support. That does not mean that every X application now supports multitouch, it merely means that they can now use multitouch if they want to. That also includes gestures, they need application support.

A car analogy: X provides a new road, the applications still have to opt to drive on it.

Multitouch events

XI 2.2 adds three main event types: XI_TouchBegin, XI_TouchUpdate and XI_TouchEnd. These three make up a touch sequence. X clients must subscribe to all three events at once and will then receive the events as they come in from the device (more or less, grabs can interfere here). Each touch event has a unique touch ID so clients can track the touches over time.

We support two device types: XIDirectDevice includes tablets and touchscreens where the events are delivered to the position the touch occurs at. XIDependentDevice includes multitouch-capable touchpads. Such devices still control a normal pointer by default, but for multi-finger gestures are possible. For such devices, the touchpoints are delivered to the window underneath the pointer.

That is pretty much the gist of it. I'll post more information over time as the release gets closer, so stay tuned.

Pointer emulation

Multitouch can be a compelling interaction method but as said above, X only provides support for multitouch. It will take a while for applications to pick it up (Carlos Garnacho is working on GTK3) and some never will. Since we still need to interact with those applications, we provide backwards-compatible pointer emulation. Again, the details are in the protocol but the gist of it is that for the first touchpoint we emulate pointer events.

That's the really nasty bit, because you now have to sync up the grab event semantics of the core, XI 1.x and XI2 protocols and wrap it all around the new grab semantics. So that if you have a multitouch app running under a window manager without multitouch support everything still works as expected.
That framework is now in place too though I expect it to still have bugs, especially in the hairier corner cases.

But other than that, it should work just as intended. I can interact with my GNOME3 desktop quite well and I get multitouch events to my test applications.

[edit Dec 20: typo fix]

Monday, September 5, 2011

What's new in XI 2.1 - smooth scrolling

This post is the last part of a series about new changes in XI 2.1. See also:


The core protocol allows for up to 255 buttons but only two axes: x and y. When scroll wheels became ubiquitous, the solution was to send button events for the four scroll wheel directions. Even today, a logical button click 4 results in scroll down, 5, 6, and 7 result in scroll up, left, and right, respectively.

However, scrolling is more complex than that. Velocity information and subpixel scrolling are handy to have on scroll axes. With XI 2.1, servers may now mark some axes as axes providing scrolling information. These axes may represent real axes on the device (e.g. a mouse wheel) or be virtual axes (e.g. the edge scrolling on synaptics devices). For each scrolling axis, XIQueryDevice(3) provides an XIScrollClassInfo.

XIDeviceInfo *info;

int ndevices;

info = XIQueryDevice(display, XIAllDevices, &ndevices);
for (i = 0; i < ndevices; i++) {
XIScrollClassInfo *scroll;
for (j = 0; j < info[i].num_classes; j++) {
scroll = info[i].classes[j];
if (scroll->type != XIScrollClass)
continue; /* Not interested in others */


printf("Valuator %d is a valuator for ", scroll->number);
switch(scroll->scroll_type) {
case XIScrollTypeVertical: printf("vertical"); break;
case XIScrollTypeHorizontal: printf("horizontal"); break;
}
printf(" scrolling\n");
printf("Increment of %d is one scroll event\n", scroll->increment);
if (scroll->flags & XIScrollFlagsNoEmulation)
printf("No emulated button events for this axis\n");
if (scroll->flags & XIScrollFlagsPreferred)
printf("No emulated button events for this axis\n");
}
}

XIFreeDeviceInfo(info);


The above code example runs through each class on each device, printing only the scrolling information bit. Note that for each XIScrollClassInfo, a XIValuatorClassInfo with the same axis number must be present. That valuator info then contains the actual details of the device.

Let's say a XIScrollClassInfo is available for valuator 3. This simply means that any coordinates on valuator 3 should be interpreted as scrolling events. The increment defines what delta the driver considers to be one scroll event. For an increment of +5, each delta of 5 should be regarded as one scroll unit down. For an increment of -3, each delta of 3 should be regarded as one scroll unit up (i.e. inverted).

Note that a driver may change this at run-time. As always, a client should listen to XIDeviceChangedEvents.

Scroll button emulation


If a driver provides smooth-scrolling valuators, the server provides two-way emulation of scroll events to remain backwards-compatible to existing X clients. For any button 4-7 event from the device, the server emulates a scroll event with the right increment on the respective axis. (Note that drivers are discouraged to use button events for scrolling if they support smooth scrolling). For any scroll event on a scroll axis axis, the server emulates the matching button 4-7 events. If the scroll event has an abs(value) less than the increment on this axis, the server only emulates once the cumulative value of several events hits a multiple of the increment (positive or negative).

For clients, this means they get two events for each hardware scrolling event. The one that was emulated by the server has the XIPointerEmulated flag set and can be safely ignored by the client. The same flag is set for XIRawEvents as well.

A device may have multiple scoll axes of the same type. If so, one of them is marked as XIScrollFlagsPreferred and any legacy button events will be emulated on that axis. This bit is informative only, I don't think clients need to care about it. Plus, we're aiming for drivers to do smooth scrolling only, so hopefully we don't need to do button → scroll valuator emulation.

Potential side-effects


  • Pre-2.1 clients may now get valuator information from a scroll axis that they don't identify as scroll axis. For 1.x clients, this is not much of an issue. 1.x clients never knew what an axis represented anyway and needed a UI for users to select what an axis meant. 2.0 clients have axis labels and should act on those. So I think this is mostly a non-issue.

  • Clients that implement smooth-scrolling may act on information less than one increment. Permanent up/down movement of values less than one increment may then cause smooth scrolling events but no legacy events. I don't think this is an issue, users usually trigger the scrolling axis until they see the effect on-screen. The worst-case scenario here is that scrolling in one client requires more finger movement than in another client.

  • Clients cannot yet change the increment. I don't know if this will be an issue in the real world given that the actual interpretation is in the client anyway.


Saturday, September 3, 2011

What's new in XI 2.1 - raw events

This post is part of a series about new changes in XI 2.1. See also:


Raw events were introduced in XI 2.0 to allow clients to track direct device data, before the server gets its hands on it. The prime target for these was games that needed relative mouse events.

However, there was one flaw with raw events: they were delivered either to the root window or to the grabbing client only. Which made them less-than-useful, since grabs are virtually everywhere: active grabs by clients, passive grabs whenever a popup or dropdown menu appears, implicit grabs whenever the button is down.

This behaviour changes in XI 2.1. If a client announces support for XI 2.1, it will get raw events delivered to the root window at all times.

The second change to raw event behaviour is that the events now have the sourceid set (Bug 34420). The libXi interface catered for this but we never sent the value down the wire. This is fixed now in XI 2.1.

Friday, September 2, 2011

What's new in XI 2.1 - XI2 defines

This post is part of a series about new changes in XI 2.1. See also:


A while ago, I spent some time staring at gdb wondering why a passive grab didn't activate. Turns out it wasn't a bug in the server after all, it was a bug in the program. The modifiers were set like this:


XIGrabModifiers modifiers[1];
modifiers[0].modifiers = AnyModifier;


AnyModifier is a define that comes from the core protocol:
#define AnyModifier             (1<<15)  /* used in GrabButton, GrabKey */

but XI2 provides its own define:
#define XIAnyModifier           (1U << 31)


Passive grabs only activate if the modifiers match the current modifier state. An XI2 grab that uses AnyModifier won't activate as expected - AnyModifier looks like a normal modifier bitmask to the server.

I wanted to reply to the reporter with "don't mix core and XI2 defines" but there's one problem: you couldn't currently write an XI2 application without using some core defines (e.g. GrabModeAsync). I've looked through the protocol spec and added those defines that previously forced clients to mix input-related constants. New defines now available are:


  • XIPropModeReplace, XIPropModePrepend, and XIPropModeAppend

  • XIAnyPropertyType

  • XIGrabModeSync and XIGrabModeAsync

  • XIGrabSuccess, XIAlreadyGrabbed, XIGrabInvalidTime, XIGrabNotViewable, and
    XIGrabFrozen

  • XIOwnerEvents, XINoOwnerEvents



These will be available to any application compiling against XI 2.1 headers. All the above are identical to the core defines. Clients are still expected to use CurrentTime and None from the core protocol headers since neither of those two is input-related. For applications building against 2.1, the guideline to using XI2 can only be summarised as: Only use a define Foo if there is no equivalent XIFoo version.

The libXi man pages have also been updated where applicable to point out the new defines.
Many thanks to Timur Kristóf for uncovering this.

XIOwnerEvents and XINoOwnerEvents are new additions to improve readability. They simply map to True/False.

What's new in XI 2.1 - versioning

This post is the start of a mini-series describing the new features added to the X Input Extension version 2.1, or short XI 2.1.

Also, I need to point out that XI 2.1 is not yet released, some minor changes may still be added. Nonetheless, this series should help you understand the new additions and if you see any issues - by all means let me know so we can amend before it is set in stone.

In this post, I'll talk about versioning.

First, for all those looking for big headlines: multitouch in X has been postponed to XI 2.2. XI 2.1 is a small incremental change with two bugfixes and one new feature. These are uncontroversial and more importantly, they are ready now. So XI 2.1 is released as a small change, with bigger changes coming for XI 2.2. We currently still plan to get XI 2.2 into server 1.12 so this is really just a numbers game and ticking off work that is complete.

The small list of changes for XI 2.1:

  • New raw events behaviour

  • Smooth scrolling

  • XI2 defines


I'll describe each of those in a separate post.

Announcing support for XI 2.1


Any client that supports XI2 must already announce so with XIQueryVersion(3). To unlock XI 2.1 behaviour, a client must announce support for XI 2.1 or later.


int major = 2;
int minor = 1;
int rc;

rc = XIQueryVersion(display, &major, &minor);
if (rc == Success)
printf("Server supports XI %d.%d\n", major, minor);
else if (rc == BadRequest)
printf("Server does not support XI2.\n");
else
printf("Internal error\n");


The server is compatible with XI 2.0 and XI 2.1. Thus, a client that does not announce XI 2.1 support will get XI 2.0 behaviour.


Using XI2 version defines


Because I fell into this trap myself, a word of warning about a potential bug. If you are currently using the XI_2_Major and XI_2_Minor defines for versioning, recompiling against new inputproto headers will automatically bump your client up to the latest version, including possible unexpected behaviours. Don't do this, XI2 is versioned through numbers, not defines. Have a look at this xinput commit to get a better idea.

Friday, June 3, 2011

Linking kernel and X devices

XI2 has made it possible to access any device the X server knows about and get events from those devices. Alas, one piece was missing: short of parsing the Xorg.log file it was so far impossible to link the system devices with kernel device. You had to guess that /dev/input/event3 was your "AT Translated Set 2 keyboard" with the X device ID of 6.

Why would one need this? Some properties of a device are not exported in X. These included for example anything in sysfs.

I've now pushed patches to evdev, synaptics and the wacom driver to export the device node as a simple string property. From this device node, a client can navigate to the matching sysfs dir, get other events out of the kernel or do whatever they like to do.

The property name is "Device Node" and it is a standard char* that contains the device path. The path is set by the driver, so even if you use e.g. auto-dev on synaptics, the value will be filled in correctly. On the downside, if the driver doesn't support it, the property will not be available on a device. If you write clients that require this information, ensure that you can cope with a missing property and remember that several X devices may share the same device node (e.g. wacom).

These patches do not require a new server, though we have added a #define XI_PROP_DEVICE_NODE to the server to make it easier for others to use this property without misspelling it.

Another property that was pushed in the same series was the "Device Product ID" property (#define XI_PROP_PRODUCT_ID once the new server is out). This property contains two unsigned 32-bit values: vendor ID and product ID. Having this information available makes it simpler for X clients to apply configuration based on specific devices. This can come in handy if you e.g. need different acceleration for a set of mice or you simply want to remap buttons on a specific device.
Again, the two values are set by the driver and all the above rules apply.

I may end up writing those patches for mouse and keyboard too but feel free to help me out with it.

The obligatory not-quite-a screenshot:

:: whot@barra:~> xinput list-props "SynPS/2 Synaptics TouchPad"
Device 'SynPS/2 Synaptics TouchPad':
[...]
Device Product ID (252): 2, 7
Device Node (253): "/dev/input/event4"



Update June 7 2011: As Peter pointed out in the comments, the device node is always local to the host the server runs on. It is up to the client to figure out if it runs on the same host.

Friday, August 6, 2010

First draft of multitouch protocol spec published

In case you're not subscribed to xorg-devel, I've sent off the first draft for the multitouch X protocol specification (as part of the X Input Extension 2.1) today. If you have interest in multitouch, please read through it and see if you can poke holes into the current approach.

http://lists.freedesktop.org/archives/xorg-devel/2010-August/011759.html

Note that this is very low-level stuff. The data described by the protocol is what's sent on the wire to the client. Most applications are separated from this data by toolkits (GTK, Qt, etc.) and only few people are unlucky enough to have to care about the protocol itself.

Wednesday, May 26, 2010

GTK merged XI2 support

From GNOME Bug 596725 - Add XInput2 support:
- Comment #11 from Matthias Clasen <...> 2010-05-26 00:07:29 +UTC ---
Merged


Nice!
Now I just need to find time to play with it...

Friday, October 2, 2009

XI2 and MPX released!

It finally happened! After nearly 4 years of development, MPX has been released as part of XI2 in the new X Server 1.7.

The whole thing started when I started my PhD in late 2004. The problem I found was that there was no support for collaboration on a single shared display. All the solutions at the time were hacks at the toolkit or application level. I found that the only way we can get truly collaborative interfaces is by adding it into the windowing system itself. So started hacking on X in late 2005. I went from scratching my head and wondering how some of the stuff could compile (I had never heard of K&R function declarations) to rewriting large parts of the input subsystem and even ended up as release manager. Not in a single day though.

Now we're done. MPX is out, and we have generic low-level support for multiple input devices. You know the whole one keyboard-one mouse paradigm we've had since Doug Engelbart invented the mouse? It's over, you don't have to restrict yourself anymore when writing an app.

Of course, this is a low-level change and when you wake up tomorrow, not a lot will have actually changed. We still need the toolkits to support it, we need apps to pick it up, we need the desktop environments to start thinking about what can be made useful. Nonetheless, basic collaboration features are already there and it can only get better from here.

Let's see what will happen.

Thursday, July 30, 2009

XI2 ready for the final mile

As announced on the xorg-devel list, I think XI2 is ready for branching to 1.7.

This means that I consider the protocol stable enough and I will focus only on bugfixing. In a few days time I'll cut a 901 release (release candidate 1) for inputproto and libXi.

Any testing is appreciated, feel free to file bugs for anything that's broken.

Thursday, July 23, 2009

XI2 Recipes, Part 6

This post is part of a mini-series of various recipes on how to deal with the new functionality in XI2. The examples here are merely snippets, full example programs to summarize each part are available here.

In the first five parts, I covered how to get and manipulate the device hierarchy, how to select for events, how to get extended device information, the common event types and active and passive grabs. In this part, I will focus on the client pointer.

The ClientPointer principle


The ClientPointer (CP) principle is only partly interesting for normal applications since no XI2 client should ever need it. The exception is the window manager. About one quarter of XI2 protocol requests and replies are ambiguous in the presence of multiple master devices. The best example is XQueryPointer(3). If there are two or more master pointers, XQueryPointer has a <50% chance of returning the right data.

The ClientPointer is a dedicated master pointer assigned to each application, either implicitly or explicitly. This pointer is then used for any ambiguous requests the application may send. This adds predictability as the data returned is always from the same device. Given the above example, XQueryPointer requests from one client will always return the same pointer's coordinates. Thinking of xeyes this means that the eyes will follow the same cursor. For any requests or replies that require keyboard data, the master keyboard paired with the CP is used.

The CP is implicitly assigned whenever an application sends an ambiguous request. Then the server picks the first master pointer and assigns it to the client. This of course happens only when the client doesn't have an assigned CP yet.

Alternatively, the CP can be explicitly assigned. The XISetClientPointer(3) and XIGetClientPointer(3) calls are to set and query the current ClientPointer for a client.


Status XISetClientPointer(
Display* dpy,
Window win,
int deviceid
);

Bool XIGetClientPointer(
Display* dpy,
Window win,
int* deviceid
);


Both calls take a window and a deviceid. If the window is a valid window, the client owning this window will have the CP set to the given device. The window parameter may also be just a pure client ID. Finally, the window parameter may be None, in which case the requesting client's CP is set to the given device. This is not useful beyond debugging, if the client understands enough XI2 to set the CP it should be able to handle multiple devices properly.

Getting the CP takes the same parameters but it returns the deviceid and it returns True if the CP has been set for the target client, regardless of whether it was set implicitly or explicitly. If no CP is set yet, XIGetClientPointer returns False.

Event delivery, XI2 and and grabs


The CP setting does not affect event delivery in any way. Regardless of which master pointer is the ClientPointer, any device can still interact with the client. This also means that the CP has no effect whatsoever on XI2 or XI1 requests since they are not ambiguous.

Grabs are a different matter. Since the activation of a grab is ambiguous in the core protocol (XGrabPointer - well, which pointer?) a grab will by default activate on the CP. This can be a bit iffy since an application that just grabs the pointer may not grab the one currently within the window boundaries. So the grabbing code has two exceptions. One, if a device is already grabbed by the client, a grab request will act on the already-grabbed device instead of the CP. Two, if a passive grab activates it will activate on the device triggering the grab, not on the CP.

In practice, this means that if a client has a passive grab on a button and any device presses this button, the passive grab activates on this device. If the client then requests and active grab (which toolkits such as GTK do), the active grab is set on the already-grabbed device.
The result: in most cases the grab happens on the "correct" device for the current situation.

How to use the ClientPointer


As said above, the only application that should really need to know about the CP is the window manager who manages core applications as well as XI apps. The most straightforward manner to managing an application is to set the CP whenever a pointer clicks into a client window. This ensures that if the applications requests some ambiguous data, a pointer that is interacting with the application is used.

I have used this method in a custom window manager written for a user study several moons ago and it works well enough. Of course, you are free to contemplate situations where such a simple approach is not sufficient.

Wednesday, July 22, 2009

XI2 recipes, Part 5

This post is part of a mini-series of various recipes on how to deal with the new functionality in XI2. The examples here are merely snippets, full example programs to summarize each part are available here.

In the first four parts, I covered how to get and manipulate the device hierarchy, how to select for events, how to get extended device information and the common event types. In this part, I will focus on grabs.

What are grabs?


Grabs ensure event delivery to a particular client and to this client only. A common application of grabs are drop-down and popup menus. If such a menu is displayed, the pointer is grabbed to ensure that the next click is delivered to the client displaying the popup. The client can then either perform an action or undisplay the popup.

Two different types of grabs exist: active grabs ("grab here and now") and passive grabs ("grab whenever a button/key is pressed"). Both types of grabs may be synchronous or asynchronous. Asynchronous grabs essentially do just the above - deliver all future events to the grabbing client only (in relation to the grab window). Synchronous grabs stop event reporting after an event reported to the grabbing client and it is up to the client what happens next (see below). I recommend reading the XGrabPointer and XAllowEvents man pages for more detail, XI2 grabs work essentially in the same manner.

Active grabs


Devices may be actively grabbed with the XIGrabDevice() call. This call is quite similar to the core protocol's XGrabPointer, so I encourage you to read the matching man page for more detail than provides here.


XIGrabDevice(Display* dpy, int deviceid, Window grab_window, Time time,
Cursor cursor, int grab_mode, int paired_device_mode,
Bool owner_events, XIEventMask *mask)

The device may be a master or slave device that is currently not grabbed, the window must be viewable and time must be CurrentTime or later for the grab to succeed. If the device is a master pointer device, the cursor specified in cursor is displayed for the duration of the grab. Otherwise, the cursor argument is ignored.
Grab mode is either synchronous or asynchronous and applies to the device. If the device is a master device, the paired_device_mode applies to the paired master device. Otherwise, this argument is ignored.
The mask itself is a simple event mask, specifying the events to be delivered to the grabbing client. Depending on owner_events, events are reported only if selected by the mask (if owner_events is false) or are reported if selected by the client on the window (if owner_events is true).

The matching call to release an active grab is XIUngrabDevice(). If a synchronous grab is issued on a device, the client must use XIAllowEvents() to control further event delivery - just as with synchronous core grabs.

Noteworthy about XIGrabDevice is that the deviceid must specify a valid device. The fake deviceids of XIAllDevices or XIAllMasterDevices will result in a BadDevice error.

Passive grabs on buttons and keys


Devices may be passively grabbed on button presses and key presses, similar to the core protocol passive grabs.
Let's have a look at button presses. Passive button grabs ensure that the next time the specified button is pressed, a grab is activated on the device and the client receives the event until the button is released again. This happens automatically provided the matching modifiers are down.
The API to set a button grab is similar to the one from the core protocol with one notable exception. Button grabs can be registered for multiple modifier combinations in one go:


int XIGrabButton( Display *display, int deviceid,
int button, Window grab_window,
Cursor cursor, int grab_mode,
int paired_device_mode, Bool owner_events,
XIEventMask *mask, int num_modifiers,
XIGrabModifiers *modifiers_inout);


The first couple of parameters are identical to the active grab calls. The num_modifiers parameter specifies the length of the modifiers_inout array. The modifiers_inout itself specifies the modifiers states when this grab should be activated. So instead of one request per button+keycode combination, a client can submit all modifier combinations in one go. If any of these combinations fails, the server returns it in the modifiers_inout array.


typedef struct
{
int modifiers;
int status;
} XIGrabModifiers;


The client sets modifiers to the modifiers that should activate the grab, status is ignored by the server. The server returns the number of modifier combinations that could not be set and their status value. The modifiers themselves are always the effective modifiers. No difference between latched, locked, etc. is made.

In pseudo-code, setting a button grab looks something like this.


int nmodifiers = 2;
XIGrabModifiers modifiers[nmodifiers];

modifers[0].modifiers = ShiftMask;
modifiers[1].modifiers = ShiftMask | Mod1Mask;

if ((nmodifiers = XIGrabButton(..., nmodifiers, modifiers)) {
for (i = 0; i <nmodifiers; i++)
printf("Modifiers %x failed with %d\n", modifiers[i].modifiers, modifiers[i].status);



The API for grabbing keycodes is essentially the same as for button grabs.


int XIGrabKeycode( Display *display, int deviceid,
int keycode, Window grab_window,
int grab_mode, int paired_device_mode,
Bool owner_events, XIEventMask *mask,
int num_modifiers, XIGrabModifiers *modifiers_inout);


Both button and keycode grabs can be removed with XIUngrabButton and XIUngrabKeycode.

The fake deviceids of XIAllDevices and XIAllMasterDevices are permitted for passive button and key grabs.

Passive grabs on enter/focus events



Grabbing on enter and focus events is a new feature of XI2. Similar in principle to button presses except that the device is grabbed when the pointer or the keyboard focus is set to the specified window.


int XIGrabEnter( Display *display, int deviceid,
Window grab_window, Cursor cursor,
int grab_mode, int paired_device_mode,
Bool owner_events, XIEventMask *mask,
int num_modifiers,
XIGrabModifiers *modifiers_inout);

int XIGrabFocusIn ( ...)


The parameters are the same for both (XIGrabFocusIn doesn't take a cursor parameter) and they are the same as for button/keysym grabs anyway. When an enter and focus grabs activates, the grabbing client receives an additional XI_Enter of XI_FocusIn event with detail XINotifyPassiveGrab (provided it is set on the window or in the grab mask). To avoid inconsistencies with the core protocol, this event is sent after the enter event is sent to underlying window.

As an example, consider the following pseudo code:


Window win = create_window();

int nmodifiers = 1;
XIGrabModifiers modifiers[nmodifiers];
modifiers[0].modifiers = 0;

if ((nmodifiers = XIGrabEnter(display, 2, win, ..., num_modifiers, modifiers))
handle_errors();

while(1)
{
XEvent ev;
XNextEvent(dpy, &ev);
if (XGetEventData(dpy, &ev.xcookie) &&
ev.xcookie.extension == xi_opcode &&
ev.xcookie.evtype == XI_Enter)
{
XIEnterEvent *enter = ev.xcookie.data;

if (enter->detail == XINotifyPassiveGrab)
printf("we have grabbed the device\n");
}
XFreeEventData(dpy, &ev);
}


Provided that nmodifiers is 0 after the call to XIGrabEnter, each time the pointer moves into the window win the device will be grabbed (provided no modifiers are logically down). Once the pointer leaves the window again, the device is automatically ungrabbed.

Passive enter/focus grabs can be removed with the XIUngrabEnter and XIUngrabFocusIn calls.

The fake deviceids of XIAllDevices and XIAllMasterDevices are permitted for passive button and key grabs.

Wednesday, July 15, 2009

XI2 recipes, Part 4

This post is part of a mini-series of various recipes on how to deal with the new functionality in XI2. The examples here are merely snippets, full example programs to summarize each part are available here.

In the first three parts, I covered how to get and manipulate the device hierarchy, how to select for events and how to get extended device information. In this part, I will cover the common input events and the data they include.

All XI2 events are cookie events and must be retrieved with XGetEventData.


XIDeviceEvent


The XIDeviceEvent is the default event for button and key press and releases and motion events.
The event types that produce an XIDeviceEvent are XI_Motion, XI_ButtonPress, XI_ButtonRelease, XI_KeyPress and XI_KeyRelease. Note that proximity events do not exist in XI2. A device that supports proximity should instead provide another axis and send valuator events on this axis.

The event itself is (reasonably) close to the core events, so most fields may be familiar in one way or another. It is a GenericEvent, so the type is always GenericEvent (35), extension is the X Input extension's opcode and evtype is the actual type of the event: XI_KeyPress, XI_KeyRelease, XI_ButtonPress, XI_ButtonRelease, XI_Motion. XI2 does not have proximity events like XI1 has, on the basis that if a device supports proximity, it should just provide an axis that reports proximity data (even if that axis has a range of 0-1).

Each event provides the device the event came from and the actual source the event originated from. So for applications that listen to master device events, deviceid is the id of the master device, and sourceid is the id of the physical device that just got moved/clicked/typed.


static void print_deviceevent(XIDeviceEvent* event)
{
printf(" device: %d (%d)\n", event->deviceid, event->sourceid);
printf(" detail: %d\n", event->detail);
if (event->flags & XIKeyRepeat)
printf(" event is a key repeat.\n");

...
}


For button events, detail is the button number (after mapping applies of course). For key events, detail is the keycode. XI2 supports 32-bit keycodes, btw. For motion events, detail is 0. The flags field is a combination of various flags that apply for this event. Right now, the only defined flag is XIKeyRepeat for XI_KeyPress events. If this flag is set, the event is the result of an in-server key repeat instead of a physical key press (waiting for daniels to send me the patch for the server).

Each event includes root-absolute and window-relative coordinates with subpixel precision. For example, if you have your mouse slowed down by constant deceleration, you'll see the pointer's X coordinate move from 100.0 to 100.25, 100.5, 100.75, 101, etc. The same happens with devices that have their own coordinate range (except that that bit is missing in the server right now.).

XIDeviceEvents contain the button, modifier and valuator state in four different structs:

XIButtonState buttons;
XIValuatorState valuators;
XIModifierState mods;
XIGroupState group;


The buttons include the button state for each button on this device. Since we don't have any restrictions on the number of buttons in the protocol, the mask looks like this:


typedef struct {
int mask_len;
unsigned char *mask;
} XIButtonState;


The mask_len specifies the length of the actual mask in bytes. The bit for button N is defined as (1 << N), if it is set then the button is currently logically down. Quite similar is the valuator state, except that the mask specifies which valuators are provided in the values array.


typedef struct {
int mask_len;
unsigned char *mask;
double *values;
} XIValuatorState;


Again, mask_len is in bytes and for each bit set in mask, one double represents the current value of this valuator in this event. These coordinates are always in the device-specific coordinate system (screen coordinates for relative devices). To give you an example, if mask_len is 1 and bits 0 and 5 are set in mask, then values is an array size 2, with the values for axis 0 and axis 5.

Finally, the event contains the state of the modifier keys and the current XKB group info.

typedef struct
{
int base;
int latched;
int locked;
int effective;
} XIModifierState;

typedef XIModifierState XIGroupState;


The base modifiers are the ones currently pressed, latched the ones pressed until a key is pressed that's configured to unlatch it (e.g. some shift-capslock interactions have this behaviour) and finally locked modifiers are the ones permanently active until unlocked (default capslock behaviour in the US layout). The effective modifiers are a bitwise OR of the three above - which is essentially equivalent to the modifiers state supplied in the core protocol events.
The group state is a bit more complicated, since the effective group it is the arithmetic sum of all 3 after the group overflow handling is taken into account. The meaning of base, latched and locked is essentially the same otherwise.

Enter/leave and focus events


One of the biggest deficiencies of XI1 is the lack of enter/leave events for extended devices. XI2 provides both and they are essentially the same as the device events above with three extra fields: mode, focus and same_screen. Both enter/leave events and focus events are the same as core events squashed in an XI2 format. I recommend reading the core protocol spec for these events, it's much more verbose (and eloquent) than this blog.

Unsurprisingly, enter/leave events are generated separately for each device. While the core protocol has a quite funky model to ensure that applications aren't confused when multiple pointers exit or leave a window, the XI2 events are sent for each pointer, regardless of how many devices are currently in the window.

Focus events are simply typedefs of the enter events, so there's nothing exciting there.

Property events


Property events have stayed the same, except that they use the XGenericEvent (and cookie) format now. Property events contain the property that changed, the deviceid and a field detailing what actually changed on this property (one of XIPropertyDeleted, XIPropertyCreated, XIPropertyModified).

Raw events


Raw events are something new. Normal input events are heavily processed by the server (clipped, accelerated, mapped to absolute, etc.). Raw events are essentially a container to forward the data the server works with (e.g. the data passed up by the driver) and thus do not contain state other than the new information. The three interesting fields are detail, valuators and raw_values.


typedef struct {
...
int detail;
XIValuatorState valuators;
double *raw_values;
} XIRawEvent;


That the detail for button events is the unmapped button number or the key code. Possible evtypes for raw events are XI_RawMotion, XI_RawKeyPress, XI_RawKeyRelease, XI_RawButtonPress and XI_RawButtonRelease.

Valuator information works in the same manner as in XIDeviceEvents and contains the transformed (i.e. accelerated) valuators as used in the server. The raw_values array provides the untransformed values as they were passed up from the driver. This is useful for applications that need to provide their own acceleration code (e.g. games).
For example, the following bit shows the acceleration applied on each axis:


void print_rawmotion(XIRawEvent *event)
{
int i;
double *raw_valuator = event->raw_values,
*valuator = event->valuators.values;

for (i = 0; i < event->valuators.mask_len * 8; i++) {
if (XIMaskIsSet(event->valuators.mask, i)) {
printf("Acceleration on valuator %d: %f\n",
i, *valuator - *raw_valuator);
valuator++;
raw_valuator++;
}
}
}



Since raw events do not have target windows they are delivered exclusively to all root windows. Thus, a client that registers for raw events on a standard client window will receive a BadValue from XISelectEvents(). Like normal events however, if a client has a grab on the device, then the event is delivered only to the grabbing client.