X.Org integration test suite - part 2, the registry

Back in August, I wrote about the X.Org Integration Test suite, in short XIT. Since then I've been quite busy adding tests to it, expanding it and fixing the odd bug. Jasper St. Pierre has been writing tests against it for the new pointer barrier features, making that feature probably the first to get full, repeatable testing before merging.

Aside from general cleanup so that tests are even easier to write, one of the features I've pushed today is a bug registry. One of the issues with the test suite (especially one that is not integrated with the repository directly) is that tests don't just fail or succeed, they can also be known to fail, or known to succeed.

For example, a test may succeed on git master, but fail on the 1.13 branch - either because the fix has not yet been backported, or because the fix will not be backported anyway.

Keeping track of those failures is quite a task, especially when you have multiple server versions to worry about.

This is what the bug-registry is supposed to address. It's in its early stages, but has already been quite helpful. At it's very base is the commandline client xit-bug-registry, and the matching data file (xml) that keeps track of the various test cases.

This is a simple introduction on how to use it.

To get started, run the tests, then create a new registry from the test output

$> sudo ./tests/server/server --gtest_output="xml:server-results.xml"
$> xit-bug-registry create server-results.xml > registry.xml

Information about the tests can now be queried:

$> xit-bug-registry -f registry.xml list

TestSuite                     TestCase                               Success
---------                     --------                               -------
BarrierConflictingDirections  InvalidConflictingDirectionsBarrier/0     True
BarrierConflictingDirections  InvalidConflictingDirectionsBarrier/1     True
EventQueueTest                mieqOverflow                              True
MiscServerTest                DoubleSegfault                            True
PointerGrabTest               GrabDisabledDevices                       True
PointerGrabTest               ImplicitGrabRawEvents                     False
ScreenSaverTest               ScreenSaverActivateDeactivate             False
...

The above shows a bunch of tests that are expected to succeed, and two that are expected to fail. Ideally, these tests will be annotated, so that looking up info will look like this:

$> xit-bug-registry -f registry.xml info XTest DisabledDevicesProperty
XTest DisabledDevicesProperty:  Expected result: Success
Known bugs: 
0: https://bugs.freedesktop.org/show_bug.cgi?id=56380
Known fixes: 
0: aad65415bff12c6860c19beac42e4165e598a40f

To get there, some manual interaction is required:

$> xit-bug-registry -f registry.xml edit \
 XTest DisabledDevicesProperty \
 add-bug http://bugs.freedesktop.org/56380
$> xit-bug-registry -f registry.xml edit \
 XTest DisabledDevicesProperty \
add-commit aad65415bff12c6860c19beac42e4165e598a40f

There are other bits one can add, but bugs and commits are likely the interesting bits. Right now, that's all it does, but in the future I hope to expand the script to query the bug database for the bug status, and query repositories to check if the fix is on the branch yet.

That's all nice, but the real important bit is to verify that after fixing a bug, one hasn't broken anything.

$> sudo ./tests/server/server --gtest_output="xml:server-results.xml"
$> xit-bug-registry -f registry.xml verify server-results.xml
Code TestSuite                    TestCase                                Result  Expected
---- ---------                    --------                                ------  --------
++   BarrierConflictingDirections InvalidConflictingDirectionsBarrier/0   true      true
++   BarrierConflictingDirections InvalidConflictingDirectionsBarrier/1   true      true
++   BarrierConflictingDirections InvalidConflictingDirectionsBarrier/2   true      true
++   BarrierConflictingDirections InvalidConflictingDirectionsBarrier/3   true      true
++   BarrierConflictingDirections InvalidConflictingDirectionsBarrier/4   true      true
++   BarrierConflictingDirections InvalidConflictingDirectionsBarrier/5   true      true
--   PointerGrabTest              ImplicitGrabRawEvents                  false     false
XX   ScreenSaverTest              ScreenSaverActivateDeactivate           true     false
++   TouchDeviceChangeTest        DeviceChangedEventPointerToTouchSwitch  true      true
++   TouchDeviceChangeTest        DeviceChangedEventTouchToPointerSwitch  true      true
XX   TouchDeviceChangeTest        NoCursorJumpsOnTouchToPointerSwitch     true     false
++   TouchDeviceTest              DisableDeviceEndTouches                 true      true
--   TouchEventHistoryTest        EventHistoryReplay/0                   false     false
--   TouchEventHistoryTest        EventHistoryReplay/1                   false     false
--   TouchEventHistoryTest        EventHistoryReplay/10                  false     false
...

This simply shows us which tests had which result. Cases with code ++ succeeded when they were expected to, -- failed when they were expected to. XX is what you need to look for, it indicates that the test outcome differs to the status in the registry. In this case, two tests now succeed when before they didn't - usually a good outcome.

Once a fix is merged, you can update the status of the bug:

$> xit-bug-registry -f registry.xml edit \
 XTest DisabledDevicesProperty \
 set-status success

If this test fails in the future, you'll now be able to spot it.

The information is all in the registry file and though I admit the interface is still clumsy, it's quite simple to keep a set of registries (e.g. upstream, Fedora, RHEL, etc.) and thus a known set of test outcomes for each.

Finally, as test cases are being added, it's important to update the registry. This is as simple as:

$> xit-bug-registry create results-with-new-testcases.xml > new-registry.xml
$> xit-bug-registry merge registry.xml new-registry.xml

This command will merge any new test cases into our original registry, but won't update the status on any existing ones.