Back in August, I wrote about the X.Org Integration Test suite, in short XIT. Since then I've been quite busy adding tests to it, expanding it and fixing the odd bug. Jasper St. Pierre has been writing tests against it for the new pointer barrier features, making that feature probably the first to get full, repeatable testing before merging.
Aside from general cleanup so that tests are even easier to write, one of the features I've pushed today is a bug registry. One of the issues with the test suite (especially one that is not integrated with the repository directly) is that tests don't just fail or succeed, they can also be known to fail, or known to succeed.
For example, a test may succeed on git master, but fail on the 1.13 branch - either because the fix has not yet been backported, or because the fix will not be backported anyway.
Keeping track of those failures is quite a task, especially when you have multiple server versions to worry about.
This is what the bug-registry is supposed to address. It's in its early stages, but has already been quite helpful. At it's very base is the commandline client xit-bug-registry, and the matching data file (xml) that keeps track of the various test cases.
This is a simple introduction on how to use it.
To get started, run the tests, then create a new registry from the test output
$> sudo ./tests/server/server --gtest_output="xml:server-results.xml" $> xit-bug-registry create server-results.xml > registry.xmlInformation about the tests can now be queried:
$> xit-bug-registry -f registry.xml list TestSuite TestCase Success --------- -------- ------- BarrierConflictingDirections InvalidConflictingDirectionsBarrier/0 True BarrierConflictingDirections InvalidConflictingDirectionsBarrier/1 True EventQueueTest mieqOverflow True MiscServerTest DoubleSegfault True PointerGrabTest GrabDisabledDevices True PointerGrabTest ImplicitGrabRawEvents False ScreenSaverTest ScreenSaverActivateDeactivate False ...The above shows a bunch of tests that are expected to succeed, and two that are expected to fail. Ideally, these tests will be annotated, so that looking up info will look like this:
$> xit-bug-registry -f registry.xml info XTest DisabledDevicesProperty XTest DisabledDevicesProperty: Expected result: Success Known bugs: 0: https://bugs.freedesktop.org/show_bug.cgi?id=56380 Known fixes: 0: aad65415bff12c6860c19beac42e4165e598a40fTo get there, some manual interaction is required:
$> xit-bug-registry -f registry.xml edit \ XTest DisabledDevicesProperty \ add-bug http://bugs.freedesktop.org/56380 $> xit-bug-registry -f registry.xml edit \ XTest DisabledDevicesProperty \ add-commit aad65415bff12c6860c19beac42e4165e598a40f
There are other bits one can add, but bugs and commits are likely the interesting bits. Right now, that's all it does, but in the future I hope to expand the script to query the bug database for the bug status, and query repositories to check if the fix is on the branch yet.
That's all nice, but the real important bit is to verify that after fixing a bug, one hasn't broken anything.
$> sudo ./tests/server/server --gtest_output="xml:server-results.xml" $> xit-bug-registry -f registry.xml verify server-results.xml Code TestSuite TestCase Result Expected ---- --------- -------- ------ -------- ++ BarrierConflictingDirections InvalidConflictingDirectionsBarrier/0 true true ++ BarrierConflictingDirections InvalidConflictingDirectionsBarrier/1 true true ++ BarrierConflictingDirections InvalidConflictingDirectionsBarrier/2 true true ++ BarrierConflictingDirections InvalidConflictingDirectionsBarrier/3 true true ++ BarrierConflictingDirections InvalidConflictingDirectionsBarrier/4 true true ++ BarrierConflictingDirections InvalidConflictingDirectionsBarrier/5 true true -- PointerGrabTest ImplicitGrabRawEvents false false XX ScreenSaverTest ScreenSaverActivateDeactivate true false ++ TouchDeviceChangeTest DeviceChangedEventPointerToTouchSwitch true true ++ TouchDeviceChangeTest DeviceChangedEventTouchToPointerSwitch true true XX TouchDeviceChangeTest NoCursorJumpsOnTouchToPointerSwitch true false ++ TouchDeviceTest DisableDeviceEndTouches true true -- TouchEventHistoryTest EventHistoryReplay/0 false false -- TouchEventHistoryTest EventHistoryReplay/1 false false -- TouchEventHistoryTest EventHistoryReplay/10 false false ...
This simply shows us which tests had which result. Cases with code ++ succeeded when they were expected to, -- failed when they were expected to. XX is what you need to look for, it indicates that the test outcome differs to the status in the registry. In this case, two tests now succeed when before they didn't - usually a good outcome.
Once a fix is merged, you can update the status of the bug:
$> xit-bug-registry -f registry.xml edit \ XTest DisabledDevicesProperty \ set-status successIf this test fails in the future, you'll now be able to spot it.
The information is all in the registry file and though I admit the interface is still clumsy, it's quite simple to keep a set of registries (e.g. upstream, Fedora, RHEL, etc.) and thus a known set of test outcomes for each.
Finally, as test cases are being added, it's important to update the registry. This is as simple as:
$> xit-bug-registry create results-with-new-testcases.xml > new-registry.xml $> xit-bug-registry merge registry.xml new-registry.xmlThis command will merge any new test cases into our original registry, but won't update the status on any existing ones.
