One of the problems we face in X is that testing can be incredibly painful. Somebugs only happen with a specific server state, a specific device and a specific application. Sometimes we don't spot regressions for several releases, and in some cases that means we can't easily go back and fix it either - users may have gotten used to the new behaviour already.
A few months ago, Chase Douglas announced the xorg-gtest framework. I started extending xorg-gtest and writing tests against it. Olivier Fourdan soon joined in with his own tests. The result of that are the xorg integration tests (XIT) found here:
http://cgit.freedesktop.org/~whot/xorg-integration-tests/
This post is an outline of what the XIT do and what we could potentially make it do. We currently have over 50 tests in the XIT repository. Most of the tests are still quite simple, but I've already added some that test for a specific bug and that certainly helped testing. In one case, a failing test reminded me that I forgot to send a pull request :)
googletest and xorg-gtest
xorg-gtest is based on the googletest C++ framework. It provides a bunch of classes with virtual functions that are called set up or tear down a test. For the tests, it provides a set of assertion macros. xorg-gtest provides a set of classes on top of googletest that make starting an X server simpler. The XServer class provided hooks up config and log files, waits for the server to accept connections and terminates the server gracefully. It also includes a class for starting evemu devices and re-playing input event sequences. There are a few additional helper classes in the XIT, for example one that makes writing configurations easier. They may move to xorg-gtest at some point in the future.
Example tests
Your average test now essentially consists of three steps: write out a specific config, start the server and then query that server for information or test it for behaviour.
A straightforward test is the acecad driver test below.
TEST(AcecadInputDriver, WithOptionDevice)
{
XOrgConfig config;
xorg::testing::XServer server;
config.AddInputSection("acecad", "--device--",
"Option \"CorePointer\" \"on\"\n"
"Option \"Device\" \"/dev/input/event0\"\n");
config.AddDefaultScreenWithDriver();
StartServer("acecad-type-stylus", server, config);
::Display *dpy = XOpenDisplay(server.GetDisplayString().c_str());
int ndevices;
XIDeviceInfo *info;
info = XIQueryDevice(dpy, XIAllDevices, &ndevices);
XIFreeDeviceInfo(info);
ASSERT_EQ(ndevices, 7);
config.RemoveConfig();
server.Terminate(3000);
server.RemoveLogFile();
}
A slightly more involved test that makes use of the googletest class hierarchy would be the void driver test below. It makes use of a InputDriverTest class that starts the server for us and tears it down afterwards. So all we need is a couple of lines for the class definition and then the test itself.
class VoidInputDriverTest : public InputDriverTest {
public:
/**
* Initialize an xorg.conf with a single CorePointer void device.
*/
virtual void SetUp() {
InputDriverTest::SetUp("void");
}
};
TEST_F(VoidInputDriverTest, VoidDriver)
{
ASSERT_EQ(FindInputDeviceByName(Display(), "--device--"), 1);
}
These two are just examples as used in the integration test suite. I recommend reading the googletest documentation for a bit more info on test fixtures and assertions. xorg-gtest also is well-documented though I recommend building the doxygen sources.
Emulating input devices
xorg-gtest provides a Device class that, based on evemu recordings, can simulate input devices. So with the current implementation, this as simple as:
SetDevice("keyboards/AT-Translated-Set-2-Keyboard.desc");
dev->Play("keyboards/qwerty.events");
...
dev->PlayOne(EV_KEY, KEY_Q, 1, true); // press
dev->PlayOne(EV_KEY, KEY_Q, 0, true); // release
If you now check for the keysym received, you can easily write a few simple XKB layout tests.
Fighting the build system
As convenient as tests are to write, googletest is not the easiest thing to deal with. The piles of macros subclass your classes and the C++ ODR essentially prevents googletest from being built into a library. Autotools on the other hand make it difficult to compile an external project into your own file. That is the main reason for one big collection of tests instead of a per-driver test suite. Copying Makefile.am's around is painful enough, but at least manageable when only one project does it.
The current need for three different repositories (googletest, xorg-gtest and XIT) doesn't make things easier. I'm somewhat tempted to merge XIT into xorg-gtest, be it as git submodule or some other means.
Having said that, since starting to write the tests I'm also really tempted to merge the drivers into the server, it would make things a lot easier
ifdefs maze
To make the tests useful, they should be able to run on most systems. Right now, I've got some simple defines to test for RHEL6 and also XI 2.2. But I don't think this will scale too well. Every system may have different defaults, and short of having platform-specific tests (which may only be different in one single line) I don't know what the good answer is. I do want it answered soon, before it gets out of hand.
Scalability
Running all tests takes a while. In the current test design, every single test starts a new server and thus it takes a couple of seconds for even the quickest test to succeed. This works with 50 test cases, but it won't work with 500 and even less with 5000. So we need some solution for that. Not re-starting the server is the obvious one and there will be tests that won't require that.
Output testing
Most of the test runs I do either on my laptop or on a virtual machine. For input tests, this is easy since we only need to set up the dummy driver and the rest can be emulated. For output tests, not so much so a good approach of only running tests applicable for the hardware would be useful. I haven't thought too much about this yet.
Testing XIT
Right now, it sits in my personal home directory but I hope that after XDC this year, we can decide what the future of this test suite can be and where to host it proper.
In the meantime, grab it from here:
