The libinput test suite takes somewhere around 35 minutes now for a full run. That's annoying, especially as I'm running it for every commit before pushing. I've tried optimising things, but attempts at making it parallel have mostly failed so far (almost all tests need a uinput device created) and too many tests rely on specific timeouts to check for behaviours. Containers aren't an option when you have to create uinput devices so I started out farming out into VMs.
Ideally, the test suite should run against multiple commits (on multiple VMs) at the same time while I'm working on some other branch and then accumulate the results. And that's where git notes come in. They're a bit odd to use and quite the opposite of what I expected. But in short: a git note is an object that can be associated with a commit, without changing the commit itself. Sort-of like a post-it note attached to the commit. But there are plenty of limitations, for example you can only have one note (per namespace) and merge conflicts are quite easy to trigger. Look at any git notes tutorial to find out more, there's plenty out there.
Anyway, dealing with merge conflicts is a no-go for me here. So after a bit of playing around, I found something that seems to work out well. A script to run make check and add notes to the commit, combined with a repository setup to fetch those notes and display them automatically. The core of the script is this:
make check rc=$? if [ $? -eq 0 ]; then status="SUCCESS" else status="FAIL" fi if [ -n "$sha" ]; then git notes --ref "test-$HOSTNAME" append \ -m "$status: $HOSTNAME: make check `date`" HEAD fi exit $rcThen in my main repository, I add each VM as a remote, adding a fetch path for the notes:
[remote "f22-libinput1"] url = f22-libinput1.local:/home/whot/code/libinput fetch = +refs/heads/*:refs/remotes/f22-libinput1/* fetch = +refs/notes/*:refs/notes/f22-libinput1/*Finally, in the main repository, I extended the glob that displays notes to 'everything':
$ git config notes.displayRef "*"Now git log (and by extension tig) displays all notes attached to a commit automatically. All that's needed is a git fetch --all to fetch everything and it's clear in the logs which commit fails and which one succeeded.
:: whot@jelly:~/code/libinput (master)> git log commit 6896bfd3f5c3791e249a0573d089b7a897c0dd9f Author: Peter Hutterer
Date: Tue Jul 14 14:19:25 2015 +1000 test: check for fcntl() return value Mostly to silence coverity complaints. Signed-off-by: Peter Hutterer Notes (f22-jelly/test-f22-jelly): SUCCESS: f22-jelly: make check Tue Jul 14 00:20:14 EDT 2015
Whenever I look at the log now, I immediately see which commits passed the test suite and which ones didn't (or haven't had it run yet). The only annoyance is that since a note is attached to a commit, amending the commit message or rebasing makes the note "go away". I've copied notes manually after this, but it'd be nice to find a solution to that.
Everything else has been working great so far, but it's quite new so there'll be a bit of polishing happening over the next few weeks. Any suggestions to improve this are welcome.