Wednesday, July 15, 2015

Using git-notes for marking test suite successes

The libinput test suite takes somewhere around 35 minutes now for a full run. That's annoying, especially as I'm running it for every commit before pushing. I've tried optimising things, but attempts at making it parallel have mostly failed so far (almost all tests need a uinput device created) and too many tests rely on specific timeouts to check for behaviours. Containers aren't an option when you have to create uinput devices so I started out farming out into VMs.

Ideally, the test suite should run against multiple commits (on multiple VMs) at the same time while I'm working on some other branch and then accumulate the results. And that's where git notes come in. They're a bit odd to use and quite the opposite of what I expected. But in short: a git note is an object that can be associated with a commit, without changing the commit itself. Sort-of like a post-it note attached to the commit. But there are plenty of limitations, for example you can only have one note (per namespace) and merge conflicts are quite easy to trigger. Look at any git notes tutorial to find out more, there's plenty out there.

Anyway, dealing with merge conflicts is a no-go for me here. So after a bit of playing around, I found something that seems to work out well. A script to run make check and add notes to the commit, combined with a repository setup to fetch those notes and display them automatically. The core of the script is this:

make check
if [ $? -eq 0 ]; then

if [ -n "$sha" ]; then
    git notes --ref "test-$HOSTNAME" append \
        -m "$status: $HOSTNAME: make check `date`" HEAD
exit $rc
Then in my main repository, I add each VM as a remote, adding a fetch path for the notes:
[remote "f22-libinput1"]
        url = f22-libinput1.local:/home/whot/code/libinput
        fetch = +refs/heads/*:refs/remotes/f22-libinput1/*
        fetch = +refs/notes/*:refs/notes/f22-libinput1/*
Finally, in the main repository, I extended the glob that displays notes to 'everything':
$ git config notes.displayRef "*" 
Now git log (and by extension tig) displays all notes attached to a commit automatically. All that's needed is a git fetch --all to fetch everything and it's clear in the logs which commit fails and which one succeeded.
:: whot@jelly:~/code/libinput (master)> git log
commit 6896bfd3f5c3791e249a0573d089b7a897c0dd9f
Author: Peter Hutterer 
Date:   Tue Jul 14 14:19:25 2015 +1000

    test: check for fcntl() return value
    Mostly to silence coverity complaints.
    Signed-off-by: Peter Hutterer 

Notes (f22-jelly/test-f22-jelly):
    SUCCESS: f22-jelly: make check Tue Jul 14 00:20:14 EDT 2015

Whenever I look at the log now, I immediately see which commits passed the test suite and which ones didn't (or haven't had it run yet). The only annoyance is that since a note is attached to a commit, amending the commit message or rebasing makes the note "go away". I've copied notes manually after this, but it'd be nice to find a solution to that.

Everything else has been working great so far, but it's quite new so there'll be a bit of polishing happening over the next few weeks. Any suggestions to improve this are welcome.


jpritikin said...

You might want to correct your spelling of success.

Peter Hutterer said...

whoops. corrected, thanks for that

mpe said...

You can automatically copy notes on rebase:

But the test results may no longer be valid if you've significantly changed the commit.

Philip said...

A vaguely related project I worked on recently stores D-Bus API data in git-notes for later comparison between releases, to detect API breaks:

git-notes are really quite useful, despite the odd API! :-D

Michael Haggerty said...

If your tests only depend on the contents of the tree, and not on Git-level stuff like the name of the branch/tag or the SHA-1 of the corresponding commit, then it makes more sense to record the test results against the *tree* rather than against the commit. That way, if you change a commit message without changing the contents of the tree, the old test results remain valid.

Unfortunately, the standard tools don't display notes that are attached to trees, so you would need your own tool for inquiring about the test status.