January 7, 2011
Hunting Regressions

Nobody likes doing repetitive boring tasks.  There’s a certain threshold at which I’ll bite the bullet and write a script/tool/what have you to automate that task.  There’s another lower threshold where I’ll complain about the task on the internet.  Hunting regressions with hg bisect falls somewhere between those thresholds.

What I want is a script (ideally in Python!) that will drive an hg bisect against a test from our test suite.  The requirements for said script are:

  • Accept tests that would run in our test suites (mochitest and crashtest are probably the most useful to me, but would be good to have them all eventually)
  • Accept a starting good revision and an ending bad revision
  • Start hg bisect and build the tree from source (clobber build)
  • Run the test against the build and proceed with the bisection appropriately
  • Handle revisions that fail to compile intelligently

This obviously won’t handle everything, it’s easy to imagine a revision that builds but crashes at startup, or a revision that changes a leak to a crash, but this would be very useful to me and many others.

If somebody is willing to write this script (prior art here, here, and here) I will provide a fast machine to run it on (clobbers from scratch in 8 minutes, so < 2 hrs for a full regression search) and set up a web or email gateway to it for the community to use.

Any takers? :-)

  1. khuey posted this
blog comments powered by Disqus