Some Random Blog

Recently I've been hearing more and more about automated "mutation testing" tools such as Jester (for Java) and Heckle (for Ruby). These are tools that will try to break your code by doing things like changing the truth values of clauses of if statements, modifying immediate values, changing array indices, and returning bad data, then running your unit test suite on the modified code. If the modified code passes your test suite, then chances are your test suite is either lacking coverage or you have dead code.

Now, I love automated tests. It's not just the bugs I catch early or the debugging time I save or the live documentation aspects of the tests, but the confidence having my code fully tested gives me to work with and refactor the code to make it the best code possible. However, I could never shake the feeling that this confidence was a little misplaced, especially when I discovered bugs in the tests or code that wasn't being tested despite coverage numbers being high.

It was always hard when I'd evangelize unit testing and people would ask, somewhat sarcastically, "How do I know the tests are right? Do I have to write tests for the tests?" It always seemed to me to lead to an infinite regression of questions that I didn't have the answer to.

Luckily, someone (or some group) was smart enough to come up with the idea of brute-force testing tests, and I think these sorts of tools will lead to code that can be considered "correct" for all real purposes without resorting to formal methods. My somewhat shaken confidence in my unit testing tools can be regained.

I haven't had the chance to play with these tools in-depth yet, but I'm definitely going to use them on my next project. I'll probably talk more about them then.