Some Random Blog

For my day job, I work on Legacy Code. As legacy code, it needs unit tests. Unfortunately, this particular legacy code is used by millions of people on a regular basis, precluding refactoring to make unit testing easier. I needed a way of testing our code without changing any of the code that was put into production. Luckily, I discovered a way of exploiting the C/C++ linker to allow me to stub out our dependencies, which I'll talk about here.

For me, the hardest part of writing C/C++ code is managing dependencies. It's very easy to get locked into a design due to dependencies that will make life harder later. The code that I've been working on is over ten years old in places, and wasn't written with dependency management in mind. This means that most of the functionality of the code calls APIs that I have no control over and I don't want to run in my unit tests, because they do things that change the state of the underlying system. Luckily for me, the C/C++ linker that I've cursed on many an occasion has become a useful tool for softening these dependencies. I'm sorry for doubting you, linker.

The current archtecture of the system looks like this:

I wanted it to look something like this:

In order to control both ends of the system from my tests, however, I needed to override the methods that my code calls that are exposed by the underlying layer that I have no control over. Of course, in the Software Engineering world, the only problem that can't be solved by another layer of abstraction is performance, so that's what I did -- created a proxy layer as a library that re-implemented each of the methods exposed by the underlying layer. This library would forward these calls to a global instance of a proxy class with stub functionality that I could override in my test code. This looked something like this:

Ruby came to the rescue here, as it tends to do, allowing me to write a quick and dirty (ok... filthy) script to parse the header file of the dependency and generate these classes for me. This saved me the trouble of doing it by hand, which would make me cry. I took these generated files and generated a .dll and a .lib, which I statically linked into my test code, making sure to link it before the real .lib I depended on.

As you can see, I could now pinch my production code between my test code, enabling me to control what goes in and look at what comes out. The best part of all of this is that it didn't require a single change to the code that goes into production, allowing me to cover methods with tests before refactoring. For easily broken code that is used by an enormous population, this is a huge deal. But it's not the end.

I've heard many experienced unit testers say that the most difficult places to test are the "Ends of the World..." that is, upstream and downstream dependencies. The most common way I've seen people avoid this is by writing proxy objects that isolate these dependencies from the code. This leads to an architecture that looks like this:

This leads to a few benefits. First, your interface to the dependent code is written by you, so you can expose only the functionality that you actually use in a way that you like. Second, when your dependency breaks you, you have a simple test case that you can toss at them, which avoids the communication problems that plague large, distributed teams. Finally, you can subclass and override this proxy, allowing the injection of test code on both ends of the code. That's a much easier way of doing what I described above, and doesn't require any build hackery. It's the route I'm planning on taking once I get good enough test coverage that I feel comfortable making wild, invasive surgery on the code.

Much love for Michael Feathers' book "Working Effectively With Legacy Code", which gave me the idea and the confidence that this could in fact be done.

Recently I've been hearing more and more about automated "mutation testing" tools such as Jester (for Java) and Heckle (for Ruby). These are tools that will try to break your code by doing things like changing the truth values of clauses of if statements, modifying immediate values, changing array indices, and returning bad data, then running your unit test suite on the modified code. If the modified code passes your test suite, then chances are your test suite is either lacking coverage or you have dead code.

Now, I love automated tests. It's not just the bugs I catch early or the debugging time I save or the live documentation aspects of the tests, but the confidence having my code fully tested gives me to work with and refactor the code to make it the best code possible. However, I could never shake the feeling that this confidence was a little misplaced, especially when I discovered bugs in the tests or code that wasn't being tested despite coverage numbers being high.

It was always hard when I'd evangelize unit testing and people would ask, somewhat sarcastically, "How do I know the tests are right? Do I have to write tests for the tests?" It always seemed to me to lead to an infinite regression of questions that I didn't have the answer to.

Luckily, someone (or some group) was smart enough to come up with the idea of brute-force testing tests, and I think these sorts of tools will lead to code that can be considered "correct" for all real purposes without resorting to formal methods. My somewhat shaken confidence in my unit testing tools can be regained.

I haven't had the chance to play with these tools in-depth yet, but I'm definitely going to use them on my next project. I'll probably talk more about them then.