Measure the right coverage
I’ve found many people to care for a high unit test coverage. It tells you something about how well your code is tested. Or does it?
Unit tests typically test the smallest piece of code. It is an excellent strategy to write your tests in conjunction with the production code. The tests help you shape the interfaces and help explore the problem domain.
Big question is: does the business/product owner care? What do those tests tell him (or her) about the actual functionality delivered? Fairly little really, if any at all. This boils down to the next question: why care about unit test coverage then?
The obvious thing to do here is to test the coverage of your acceptance tests. Acceptance tests define a requirement from a business point of view, i.e. they harness the real requirement for the product.
Agile software development is about creating business value. The acceptance tests represent the business value created.
Having a coverage figure on those tests tells you one or two things. It’s the proof of the pudding. If your coverage figure is fairly low, there are areas of the application not covered by any acceptance tests. Does this mean there are no requirements related to this code? Or is it left uncovered for a reason?
Do I advocate to abandon unit tests all together? Most definitely not. As I stated at the beginning of this post, unit tests make an excellent vehicle for designing your code. Just do not focus on unit test coverage: use unit tests to construct and test the more difficult parts of the code. After the code is constructed and the acceptance tests turn green, figure out which unit tests have value (e.g. for future development or maintainance) and get rid of the rest.
You’ll need to maintain your unit tests, just like you do with the “production” code. Figure out which tests are important and which are not. Tests that are close to the code and structure are possibly not the best tests to keep around. They’ll need to be changed with every code modification or refactoring.
Just keep in mind we’re creating business functionality (value) here. Care about that. Measure that. Leave the details for what they are and measure what your business cares about: functionality.
Acceptance tests are your stable entry point in making sure modifications do not break the system.
Mind the acceptance tests might be written/executed using a unit test framework, although the readability for the stakeholders should not be cluttered by technical details. ↩