A lot of automated acceptance testing pioneers have come around and denounced their fate in heavy automated test suites. A recent article on InfoQ sums up the trend quite nicely. I am not going to jump on that bandwagon, but I will try to find the safe middle ground between the overzealously created maintenance burden and anarchy. The main point is that automating acceptance tests is the way to go, you just shouldn't automate and maintain useless tests. The tricky part is to find out what tests are useful and what tests are not. Before I start let me emphasize the difference between automated acceptance testing and automated integration testing or unit testing. Unit testing is absolutely essential. Anyone that tells you otherwise is either ignorant because he hasn't tried yet, or a moron. We can have a long discussion on how to do unit testing properly, but that's not the topic of this post. Automated integration testing (within reason) is extremely useful, and essential in some areas. Where you touch external systems or things that are otherwise very expensive to mock, some of reservations might be defensible. Automated acceptance testing is testing where the aim is to simulate user interaction with the system. When and how this is useful is at stake here. Let's list the arguments pro automated acceptance testing first:
- If you don't test it will break, the shorter the iteration the more you will have to test, hence in the shorter the iteration the faster the ROI.
- If you use an automated test framework that is understandable to the customer you can delegate test writing back to them, making properly defined functionality their problem.
- Human beings are very easily persuaded to overlook certain details, automated tests are impervious to that sort of corruption
- Tests are expensive to maintain. They will never ROI because the more you change the more you need to change your tests ultimately increasing the cost of the project.
- Customers don't understand the test tools, so the developers end up writing and maintaining these tests anyway
- Tests fail all the time when you change little things where a tester would use his head and leave you alone if the change was sensible