Acceptance Testing, the Agile way

There is one thing in software development that any developer should remember at all times: test, retest and test again. Any true Agile project revolves around testing. Test your code, screens, databases and test whether your user stories actually work. Nothing is left to chance and testing is done before everything else and after your done and your code will undergo regression testing when you’re long gone. Pretty obvious, you might say, but some parts of testing are more obvious than others. Unit tests are pretty much covered, there are dozens of tools and frameworks for that, the same goes for screen testing. But one thing that still feels like new ground is functional testing (or acceptance testing).
In Agile projects there are a two constraints that apply to the ‘traditional’ way acceptance tests are performed: test first and agile (flexible) test cases. When working with dedicated testers, the developers need to work together with them as a single team. Using the ‘test-first’ approach, you need the input from the tester before you can start coding, otherwise you won’t know when your work does what it needs to. One other problem that both developers and testers face is responding to changes. When the customer wants a new feature, you can’t reply: “It was never build to do that!”. We’re agile, so we’ll refactor the code if needed. Changes also apply to tests, requirements change and the software changes (database changes, user interface changes, etc.). So the tester needs to be able to deal with those changes.
There are a few things we can tackle. The magic word here is ‘Fitnesse’. Fitnesse allows you to combine test case specification and execution on one page. This allows for very smart test cases, you could have something as simple as:
My test data:

The test:

You can actually ‘run’ this table and using some glue code (called Fixtures) you can actually test your application for searching people. This kind of test case format doesn’t require any knowledge of the inner workings of the application. Whether it’s a swing or web application, uses a database or a mainframe to store information. The fixture will make sure the test will work. That makes it more readable and agile because I can refactor my application and change the way we store data (let’s stop using that mainframe) and as long as the fixture is refactored too, the test is still executable.
That’s how we like to test, leaving the implementation details out of the test and define them on a functional level. We implement our fixtures using selenium so we can run the test directly in a web browser (such as firefox).
Although we have been working using this approach for quite some time now, there are still a few problems we haven’t quite solved yet. User acceptance testing is about proving that the application works exactly as it should. The tester needs to prove that no (serious) bugs exist, by testing complete scenarios (for example test the whole process of ordering a product, from registration to payment) but also test corner cases. But there is a problem with all these test cases, you need to change them when the application changes. After about a year of development, you’ll acquire hundreds of test cases. Suppose the registration process changes, and you used registration in dozens of cases, how do you change all of them?
One way to minimize impact of changes is to create the idea of unit testing in acceptance testing. Unit testing means you test one unit at a time, when unrelated units change, your test won’t break (because it wasn’t testing those). The same could apply to User Interface tests, you could test the registration part in many different ways, but when testing other parts of the system you should just be able to assume there is a user that can log in (again something the fixture can solve). This user could be specified as a default application user (a ‘John Doe’). Now you don’t have to go through the whole registration process, just to be able to log in to the application. Our test cases (for our web shop application) can now be written like this: “John searches for ‘ipod’ and finds these items…” You could call this a Domain Specific Language specific for testing user stories. You can have very powerful tests this way.
Although Fitnesse is a good tool for Agile testing, doing Acceptance Testing is always demanding and maybe more so when doing it Agile. Most customers use professional testers, but they are usually not used to Agile development. Often they rely on lots of excel sheets, lots of specifications and sometimes even loads of SQL scripts for their input-output tests. A lot of knowledge about the application is spread and duplicated across all those files. Helping them automate their tests in a more expressive and condensed way will keep the tests easy to change and readable by everyone. However, actually finding the most efficient way to give the testers the insight and control they need whilst staying Agile remains a challenge.