Regression Testing with an Agile Mindset

When applying agile methods to existing software development practices, it is often useful to start from the basics. For regression testing this boils down to:

1. Regression testing ensures continuity of business functions.
2. In an agile context it helps the team focus on the (new) functionality to develop in the sprint while maintaining overall stability.
3. To maximally utilize regression testing in an agile context, test automation and continuous execution should be applied.
4. It is important to realize the difference between functional tests and regression tests to avoid some common pitfalls.

Regression testing ensures continuity of business functions The target area of the regression test is the complete set of (end-to-end) business functions that a system encompasses. The same tests are executed repeatedly over time to ensure that the required behaviours of the system remain stable. In an agile context this means that regression test results are of great value to the product owner at the end of each sprint, when he has to evaluate and accept not only the newly developed functionality, but the overall functionality of the system after the changes have been made. Ideally, there should be no regression in the functionality of the system. In real-life, complex systems however, a trade off can be made between the immediate value of the new functionality and regression bugs that have (unintentionally) been introduced. But the Product Owner should be very careful! A zero-tolerance policy towards bugs is in the long run far more effective than accepting even small regressions for the sake of faster development. Therefore it is better to decrease the scope then to accept regression. Of course expectations and sometimes entire tests have to be adapted as business functions change or new tests have to be added when additional functionality is developed. In an agile context, where change is continuous and gradual, changes to business functions should not be disruptive. Maintaining the regression test suites can be a relatively easy job. Focus on new functionality One of the main success factors of the agile approach (specifically SCRUM) is, that it allows the entire team to focus on the functionality that has been planned for the current sprint. This focus allows the team to aim for the best quality of the solution in every aspect, be it architecture, design, implementation or testing. In a complex environment, you cannot expect a team to focus on a particular part while at the same time understanding the consequences and risks for the entire system. Regression tests provide a safety net for the team in this situation. Any areas of the environment that might have been missed in the analysis of the problem or the proposed solution, will be shown by the regression test provided that the coverage of the regression test suites is sufficient. Continuous automated regression tests give rapid feedback To allow the team to respond efficiently to any findings from the regression tests, it is imperative that the regression tests are executed shortly after changes have been made. Late feedback means that it is harder to track the source of the unexpected behaviour, to analyse the error and to correct the implementation. The team might have moved on to implement other changes in other areas of the system, which means that it will take more effort to switch back to the area in which the problem was found. Since the problem did not surface during functional testing, it is very probable that the cause of the problem is of a more structural nature and the architecture and design of the solution have to be re-evaluated. Most likely new functional tests have to be added to ensure that the issues found are not just the tip of the iceberg. The required rapid feedback can often only be achieved when the regression tests are executed fast and automatically. To this end regression tests need to be scripted and run on an automatic build environment. Regression test suite != sum(Functional test suites) The purpose of functional tests is to explore the behaviour of specific business functions and highlight corner cases. As discussed above, the purpose of regression testing is to give and overall overview of the entire system. These two purposes are distinct and therefore functional test suites should not simply be used as regression tests. In an agile context, functional tests are used to prove that the team has delivered the functionality that is planned for a sprint. The development of the functional test cases is part of the sprint in which the functionality is developed. The functional test cases are thus a basis for the product owner to accept the new functionality. In many agile teams I have seen that functional tests that have been developed for previous sprints are run in succession as regression tests for the current and coming sprints. Although this is an easy and cheap approach to build a regression test suite, there are a number of problems with this approach:

1. Since the system is developed in iterative and evolutionary fashion, the same business functions are often changed and enhanced multiple times over many sprints. As a result many functional tests are written for the same business function. This means a lot of "test duplication". Since the business function is changing, the old test scripts have to be adapted and the required effort will grow with every sprint.
2. At the same time, the value of the regression tests has not increased, since the same business function is being tested. The overall overview of the system has not improved, but rather been obfuscated by adding more data.
3. As a side effect it also takes longer to execute the regression tests and to analyse the results.

Therefore it is better to work with (or towards) a smaller, relatively stable set of regression tests. This set should be constructed on the basis of knowledge of the business functions and of the technical implementation, to ensure sufficient but not superfluous coverage. It is thus the responsibility of the entire team. So how does the team develop this set of regression tests?

1. The product owner compiles a list of all business functions that should be included in the regression test. This can happen gradually over several sprints while the team learns what functions are important. Creating the tests can also be done gradually.
2. Functional tests in the sprint are basically "throw away" tests: They are not intended to be maintained for long. Perhaps you keep the tests so you can update and reuse some of them later.
3. Some functional tests can be upgraded to regression tests, when they cover certain business functions better then existing tests. The existing tests can then be taken out of the regression test suite.
4. Regression test scripts are updated in the sprint when necessary. This is part of the UserStory and should thus be taken into account when estimating.

Conclusions

1. Regression testing has an important role in Agile development. In addition to its usual purpose of proving stability, it helps the team focus on the functionality in the current sprint.
2. Although it is tempting to use the functional tests of the past sprints as regression tests for the current sprint, this will lead to an unwieldy regression test suite and will blur the purpose of the regression test suite by introducing a lot of duplication. In stead the regression test suite should be constructed to reflect the (most important) business processes.