Xebia Background Header Wave

In many organizations, automated testing lags behind and becomes a bottleneck for successful continuous delivery. Either tests do not provide enough confidence or companies take a very traditional approach, resulting in releases either introducing substantial risks or becoming costly. These situations often seem hard to fix and “Fix Testing” becomes a painful epic somewhere in your backlog. 
This article helps you challenge your automated testing and to improve upon it in a more continuous fashion. 

Test automation is one of the practices that helps teams gain confidence earlier in the development process. It reduces the cost of testing and supports keeping long-term, high-development velocities. Test automation is not a project or a one-off—it is part of the development lifecycle and should be seen as its own system that needs to evolve over time as the requirements and system under test change and mature.

Prerequisites

To start improving, the team must have a basis and agreement. It is okay to start out simple and not overthink, but these two things are key:

  • Have tests – having some tests is better than no tests, as it provides a feedback mechanism for improvement
  • Definition of Done – a team needs to agree that executing these tests is part of delivering changes and when changing code, new tests should be added. This results in a steadily-growing set of tests becoming part of the development cycle. If this is not commonly accepted as part of ‘Done’, you might find automated test cases become a thing at the bottom of your backlog that becomes hard to plan or get budget for.

When writing tests, it is good to stick to the F.I.R.S.T principles of testing (popularized by the book Clean Code by Robert C Martin). These principles tell us our tests should be:

  • Fast
  • Independent
  • Repeatable
  • Self-validating
  • Thorough

“Independent” is an especially common pitfall – it is very tempting to have tests depend on program state, its environment or on other tests. In the long run, this will result in tests that are hard to maintain and understand.

5 ways to improve

When we have the basis for our automated testing, we can continuously challenge ourselves to improve. As the number of tests grows, or other parameters change, you may feel your current approach does not scale well enough anymore (e.g., it gets slow, expensive, or hard to understand). These 5 ways should help you challenge your current testing and identify opportunities to secure your team’s long-term throughput.

  1. Execute Tests Earlier

Ideally, you want your test feedback at the same time you make a change. From a developer perspective, tests should be able to run locally and fast. From a process perspective, you should aim to ensure they are executed before the code is merged. This is the easiest way to relate a failure to a code change, while keeping the main branch stable (and releasable). Delivery pipelines and source code management systems help—for example, you can gate pull requests on a failing test.

  1. Increase Coverage

One way to increase the confidence you get from your tests is to have more of them. For example, if regression tends to happen around the same functionality or component, it might be worth investing more in that area when it comes to test coverage. Keep in mind that tests do not come for free (writing time, execution time, maintenance), so make sure the effort is balanced by realistic risk.

  1. Improve Testability

This is often the hardest, yet most rewarding. By improving your system under tests in a way that invites more testing, your testing becomes easier in different ways. For example, you could refactor a crucial part so that it (or a part of it) can be tested in isolation. At the same time, improving its deployability makes the execution of tests easier. A. Tarlinder says a lot on testability in his book “Developer Testing” and provides good insights on what to look for.

Testability from Developer Testing

 

  1. Lower-level testing

When your testing stops scaling, parts can be extracted into tests that focus on a smaller part of your application. How much this is possible depends on the testability of the system. If the people who own the tests do not own the code, they need very good communication with those who do. If you find yourself in a situation where only a part of your team is responsible for automated testing, for example, the QA department, shift-left testing is useful.

  1. Write tests earlier

Although testing is part of ‘Done’, it can still be seen as an afterthought, a final step. This can result in testability issues, making testing harder, as well as tests becoming more tightly coupled to the implementation. By writing tests earlier, you find better opportunities to design for testability.

Conclusion

Test automation is key in keeping a long-term, high-development velocity. However, it is not something you just implement; it should become a core part of the development lifecycle and prevent large testing epics in your backlog. In your journey you will have the opportunities to improve upon:

  • Executing tests earlier to get faster feedback
  • Increase coverage to all relevant use cases
  • Improve testability to make both the tests and the system more maintainable
  • Test at a lower level for testing that scales better
  • Write tests earlier to really make them part of your user story

If we were to take it one step further, where test automation is not only a tool to validate what we have made, but rather a way to write self-testing code, we find practices like Test-Driven Development (TDD). As Dave Farley and Arjan Molenaar wrote: “TDD is not about writing unit tests”. Instead, it is a design tool that happens to result in testable designs with thoroughly tested code. With TDD, tests are written at a very early stage (before writing code), and as the feedback cycle between writing code and tests is so short, there is a more natural incentive to keep tests fast.

So, if the idea of having to improve your test automation very consciously and continuously scares you, Test-Driven Development might be just the thing for you.

Questions?

Get in touch with us to learn more about the subject and related solutions

Explore related posts