Blog

Don’t let the mutants escape!

02 May, 2022
Xebia Background Header Wave

Writing automated tests is something that every developer should do. Unit testing is one of the most used types. But how do you know that you’re writing good quality tests? This is where mutation testing comes into the picture. It’s the test that tests the tests. Under normal circumstances it can be the most easy form of testing of them all and provides great insight in how well your unit tests are written. In this blog I will describe what mutation testing is and most important: how you don’t let the mutants escape!

Unit testing and code coverage

To understand mutation testing, we first need to briefly touch unit testing and code coverage. When writing code you should always make sure that sufficient unit tests are written. There is a somewhat arbitrary number floating out there, taken from the pareto principle, that 80% of the code should be covered by unit tests.
I personally do not agree with this statement and always advise aiming for 100% code coverage. This with a disclaimer, but the 80% is too arbitrary. It can already be achieved by writing the simplest tests and padding them with some of the harder tests. This can come to light during code reviews, but can also be missed quite often. Therefore my point of view is that if you do not want to write a test for a piece of code, then exclude it from your code coverage. That way, during code reviews, others can have an opinion about this. You’ve consciously thought about what you would and would not test.

Why you should use mutation testing

Testing is all about building confidence. This confidence is necessary to make sure that you can deploy your service to your customers consistently. One important aspect of this is that the service will ‘always’ respond expectedly and is stable. As a developer you’re responsible for writing the automated tests for the code that you’ve written. How will you validate that the tests you’ve written are any good? That’s where mutation testing shines.

What is mutation testing

Mutation testing is a form of fully automated testing that uses your source code, your unit tests and your code coverage. The only thing required is that you provide your testing tool with all the necessary information (paths, coverage reports etc.). From this information the mutation testing tool will try to find pieces of code that can be altered. This is called the mutation.

Example

Look at the following example.


if (x > 0) {
  print("X is a valid number");
}

The mutation testing tool will try to change it and run the applicable tests again.


if (x >= 0) {
  print("X is a valid number");
}

As you can see this can drastically change the flow of your code, because it can enter a branch unexpectedly. This is a very straightforward example, but these kinds of mistakes can easily be made accidentally.
The mutation testing tool will do these kinds of changes throughout the code under test. It will gather, from the code coverage report, the unit tests that are reported to test a specific piece of code and rerun the tests. If none of the tests fail on the change, then this change can be made accidentally and a bug can unknowingly be introduced. This is what is called an “escaped mutant“. If at least one test fails, then we’ve killed the mutant. Other outcomes may be that the change results in broken code or endless loops. These will generally be marked and ignored.

What to do with escaped mutants

When your mutation testing tool has completed its tests, you will be presented with a report of findings. Typically the report will provide a list of the exact changes that have been made with escaped mutants. From this report you should check your tests and either make the necessary changes to at least one of the tests to cover the mutation or add an extra test for this. My advice would be to aim for 100% mutant free tests, because: How will you decide what mutant is not important, while you’ve already consciously decided on what test would be important?

Where to do mutation testing

Mutation testing is done in the testing pipeline, right after the stage where you have run your unit tests and collected your code coverage report. Obviously enough this should be done before the deployment pipeline. Generally speaking all testing pipelines should have completed, before any merge into the main branch should be permitted. Once in the main branch the code should already have been validated.

Mutation testing tools

Of course you can use your preferred search engine to look for a tool fitting your programming language. Most modern programming languages will have at least one. If none exist, then maybe you can start an open source initiative to create one.
For readers who do not want to spend much time on searching, here is a non-exhaustive list of mutation testing tools:

Conclusion

Mutation testing is a technique used to validate the quality of your unit tests. It is a fully automated testing tool that will make changes to your source code and then run the unit tests to verify that at least one is failing. If a test fails we have killed the mutant. If not then… don’t let the mutants escape!
Photo by Sangharsh Lohakare on Unsplash.

Questions?

Get in touch with us to learn more about the subject and related solutions

Explore related posts