Mutation Testing: How Good are your Unit Tests?

You write unit tests for every piece of code you deliver. Your test coverage is close to 100%. So when the point comes when you have to make some small changes to the existing code, you feel safe and confident that your test suite will protect you against possible mistakes.
You make your changes, and all your tests still pass. You should be fairly confident now that you can commit your new code without breaking anything, right?

Well, maybe not. Maybe your unit tests were fooling you. Sure they covered every line of your code, but they could have performed the wrong assertions.
In this post I will introduce mutation testing. Mutation testing can help you find omissions in your unit tests.

Let’s begin with a small example:

[java]
package com.xebia;

public class NameParser {
public Person findPersonWithLastName(String[] names, String lastNameToFind) {
Person result = null;
for(int i=0; i <= names.length; i++) { // bug 1
String[] parts = names[i].split(” “);
String firstName = parts[0];
String lastName = parts[1];
if(lastName.equals(lastNameToFind)) {
result = new Person(firstName, lastName);
break;
}
}
return result;
}
}
[/java]

NameParser takes a list of strings which consist of a first name and a last name, searches for the entry with a given last name, instantiates a Person object out of it and returns it.
Here is the Person class:

[java]
package com.xebia;

public class Person {
private final String firstName;
private final String lastName;

public Person(String firstName, String lastName) {
this.firstName = firstName;
this.lastName = lastName;
}

public String getFirstName() {
return firstName;
}

public String getLastName() {
return firstName; // bug 2
}
}
[/java]

You can see that there are two bugs in the code. The first one is in the loop in NameParser, which loops past the end of the names array. The second on is in Person, which mistakenly returns firstName in its getLastName method.

NameParser has a unit test:

[java]
package com.xebia;

import org.junit.Before;
import org.junit.Test;
import static org.junit.Assert.assertEquals;

public class NameParserTest {
private NameParser nameParser;
private String[] names;

@Before
public void setUp() {
nameParser = new NameParser();
names = new String[]{“Mike Jones”, “John Doe”};
}

@Test
public void shouldFindPersonByLastName() {
Person person = nameParser.findPersonWithLastName(names, “Doe”);
String firstName = person.getFirstName();
String lastName = person.getLastName();
assertEquals(“John”, firstName);
}
}
[/java]

The unit tests covers the Person and NameParser code for 100% and succeeds!
It doesn’t find the bug in Person.getLastName because it simply forgets to do an assertion on it. And it doesn’t find the bug in the loop in NameParser because it doesn’t test the case where the names list does not contain the given last name; so the loop always terminates before it has a chance to throw an IndexOutOfBoundsException.

Especially the last case is easy to overlook, so it would be nice if there existed a tool which could detect these kinds of problems.
And there is one: actually there are a couple. For this post I have chosen PIT, down at the end are links to some alternatives.

But first: what is mutation testing?

A mutation test will make a small change to your code and then run the unit test(s). Such a change is called a ‘mutant’. If a change can be made and the unit tests still succeed, it will generate a warning saying that the mutant ‘survived’.
The test framework will try to apply a number of predefined mutants at every point in your code where they are applicable. The higher the percentage of the mutants that get killed by your unit tests, the better the quality of your test suite.
Examples of mutants are: negating a condition in an If statement, changing a conditional boundary in a For loop, or throwing an exception at the end of a method.

Putting NameParser’s testcase to the test with PIT

PIT stands for Parallel Isolated Test, which is what the project originally was meant for. But it turned out to be a much more interesting goal to do mutation testing, which required much of the same plumbing.

PIT integrates with JUnit or TestNG and can be configured with Maven, Gradle and others. Or it can be used directly as a plugin in Eclipse or IntelliJ.
I’m choosing for the last option: the IntelliJ plugin. The setup is easy: just install PITest from the plugin manager and you are ready to go. Once you’re done, you’ll find a new launch configuration option in the ‘edit configurations’ menu called PIT.

You have to specify the classes where PIT will makes its mutations under ‘Target classes’.
When we run the mutation test, PIT creates a Html report with the results for every class.
Here are the results for the NameParser class:

As you can read under ‘Mutations’, PIT has been able to apply five code mutations to the NameParser class. Four of them resulted in a failing NameParserTest, which is exactly what we’d like to see.
But one of them did not: when the condition boundary in line 6, the loop constraint, was changed, NameParserTest still succeeded!
PIT changes loop constraints with a predefined algorithm; in this case, when the loop constraint was i <= names.length, it changed the ‘<=’ into a ‘<‘. Actually this accidentally corrected the bug in NameParser, and of course that didn’t break the unit test. So PIT found an omission in our unit test here, and it turned out that this omission even left a bug undetected! Note that this last point doesn’t always need to be the case. It could be that for the correct behavior of your class, there is some room for some conditions to change. In the case of NameParser for instance, it could have been a requirement that the names list always contained an entry with the last name that was to be found. In that case the behavior for a missing last name would be unspecified and an IndexOutOfBoundsException would have been as good a result as anything else. So PIT can only find strong indications of omissions in your unit tests, but they don’t necessarily have to be. And here are the results for the Person class:

PIT was able to do two mutations in the Person class; one in every getter method. Both times it replaced the return value with null. And as expected, the mutation in the getLastName method went undetected by our unit test.
So PIT found the second omission as well.

Conclusion

In this case, mutation testing would have helped us a lot. But there can still be cases where possible bugs can go unnoticed. In our code for example, there is no test in NameParser test that verifies the behavior when an entry in the names list does not contain both a first name and a last name. But PIT didn’t find this omission.
Still it might make good sense to integrate mutation testing in your build process. PIT can be configured to break your Maven build if too many warnings are found.
And there’s a lot more that can be configured as well, but for that I recommend to bring a visit to the website of PIT at www.pitest.org.

Alternatives

PIT is not the only mutation testing framework for Java, but it is the most popular and the one most actively maintained. Others are µJava and Jumble.
Although most mutation testing frameworks are written for Java, probably because it’s so easy to dynamically change its bytecode, mutation testing is possible in some other languages as well: notable are grunt-mutation-testing for Javascript and Mutator, a commercial framework which is available for a couple of languages.

Qxperts. We empower companies to deliver reliable & high-quality software. Any questions? We are here to help! www.qxperts.io