Conditionally Running Tests in TestNG

In this post, my colleague Barend showed how one can conditionally ignore certain tests in JUnit. In this post we will take a look at how this can be solved in TestNG, another popular testing framework.
Unlike JUnit, TestNG does not have the Assume class. It does however provide an Exception type called SkipException, which will notify the TestNG framework that a test method should be regarded as skipped.
Using this exception type it is relatively simple to write your own Assume class:
[code lang=”java”]
public class Assumes {
public static <T> void assumeThat(T actual, Matcher<? super T> matcher) {
assumeThat("", actual, matcher);
}
public static <T> void assumeThat(String reason, T actual, Matcher<? super T> matcher) {
if (!matcher.matches(actual)) {
Description description = new StringDescription();
description.appendText(reason)
.appendText("\nExpected: ")
.appendDescriptionOf(matcher)
.appendText("\n but: ");
matcher.describeMismatch(actual, description);
throw new SkipException(description.toString());
}
}
public static void assumeThat(String reason, boolean assertion) {
if (!assertion) {
throw new SkipException(reason);
}
}
}
[/code]
If the Hamcrest Matcher fails, the SkipException will be thrown and TestNG will skip the annotated test method.
Another way in TestNG to accomplish this is more in line with what Barend did with the TestRunner. In TestNG you can easily add and/or modify behavior of the framework using TestListeners. In order to verify assumptions in a listener, we can annotate the test methods with a new annotation:
[code lang=”java”]
@Retention(RUNTIME)
public @interface Assumption {
String[] methods();
}
[/code]
The annotation contains a reference to methods that return a boolean indicating whether or not the assumption holds. For instance when we want to run a test only when the outside temperature is high enough:
[code lang=”java”]
@Test
@Assumption(methods = "isNiceOutside")
public void shouldNotBeRunTooCold() {
….
}
public boolean isNiceOutside() {
return sensor.temperature() > 25;
}
[/code]
In order for this annotation to be picked up by TestNG, we need to annotate the class with a Listener. One of the Listener types TestNG provides is the InvokedMethodListener. This Listener will be informed when TestNG is about to run a method, and when the method is finished. So we can hook in, check whether the method is annotated, and verify the assumptions, skipping over the method if needed. Let’s implement:
[code lang=”java”]
public class AssumptionListener implements IInvokedMethodListener {
@Override
public void beforeInvocation(IInvokedMethod invokedMethod, ITestResult result) {
ITestNGMethod testNgMethod = result.getMethod();
ConstructorOrMethod contructorOrMethod = testNgMethod.getConstructorOrMethod();
Method method = contructorOrMethod.getMethod();
if (method == null || !method.isAnnotationPresent(Assumption.class)) {
return;
}
List<String> failedAssumptions = checkAssumptions(method, result);
if (!failedAssumptions.isEmpty()) {
throw new SkipException(format("Skipping [%s] because the %s assumption(s) do not hold.", contructorOrMethod.getName(), failedAssumptions));
}
}
@Override
public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
}
private List<String> checkAssumptions(Method method, ITestResult result) {
Assumption annotation = method.getAnnotation(Assumption.class);
String[] assumptionMethods = annotation.methods();
List<String> failedAssumptions = new ArrayList<String>();
Class clazz = result.getMethod().getTestClass().getRealClass();
for (String assumptionMethod : assumptionMethods) {
boolean assume = checkAssumption(result, clazz, assumptionMethod);
if (!assume) {
failedAssumptions.add(assumptionMethod);
}
}
return failedAssumptions;
}
private boolean checkAssumption(ITestResult result, Class clazz, String assumptionMethod) {
try {
Method assumption = clazz.getMethod(assumptionMethod);
if (assumption.getReturnType() != boolean.class) {
throw new RuntimeException(format("Assumption method [%s] should return a boolean", assumptionMethod));
}
return (Boolean) assumption.invoke(result.getInstance());
} catch (Exception e) {
…
}
}
}
[/code]
So if we now skip back to our example, we can actually finish it in the following way:
[code lang=”java”]
@Listeners(value = AssumptionListener.class)
public void WeatherTest {
@Test
@Assumption(methods = "isNiceOutside")
public void shouldNotBeRunTooCold() {
….
}
public boolean isNiceOutside() {
return sensor.temperature() > 25;
}
}
[/code]
Both of these assumptions have been implemented in my pet project on Github called AssumeNG. The jar can also be found in the central Maven repository for easy inclusion in your project (nl.javadude.assumeng:assumeng:1.2.2).