Testing the testers: code samples from a TDD framework comparison

26 Nov, 2009
Xebia Background Header Wave

Some while back I was preparing a presentation on mocking and testing frameworks for Java. As part of the aim was to demonstrate some real, running code, I ended up spending quite some time copying, pasting, extending and correcting various examples gleaned from readmes, Javadoc, Wiki pages and blog posts. Since then, this codebase has been extended with various new features I’ve come across, and I’ve often referred to it for experiments, as a helpful reference, and suchlike.
I imagine this kind of "live" reference could also be useful to others, so I thought I’d share it.

Mock Braindump

By way of introduction, here first a braindump of various points of interest related to the frameworks in the sample1. Don’t expect a detailed overview or comparison – there are plenty out there – and for specific questions about the working of the frameworks please refer to their documentation2.
Mockito or EasyMock?
EasyMock and Mockito are the de facto standard mocking frameworks out there at present. Their feature sets are more-or-less identical, and if one of them comes up with something new you can be pretty sure it’ll be in the next version of the other.
Personally, I have to say that I find Mockito’s syntax just that little bit nicer and more intuitive. The fact that you don’t explicitly have to switch to replay mode is handy, and for things like spy (for partial mocks) or InOrder (for ordered expectations) the syntax is simply more elegant.
More fundamentally, the emphasis on stubbing rather then laboriously verifying calls to mocks was, to me, a very useful and justified distinction.
Having said all that, it might well strike you as paradoxical that I still mainly use EasyMock on a day-to-day basis: force of habit and the fact that the projects I’m working on started out with EasyMock sees to that. The advantages I believe Mockito has are not sufficiently large to justify the switch.

Statics, locals and finals: TDD’s final frontier?

"To boldly go…where no mocking framework has gone before" seems to be the mission of both Jmockit and JEasyTest.
public final class ServiceA {
public void doBusinessOperationXyz(EntityX data)
throws InvalidItemStatus {
List<?> items = Database.find("select item from EntityY item where item.someProperty=?",
BigDecimal total = new ServiceB().computeTotal(items);
public static final class ServiceB { …
The challenge: to be able to unit test this class, specifically the find and save calls on Database and the computeTotal call on the new ServiceB instance. Using convential mocking this is nigh-on impossible:

  • find and save are static
  • ServiceB is a final class and thus can’t be mocked
  • even if it could be, the ServiceB instance called is created in the code under test

My immediate reaction? If this is the kind of code you’re supposed to test, you have other problems! Yes, of course it’s an artificial example specifically chosen to highlight cases normal mocking can’t deal with. But even if the real code you’re having trouble with contains only one of these cases, I’d consider trying to refactor the code before looking for a different test framework.
Dependency Injection may have become a bit of a religion, but it is not so widespread for nothing. For code that’s doing DI, the standard mocking frameworks are almost always sufficient.
But even if one perhaps shouldn’t try to unit test the example, could one? Well, experience shows that there are few problems that cannot be solved with a sufficiently large dollop of bytecode manipulation.
JEasyTest works its brand of magic using a pre-test weaving step, which can be done by an
Eclipse plugin or by adding a plugin to your Maven build3. Jmockit uses the slightly more modern instrumentation approach and comes with a Java agent, which means that it only takes an additional VM argument to run in your IDE.
Usability issues aside, I found that I don’t feel happy with the code to prepare a test fixture and register expectations in either framework; it’s downright clumsy in some cases. Here’s the JEasyTest test:
public void testBusinessOperation() throws InvalidItemStatus {
arg("select item from EntityY item where item.someProperty=?"),
assertEquals(total, entity.getTotal());
expectStaticNonVoidMethod? ARGH! Feels more like ASM than unit testing.
Jmockit’s "expectations" mode comes closest to what Mockito/EasyMock users are likely to be familiar with4:
private final Database unused = null;
private ServiceB serviceB;
public void doBusinessOperationXyz() throws Exception {
EntityX data = new EntityX();
BigDecimal total = new BigDecimal("125.40");
List<?> items = new ArrayList<Object>();
Database.find(withSubstring("select"), withAny(""));
new ServiceB().computeTotal(items);
new ServiceA().doBusinessOperationXyz(data);
assertEquals(total, data.getTotal());
Conceptually, this is reminiscent of an EasyMock test, with record and replay phases. The tt>@MockField</tt fields stand in for the creation of actual mock objects: the field declarations only indicate to Jmockit that mock of the given types are required when the test is run, cluttering the test class with unused properties.
In addition, the "mock management" methods (withAny, returns etc.) are not static, meaning they are not visually identified by e.g. being displayed in italics. I was surprised how much this seemingly minor discrepancy alienated me – it just doesn’t look quite like a unit test.


From what I can see not much is happening around jMock anymore: the last release was in Aug 2008 and the last news posting over half a year ago. The syntax, which tries to mimic a pseudo-"natural language" DSL, is just a bit too cumbersome. jMock’s support for multithreading prompted me to take a closer look, but it’s actually simply a mechanism for ensuring that assertion errors thrown in other threads are actually registered by the test thread; there is no support for testing concurrent behaviour.

Testing concurrent code

I quite like MultithreadedTC, a small framework5 which aims to make it easy to start and coordinate multiple test threads. It does this by means of a global "clock" that moves forward whenever all threads are blocked – either "naturally" (e.g. during a call such as blockingQueue.take()
) or deliberately using a waitForTick(n) command.
As such, MultithreadedTC doesn’t offer much more than can be achieved by "manual" latches as described in Iwein Fuld’s recent blog post, but the clock metaphor does seem to make the test flow easier to understand, especially for longer tests.
Like latching, though, the main problem with MultithreadedTC is that you can’t easily control the execution of code in the classes under test.
public void thread1() throws InterruptedException {


public void thread2() throws InterruptedException {


This code will go some way to ensuring that service.someMethod() and service.otherMethod() start at almost the same time, and will guarantee that neither thread will continue until both methods have completed. But what if you want to ensure that half of someMethod completes before otherMethod is called?
For that, you’ll have to be able to get access to the implementations of someMethod and otherMethod, for instance by subclassing the service implementations, or using something like Byteman.
Ultimately, though, I think unit tests are just not the right way of going about testing concurrent code. "Choreographing" the carefully-chosen actions of a small number of test threads is a poor substitute for real concurrent usage, and the bugs you’ll find, if any, aren’t the kind of concurrency issues that end up causing nightmares.
For proper concurrency testing, there doesn’t so far seem to be a good substitute for starting a whole bunch of threads – on as many cores as possible – and running them for a good while (see, for instance, the integration tests of Multiverse, the Java STM). If it’s possible to inject a certain amount of randomness into the timing (using e.g. Byteman), all the better!

JUnit Rocks Rules!

Version 4.7 of JUnit introduced rules, sort-of around aspects that are called before and after the execution of a test. Some of the standard examples demonstrate "housekeeping" functionality such as opening and closing a resource or creating and cleaning up a temporary folder. Rules can also affect the result of a test, though, e.g. causing it to fail even if all the test’s assertions were successful.
Whilst one can see how the concept of rules can be useful, they still have a bit of "v1" roughness about them. The "housekeeping" rules are essentially convenient replacements for tt>@Before</tt/tt>@After</tt logic, and the syntax of the "test-influencing" rules feels messy:
public class UsesErrorCollectorTwice {
public ErrorCollector collector = new ErrorCollector();
public void example() {
collector.addError(new Throwable("first thing went wrong"));
collector.addError(new Throwable("second thing went wrong"));
collector.checkThat("ERROR", not("ERROR"));
collector.checkThat("OK", not("ERROR"));
System.out.println("Got here!");
Wouldn’t that be nicer if the assertions were a little more, um, assert-like? Or take:
public class HasExpectedException {
public ExpectedException thrown = ExpectedException.none();
public void throwsNullPointerExceptionWithMessage() {
throw new NullPointerException("What happened?");
I mean, it’s certainly useful to be able to make more detailed assertions about exceptions, but could this not be integrated into either of the current exception-checking patterns6?
The potential power of rules also raises an question: is it wise to get into the habit of doing full-scale resource management (e.g. starting servers or DB connections) in a unit test?


  1. Sample source code here. Check out the project using svn checkout target-folder.
  2. See the sample code’s POM.
  3. Which is not to say it’s the best approach, of course!
  4. The original code is not longer being actively developed, but there has been some recent work aimed at better JUnit 4 integration.
  5. @Test
    public void tryCatchTestForException() {
      try {
        throw new NullPointerException();
      } catch (NullPointerException exception) {
        // expected
    @Test(expected = NullPointerException.class)
    public void annotationTestForException() {
      throw new NullPointerException();

Get in touch with us to learn more about the subject and related solutions

Explore related posts