Integration testing CloudBees' RUN@cloud

As part of the preparations for the recent 1.0.0 release of jclouds, I was tidying up the existing Tweetstore demo application as well as porting it to CloudBees' Tomcat-based RUN@cloud platform. A key part of the test harness for the original versions of Tweetstore that run on the Google App Engine is the neat GoogleDevServer class. Basically, it's a clever wrapper around the KickStart class used under the covers by the GAE SDK tools that allows you to specify the SDK location, address, port and WAR file (or expanded WAR directory) to run. Better still, it can programmatically be shut down cleanly, making it ideal for integration test runs¹. For CloudBees, I was thus looking to put together a similar RunAtCloudServer. It proved more challenging than expected...²

Just spawn a container already?

Both the maven-gae-plugin and the bees-maven-plugin allow you to spawn a local GAE or RUN@cloud server simulation, of course. These can easily be integrated into your Maven build, for instance as part of your pre-integration-test setup. So why not just stick with that? Well, for some scenarios that can be perfectly sufficient. For Tweetstore, having an inline server that cleanly shuts down allows us to:

• run integration tests alongside unit tests from an IDE without having to remember to start an external process
• debug the application running in the server on demand, without having to mess with remote debuggers or figure out a way to sometimes get the Maven plugins to start in debug mode
• work some conditional magic with the server arguments

If your project's requirements are similar, an inline server might be a good option. And luckily, the code's already been written!

Hangin' around

The RUN@cloud SDK's version of KickStart is the StaxSdkAppServer³. Figuring out the correct arguments for the launchServer factory method wasn't too hard, so I hooked up a StaxSdkAppServer and kicked off the test. Started like a charm! Then the test finished and...just hung on in there. Over in the trusty debugger⁴, I could see not one, but two zombie threads still running. The design was based on a "shutdown-by-JVM-kill" scenario, I guess.

Assassins and other Ninja Tricks

The first of the two threads is a Timer that something called the WebAppEngine uses to do...well, who knows? In any case, we can get a reference to it with a decent handful of reflection and cancel it. Easy enough. Our remaining zombie is a bit more elusive, unfortunately. It's a thread spawned by an instance of the RequestMonitorValve class in a manner analogous to [java] new Thread(idleTimer, "requestMonitor").start(); [/java] So no reference, and no way of interrupting it. However, the idleTimer Runnable is referenced and repeatedly calls a handler as part of its execution. How about injecting a handler that throws an exception to end the thread? Nice idea, but it turns out the thread really doesn't want to die that way: [java] while(true) { try { // do stuff if(callbackClient != null) callbackClient.updateStatus(state); } catch(Exception e) { // increment some error counters } try { Thread.sleep(statusIntervalSecs * 1000); } catch(InterruptedException interruptedexception) { } } [/java] So I've had to go one better and throw a ThreadDeath error instead. Urgh.

Now you see it, now you don't?

The remaining gotcha involves injecting the catchily-named KillerCallback into the idleTimer. Even using reflection, this is a elementary concurrency failure because without a synchronized-with relationship⁵ between injecting the new handler and invocation of the handler by idleTimer, there is no guarantee the new handler will actually be visible to the thread we're trying to stop. Such a relationship could be established if the "assassin" was injected before the thread that runs idleTimer starts. Unless we want to override library classes, that would mean setting it after creating but before starting the StaxSdkAppServer⁶...which unfortunately doesn't work either because the instances referencing idleTimer are only created when the server starts.

So?

So, for now the potential concurrency issue is documented and we hope for the best ;-) No problems observered so far...