Monitoring Akka with Kamon

Kamon is a framework for monitoring the health and performance of applications based on akka, the popular actor system framework often used with Scala. It provides good quick indicators, but also allows in-depth analysis.

Tracing

Beyond just collecting local metrics per actor (e.g. message processing times and mailbox size), Kamon is unique in that it also monitors message flow between actors.
Essentially, Kamon introduces a TraceContext that is maintained across asynchronous calls: it uses AOP to pass the context along with messages. None of your own code needs to change.
Because of convenient integration modules for Spray/Play, a TraceContext can be automatically started when an HTTP request comes in.
If nothing else, this can be easily combined with the Logback converter shipped with Kamon: simply logging the token is of great use right out of the gate.

Dashboarding

Kamon does not come with a dashboard by itself (though some work in this direction is underway).
Instead, it provides 3 ‘backends’ to post the data to (4 if you count the ‘LogReporter‘ backend that just dumps some statistics into Slf4j): 2 on-line services (NewRelic and DataDog), and statsd (from Etsy).
statsd might seem like a hassle to set up, as it needs additional components such as grafana/graphite to actually browse the statistics. Kamon fortunately provides a correctly set-up docker container to get you up and running quickly. We unfortunately ran into some issues with the image uploaded to the Docker Hub Registry, but building it ourselves from the definition on github resolved most of these.

Implementation

We found the source code to Kamon to be clear and to-the-point. While we’re generally no great fan of AspectJ, for this purpose the technique seems to be quite well-suited.
‘Monkey-patching’ a core part of your stack like this can of course be dangerous, especially with respect to performance considerations. Unless you enable the heavier analyses (which are off by default and clearly marked), it seems this could be fairly light – but of course only real tests will tell.

Getting Started

Most Kamon modules are enabled by adding their respective akka extension. We found the quickest way to get started is to:

• Add the Kamon dependencies to your project as described in the official getting started guide
• Enable the Metrics and LogReporter extensions in your akka configuration
• Start your application with AspectJ run-time weaving enabled. How to do this depends on how you start your application. We used the sbt-aspectj plugin.

Enabling AspectJ weaving can require a little bit of twiddling, but adding the LogReporter should give you quick feedback on whether you were successful: it should start periodically logging metrics information.
Next steps are:

• Enabling Spray or Play plugins
• Adding the trace token to your logging
• Enabling other backends (e.g. statsd)
• Adding custom application-specific metrics and trace points

Conclusion

Kamon looks like a healthy, useful tool that not only has great potential, but also provides some great quick wins.
The documentation that is available is of great quality, but there are some parts of the system that are not so well-covered. Luckily, the source code very approachable.
It is clear the Kamon project is not very popular yet, judging by some of the rough edges we encountered. These, however, seem to be mostly superficial: the core ideas and implementation seems solid. We highly recommend taking a look.

Remco Beckers
Arnout Engelen