Cloudflow is a relatively new framework that helps you build distributed streaming applications and deploy them on Kubernetes. Its powerful abstractions allow you to easily split your application into independent stream processing components, called streamlets. Streamlets can be developed using several runtimes such as Akka Streams and Flink. A streamlet can have one or more input streams, inlets, and one or more output streams, outlets. You deploy you application as a whole while Cloudflow deploys streamlets individually. Cloudflow ensures data flows between the inlets and outlets of your streamlets at runtime, through Kafka, corresponding to your Avro schemas and your pipeline definition, called a blueprint.

In this post we are going to build a simple application using the Cloudflow framework and Akka Streams. We are going to start off with a simple SBT project to setup a stream processing pipeline, run it locally and eventually deploy it to GKE using [more…]

I’ve been doing Scala projects with Akka Streams for quite a few years now and I have a reasonably good feeling of things to watch out for. At my current project we are doing Java and we are using a different implementation of the Reactive Streams Specification: Reactor. While learning the library I stumbled upon many common mistakes and bad practices which I’ll be listing here.

Reactive Streams

Firstly, let’s have a look at the Reactive Streams Specification and see how Reactor maps to that.

[more…]