Building and Deploying Your First Cloudflow Application

Cloudflow is a relatively new framework that helps you build distributed streaming applications and deploy them on Kubernetes. Its powerful abstractions allow you to easily split your application into independent stream processing components, called streamlets. Streamlets can be developed using several runtimes such as Akka Streams and Flink. A streamlet can have one or more input streams, inlets, and one or more output streams, outlets. You deploy you application as a whole while Cloudflow deploys streamlets individually. Cloudflow ensures data flows between the inlets and outlets of your streamlets at runtime, through Kafka, corresponding to your Avro schemas and your pipeline definition, called a blueprint.

In this post we are going to build a simple application using the Cloudflow framework and Akka Streams. We are going to start off with a simple SBT project to setup a stream processing pipeline, run it locally and eventually deploy it to GKE using [more…]