Today is a great day for Apache Airflow as it graduates from incubating status to a Top-Level Apache project. This is the next step of maturity for Airflow. For those unfamiliar, Airflow is an orchestration tool to schedule and orchestrate your data workflows. From ETL to training of models, or any other arbitrary tasks. Unlike other orchestrators, everything is written in Python, which makes it easy to use for both engineers and scientists. Having everything in code means that it is easy to version and maintain.
At the point of writing, Apache Airflow is in use at more than 200 organizations, including Adobe, Airbnb, Astronomer, Etsy, Google, ING, Lyft, Paypal, Qubole, Reddit, Reply, Square, Twitter, and United Airlines, and many more.
Within the cloud space, Airflow is gaining ground as well. Most notably, Airflow is being offered on the Google Cloud Platform as a managed service under the name of Cloud Composer. Furthermore, recent efforts by Amazon have resulted into seamless integration of Sagemaker within Airflow. Besides the cloud providers, also companies like Astronomer offer managed Airflow. Having this ecosystem of companies around Airflow make it a product that generalizes very well to many companies, and makes sure that there is a shared effort to continue development of Airflow.
If you’re interested in Airflow, join us at the meetup the 23rd of January at the Google HQ in London. My dear colleague Bas Beelen will give a talk on how to make testing with Airflow fun again.
For me personally, as a committer on the project, I find it truly amazing to see people from all over the world contributing code and documentation to Airflow to make it even better. I’m looking forward to what 2019 will bring to Airflow!
Interested in Apache Airflow Training?
A quick heads up that we also offer Apache Airflow Training as one of the public courses in our Academy. The next scheduled workshop is on February 4-5 2019 and you can register right here.
Can’t make that one? Not to worry, you can check out our schedule to see the other times we offer this course throughout the year, or reach out below to inquire about booking in a private, in-company training session.