Recently I had the opportunity to do a collaborative blog post with Maxime Beauchemin about Apache Airflow and the future of data engineering.
Maxime is a data engineer at Airbnb and creator of their data pipeline framework, Apache Airflow. This framework is used by numerous companies and several of the biggest unicorns — Spotify, Lyft, Airbnb, Stripe, and others to power data engineering at massive scale.
Here's a teaser:
Introduction
Every once in a while I read a post about the future of tech that resonates with clarity.
A few weeks ago it was The Rise of the Data Engineer by Maxime Beauchemin, a data engineer at Airbnb and creator of their data pipeline framework, Apache Airflow. At Astronomer, Apache Airflow is at the very core of our tech stack: our integration workflows are defined by data pipelines built in Apache Airflow as directed acyclic graphs (DAGs). A post like that gives validation as to why right now is the best time for a company like Astronomer to exist.
After reading the post, I reached out to Max about doing an interview post, and to my delight he entertained the request with thoughtful answers to our questions about Apache Airflow and the future of data engineering. You’ll find his answers below, but first I’d like to add a little context.
You might be wondering, “What is data engineering and why does it matter?”
...
You can view the entire post here:
Apache Airflow and the Future of Data Engineering: A Q&A
Discuss this post on Hacker News
Special thanks to Maxime, Laurel Brunk, Chris Hendrixson, and the Astronomer team for their support with this post.
If you enjoyed this post, enter your email below to subscribe to future posts.