I've been having a blast the last few months learning ApacheAirflow. It's become an indispensable tool in my getting stuff done toolbox.
Follow me on Twitter @jillianerowe, or send me a message with questions, topic suggestions, ice cream recommendations, or general shenanigans.
Alternately, if you just want the code you can get the project template by visiting the Dabble of DevOps Store - Docker for Python and Data Science FREE Course.
This will take a few minutes to initialize, so if you don't see anything to begin with don't worry!
unzip airflow-template.zip cd airflow-template docker-compose up -d docker-compose logs airflow_webserver
Once you see a message in your logs like 'Airflow starting' or something along those lines, visit http://localhost:8089 in your browser and see the magic!
If you are looking for more information, or a guide that you can save as an internal SOP, please check out my Apache Airflow Database Configuration Guide. I am bundling all my stacks into a completely FREE Resource Library.
To see the Apache Airflow Docker Dev Stack Template and Guide click here. This is a fully functional stack that includes running Apache Airflow with RabbitMQ and PostgreSQL, along with a ready to run configuration file, and a PDF guide that can serve as internal documentation or an SOP.
Subscribe to the newsletter! You'll get a weekly tutorial on all the DevOps you need to know as a Data Scientist. Build Python Apps with Docker, Design and Deploy complex analyses with Apache Airflow, build computer vision platforms, and more.