Introduction

I've been having a blast the last few months learning ApacheAirflow. It's become an indispensable tool in mygetting stuff done toolbox.

Follow me on Twitter @jillianerowe, or send me a message with questions, topic suggestions, ice cream recommendations, or general shenanigans.

• Learn Apache Airflow By Example - Part 1 Introduction - Get familiar with the various moving parts that make up the awesomeness that is Airflow.
• Learn Airflow By Example - Part 2 Install With Docker - Get your dev environment up and running with a simple docker-compose up -d
• Learn Airflow By Example - Part 3 Start Building - Build out a simple DAG, get familiar with the web UI, and learn 3 different ways to trigger your DAGs.
• Learn Airflow By Example - Part 4 DAG Patterns - Learn 3 different DAG patterns that can be used as building blocks for more complex analyses.

Get the Source Code

Alternately, if you just want the codeyou can get the project template by visiting the Dabble of DevOps Store - Docker for Python and Data Science FREE Course.

Bring up the Dev Server

This will take a few minutes to initialize, so if you don't see anything to begin with don't worry!

unzip airflow-template.zip
cd airflow-template
docker-compose up -d
docker-compose logs airflow_webserver

Bash

Copy

Once you see a message in your logs like 'Airflow starting' or something along those lines, visit http://localhost:8089  in your browser and see the magic!

Looking for More?

If you are looking for more information, or a guide that you can save as an internal SOP, please check out my Apache Airflow Database Configuration Guide. I am bundling all my stacks into a completely FREE Resource Library.

To see the Apache Airflow Docker Dev Stack Template and Guide click here. This is a fully functional stack that includes running Apache Airflow with RabbitMQ and PostgreSQL, along with a ready to run configuration file, and a PDF guide that can serve as internal documentation or an SOP.