Get a Fully Configured Apache Airflow Docker Dev Stack with Bitnami
I've been using it for around 2 years now to build out custom workflow interfaces, like those used for Laboratory Information Management Systems (LIMs), Computer Vision pre and postprocessing pipelines, and to set and forget other genomics pipelines.
My favorite feature of Airflow is how completely agnostic it is to the work you are doing or where that work is taking place. It could take place locally, on a Docker image, on Kubernetes, on any number of AWS services, on an HPC system, etc. Using Airflow allows me to concentrate on the business logic of what I'm trying to accomplish without getting too bogged down in implementation details.
During that time I've adopted a set of systems that I use to quickly build out the main development stack with Docker and Docker Compose, using the Bitnami Apache Airflow stack. Generally, I either deploy the stack to production using either the same Docker compose stack if its a small enough instance that is isolated, or with Kubernetes when I...
Manage High Content Screening CellProfiler Pipelines with Apache Airflow
If you are running a High Content Screening Pipeline you probably have a lot of moving pieces. As a non exhaustive list you need to:
- Trigger CellProfiler Analyses, either from a LIMS system, by watching a filesystem, or some other process.
- Keep track of dependencies of CellProfiler Analyses - first run an illumination correction and then your analysis.
- If you have a large dataset and you want to get it analyzed sometime this century you need to split your analysis, run, and then gather the results.
- Once you have results you need to decide on a method of organization. You need to put your data in a database and set up in depth analysis pipelines.
These tasks are much easier to accomplish when you have a system or framework that is built for scientific workflows.
If you prefer to watch I have a video where I go through all the steps in this tutorial.
Enter Apache Airflow
Apache Airflow is :
Airflow is a platform created by the community to programmatically author, schedule and...
Apache Airflow Tutorial – Part 4 DAG Patterns
Overview
During the previous parts in this series, I introduced Apache Airflow in general, demonstrated my docker dev stack, and built out a simple linear DAG definition. I want to wrap up the series by showing a few other common DAG patterns I regularly use.
In order to follow along, get the source code!
Bring up your Airflow Development Environment
This will take a few minutes to get everything initialized, but once its up you will see something like this:
DAG Patterns
I use 3 main DAG patterns. Simple, shown in Part 3, Linear, and Gather. Of course, once you master these patterns, you an combine them to make much more complex pipelines.
Simple DAG Pattern
What I call a simple pattern (and I have no idea if any of these patterns have official names) is a chain of tasks where each task depends upon the previous task. In this...
Apache Airflow Tutorial – Part 3 Start Building
Overview
If you've read this far you should have a reasonable understanding of the Apache Airflow layout and be up and running with your own docker dev environment. Well done! This part in the series will cover building an actual simple pipeline in Airflow.
Start building by getting the source code!
Build a Simple DAG
The simplest DAG is simply having a list of tasks, where each task depends upon its previous task. If you've spun up the airflow instance and taken a look, it looks like this:
Now, if you're asking why I would choose making an ice cream sundae as my DAG, you may need to reevaluate your priorities.
Generally, if you order ice cream, the lovely deliverer of the ice cream will first as you what kind of cone (or cup, you heathen) you want, then your flavor (or flavors!), what toppings, and then will put them all together into sweet, creamy, cold, deliciousness.
You would accomplish this awesomeness with the following Airflow code:
Now,...
Apache Airflow Tutorial – Part 2 Install with Docker
Install Apache Airflow With Docker Overview
In this part of the series I will cover how to get a nice Apache Airflow instance up and running with docker. You won't need to have anything installed locally besides docker, which is fantastic, because configuring all these pieces individually would be kind of awful!
This is the exact same setup and configuration I use for my own Apache Airflow instances. When I run Apache Airflow in production I don't use Postgres in a docker container, as that is not recommended, but this setup is absolutely perfect for dev and will very closely match your production requirements!
Following along with a blog post is great, but the best way to learn is to just jump in and start building. Get the Apache Airflow Docker Dev Stack here.
Celery Job Queue
Getting an instance Apache Airflow up and running looks very similar to a Celery instance. This is because Airflow uses Celery behind the scenes to execute tasks. Read more...
Apache Airflow Tutorial – Part 1 Introduction
What is Apache Airflow?
Briefly, Apache Airflow is a workflow management system (WMS). It groups tasks into analyses, and defines a logical template for when these analyses should be run. Then it gives you all kinds of amazing logging, reporting, and a nice graphical view of your analyses. I'll let you hear it directly from the folks at Apache Airflow
Apache Airflow is a platform to programmatically author, schedule and monitor workflows.
Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.
When workflows are defined as code, they become more maintainable, versionable, testable, and collaborative.
Source - ...
Categories
All CategoriesRecent Posts
Would you like to work together to design and implement your data science and analytics infrastructure? Do you need an AWS Batch or HPC cluster for your High Throughput analysis, or a Data Visualization Application deployed on AWS?
Receive weekly tutorials on only the DevOps you need to know as a Data Scientist.
During these tutorials I will teach you to:
- Build Python Apps with Docker
- Design and Deploy complex analyses with Apache Airflow
- Build computer vision platforms
- Deploy RShiny with Docker and on AWS
- And more!
Apache Airflow Project Template
Build out your Data Science with Apache Airflow.
- Build custom docker images
- Bring up your stack with one command
- Add plugins and custom REST APIs by understanding Airflow best practices
- Deploy to Kubernetes with Helm using a ready made Helm chart.
Deploy Kubernetes on AWS with:
- Terraform - infrastructure as code to give you a repeatable recipe for your Kubernetes deployment
- Detailed documentation
- Get setup up with a configurable cluster that can be deployed for different projects, or environments (dev/stage/prod)
This guide provides you with a birds eye view of which AWS services to use for:
- Exploratory analyses
- Production pipelines
- Data visualization platforms (including RShiny and Dash)
- and Machine Learning Pipelines
These are the services that I use every day to deploy Bioinformatics related infrastructure.
If you're looking for even more in depth guides or project templates check out my store.