Deploy a Celery Job Queue With Docker – Part 2 Deploy with Docker Swarm on AWS
Overview
In Part 1 of this series we went over the Celery Architecture, how to separate out the components in a docker-compose file, and laid the ground for deployment.
Deploy With AWS CloudFormation
This portion of the blog post assumes you have a ssh key setup. If you don't go to the AWS docs here.
What is CloudFormation?
AWS CloudFormation is an infrastructure design tool that allows users to design their infrastructure by defining file systems, compute requirements, networking, etc. If you have no interest in designing infrastructure, y0u probably don't need to worry. Cloudformation configurations are shareable through templates.
Docker AWS CloudFormation
Getting Started
Docker has come to our rescue here, with a Docker for AWS CloudFormation template. This will, with the click of a few buttons, deploy a docker swarm on AWS for us!!
Click on the page, and scroll down to quick start. Under 'Stable Channel' select '...
Deploy a Celery Job Queue With Docker – Part 1 Develop
Overview
In this post I will hopefully show you how to organize a large docker-compose project, specifically a project related to a job queue. In this instance we will use Celery, but hopefully you can see how the concepts relate to any project with a job queue, or just a large number of moving pieces.
This post will be in two parts. The first will give a very brief overview of celery, the architecture of a celery job queue, and how to setup a celery task, worker, and celery flower interface with docker and docker-compose. Part 2 will go over deployment using docker-swarm.
What is Celery?
Celery is a distributed job queuing system that allows us queue up oodles of tasks, and execute them as we have resources.
From celeryproject.org -
Celery is an asynchronous task queue/job queue based on distributed message passing.It is focused on real-time operation, but supports scheduling as well.The execution units, called tasks, are executed...
Categories
All CategoriesRecent Posts
Would you like to work together to design and implement your data science and analytics infrastructure? Do you need an AWS Batch or HPC cluster for your High Throughput analysis, or a Data Visualization Application deployed on AWS?
Receive weekly tutorials on only the DevOps you need to know as a Data Scientist.
During these tutorials I will teach you to:
- Build Python Apps with Docker
- Design and Deploy complex analyses with Apache Airflow
- Build computer vision platforms
- Deploy RShiny with Docker and on AWS
- And more!
Apache Airflow Project Template
Build out your Data Science with Apache Airflow.
- Build custom docker images
- Bring up your stack with one command
- Add plugins and custom REST APIs by understanding Airflow best practices
- Deploy to Kubernetes with Helm using a ready made Helm chart.
Deploy Kubernetes on AWS with:
- Terraform - infrastructure as code to give you a repeatable recipe for your Kubernetes deployment
- Detailed documentation
- Get setup up with a configurable cluster that can be deployed for different projects, or environments (dev/stage/prod)
This guide provides you with a birds eye view of which AWS services to use for:
- Exploratory analyses
- Production pipelines
- Data visualization platforms (including RShiny and Dash)
- and Machine Learning Pipelines
These are the services that I use every day to deploy Bioinformatics related infrastructure.
If you're looking for even more in depth guides or project templates check out my store.