Deploy a Celery Job Queue With Docker – Part 2 Deploy with Docker Swarm on AWS
Overview
In Part 1 of this series we went over the Celery Architecture, how to separate out the components in a docker-compose file, and laid the ground for deployment.
Deploy With AWS CloudFormation
This portion of the blog post assumes you have a ssh key setup. If you don't go to the AWS docs here.
What is CloudFormation?
AWS CloudFormation is an infrastructure design tool that allows users to design their infrastructure by defining file systems, compute requirements, networking, etc. If you have no interest in designing infrastructure, y0u probably don't need to worry. Cloudformation configurations are shareable through templates.
Docker AWS CloudFormation
Getting Started
Docker has come to our rescue here, with a Docker for AWS CloudFormation template. This will, with the click of a few buttons, deploy a docker swarm on AWS for us!!
Click on the page, and scroll down to quick start. Under 'Stable Channel' select '...
Deploy a Celery Job Queue With Docker – Part 1 Develop
Overview
In this post I will hopefully show you how to organize a large docker-compose project, specifically a project related to a job queue. In this instance we will use Celery, but hopefully you can see how the concepts relate to any project with a job queue, or just a large number of moving pieces.
This post will be in two parts. The first will give a very brief overview of celery, the architecture of a celery job queue, and how to setup a celery task, worker, and celery flower interface with docker and docker-compose. Part 2 will go over deployment using docker-swarm.
What is Celery?
Celery is a distributed job queuing system that allows us queue up oodles of tasks, and execute them as we have resources.
From celeryproject.org -
Celery is an asynchronous task queue/job queue based on distributed message passing.It is focused on real-time operation, but supports scheduling as well.The execution units, called tasks, are executed...
