Deploy a Celery Job Queue With Docker – Part 1 Develop

Overview

In this post I will hopefully show you how to organize a large docker-compose project, specifically a project related to a job queue. In this instance we will use Celery, but hopefully you can see how the concepts relate to any project with a job queue, or just a large number of moving pieces.

This post will be in two parts. The first will give a very brief overview of celery, the architecture of a celery job queue, and how to setup a celery task, worker, and celery flower interface with docker and docker-compose. Part 2 will go over deployment using docker-swarm.

 

What is Celery?

Celery is a distributed job queuing system that allows us queue up oodles of tasks, and execute them as we have resources. 

From celeryproject.org - 

Celery is an asynchronous task queue/job queue based on distributed message passing.It is focused on real-time operation, but supports scheduling as well.
The execution units, called tasks, are executed...
Continue Reading...

Deploy a Full Stack Web Application with Docker and Traefik

docker python traefik Feb 05, 2019

Overview

Have you ever had to deploy a full stack web application? With different services, some mapping to different ports, some static html, maybe a couple databases thrown in there? Have you ever banged your head against your desk trying to figure out how to expose different ports on a remote server, or through a cloud platform?

If you have, you understand

I'm going to go over how I deploy such an application, with nice path names instead of annoying port numbers, using traefik and docker. I'm not going to go into the specific code so much, but I may go crazy and do that at a later time. This setup makes it much easier to deploy your application to cloud platforms that don't expose ports besides 80 by default, because you don't need any other ports!

Get the Code

Like I said, I'm not going to go so much into the code itself, this post is all about Traefik, but it's all here on github. 

The final website, deployed to AWS, is here.

...

Continue Reading...

Docker Ecosystem

Uncategorized Feb 01, 2019

Containers, Compose, Swarm, Desktop?

Docker isn't simply a container or a virtual machine, its an entire ecosystem. Luckily, each component is a layer that builds upon its previous layer. There are definitely a lot of moving pieces, but from a birds eyes perspective there are three main components.

This post is only meant to be a very brief perspective. For more in-depth information check out the official getting starting docs.

Docker - A single image

A docker image is very much like a virtual machine. If you haven't worked with virtual machines before, it's basically your computer, without a desktop. You can't run around clicking on start or finder icons, but you can install packages, start web servers, create users, run applications, and spin up databases all without installing any software to your local machine. Besides docker, of course!

Examples

  •  Web Applications - python flask, node.js express, static html sites, etc.
  •  Machine Learning...
Continue Reading...

Develop a Python Flask Application with Docker and Deploy it to AWS – Part 4 Deploy our docker images

Uncategorized Jan 18, 2019

VICTORY

Now that our docker-compose.yml file is all ready, we have almost all the pieces we need to deploy our application to AWS. The last thing it so deploy our docker images to some docker storage provider.

I personally like quay.io, but docker hub is an equally excellent choice that is rapidly improving.

Get the Code

If you haven't already, clone the github repo to see the full code.

Upload your docker containers

AWS can't work with containers just hanging out on OS, so you need to upload (or push) them to the cloud. 

Create a Docker Account

Create an account on either quay.io or docker hub. It really doesn't matter where. Just pick one and go for it. Quay.io has a nice tutorial to get started pushing (uploading) and pulling (downloading) your docker images.

If you're like me, and you're really terrible at remembering things, you will want to save your password somewhere. Probably in a file. Possibly even in a file that is backed up...

Continue Reading...

Develop a Python Flask Application with Docker and Deploy it to AWS – Part 3 Bring it together with docker compose

Uncategorized Jan 18, 2019

Recap

Up until this point we covered:

  •  Building a custom docker container using the official python miniconda container as our base.
  •  Interactively running commands in our custom image
  •  Giving docker build time instructions and run time instructions with RUN and CMD.
  • Spinning up a webserver that we can access on our own computers by exposing ports
  •  Mounting volumes and directories in our container for real time development.

Now we will use docker-compose to do all these things, but do them even better and with style.

Get the Code

Clone the github repo to see the full code.

Turbo Charge your Docker Dev with Docker-Compose

What is Docker Compose?

Now, I'm going to discuss an incredibly powerful method of connecting and building our docker containers using docker-compose. Docker compose is simply a layer on top of docker that allows us to compose our containers functionality into ...

Continue Reading...

Develop a Python Flask Application with Docker and Deploy to AWS – Part 1 Docker Beginnings

aws docker flask python Jan 11, 2019

Series  - Deploy a python flask app to AWS

I am currently taking the Udacity Machine Learning Engineer Nanodegree, and as a part of my project I wanted to deploy my dockerized machine learning model, along with a (very) simple front end interface, to AWS. I needed to review some materials, and realized I was having a lot of trouble finding a single resource that went from A-Z in all the various moving pieces. In this series I will go over:

  • Build a single docker container using Miniconda
  • Create a small python flask application
  • Serve our flask application with docker-compose
  • Deploy our application to AWS

The Sweet Promise of Victory

Have you ever had an idea completely crash and burn because you couldn't move it from your computer to the remote server, or the cloud, or because you updated your operating system and broke absolutely everything?

Me too! (I break operating systems frequently. It's a problem.) 

That is why I started developing, from start...

Continue Reading...

Develop a Python Flask Application with Docker and Deploy to AWS – Part 2 Serve a Flask App with Docker

aws flask python Jan 11, 2019

Overview

In this portion of the series we will be:

  • Creating a simple flask app with a few RESTful methods
  • Testing out our RESTful methods using curl.
  • Future proofing our development environment for easy and stress free deployment.

Get the Code

Clone the github repo to see the full code.

Create a flask app

Flask is a python framework for creating web applications. It has all kinds of functionality, but here we will be using just a small portion to create a few RESTful web routes.

A Note about Directory Structures

I tend to sit down with a paper and colored pens and figure out how I want my application organized. Generally, each service is in its own directory, which is then in its own docker container. A service is a single application, such as our flask app, a database, a job queue, etc. For this particular example we only have one service, the flask app, but hopefully you can see how important it is to organize your applications.

...
Continue Reading...
1 2 3 4
Close

50% Complete

Two Step

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.