Deploy Bioinformatics Modules on HPC

Uncategorized Sep 22, 2019

Deploying scientific software in an HPC environment can be challenging. Deploying bioinformatics software anyhow anywhere, or say a High Performance Computing Cluster (HPC), can be especially challenging!

Life, at least in this regard, has become so much better in the last few years. Anaconda, the scientific python distro along with Conda, package manager and builder awesomeness made deploying software so much more streamlined. There are amazing groups contributing packages to conda. It's become a whole ecosystem of people working on infrastructure, software and packaging. Bioconda and Conda-Forge are two great groups that have added a ton of value to communities that use scientific software. EasyBuild gives you some pretty great capabilities for deploying your software as modules, which makes them available to everybody!

Disclaimer - I am a core team member of Bioconda, but I'm kind of a slacker member and they are awesome all on...

Continue Reading...

Kubernetes on AWS – Getting Started with EKS

aws docker kubernetes Jul 14, 2019

AWS Elastic Kubernetes Service (EKS) is a fully managed service AWS launched recently. Elastic services in AWS it means that the number of instances actually in use scales up or down based on the demand. This is first of all seriously cool, and second of all can cut down on costs. Fewer requests? Fewer nodes!

I'm just getting started with Kubernetes myself, and going through this walkthrough was a great learning exercise. 

I love deploying applications with docker swarm because it's fairly simple and I already know it, however, Swarm for AWS has some downsides. Firstly, it is not elastic, secondly, in order to get sticky sessions you need to add an additional service such as Traefik. With session affinity you can deploy RShiny and Python Dash applications with no other functionality besides the built in, and that's amazing!

I also personally think the moving towards Kubernetes over Swarm. It even comes installed on my mac version of Docker. Now is a great time to get...

Continue Reading...

Launch Your First AWS EC2 Instance

aws Jul 05, 2019

I'm loving creating videos, and so here is a 3 part series on getting started with AWS and EC2 Instances. Once you understand launching an EC2 instance absolutely every other part of AWS is going to make so much more sense. Behind the scenes we are all always just spinning up servers, installing all the things, and getting our stuff done. As always AWS has excellent documentation and I really encourage you to check it out!

Learning about deployment is an excellent strategy for so many reasons. Probably number one there is not having to wait around for people like me to have their acts together. 

Not only that, there are just so many cool things that are possible with all the neat new deployment strategies and distributed computing libraries out there. It used to be that software engineers would need to put in a HUGE amount of time and effort in order to really optimize for speed.

These days the barrier to getting your application running fast is just so...

Continue Reading...

Develop and Deploy Python Applications with Docker

docker python Jun 28, 2019

Dockerize all the Things!

If you've read anything on this blog you will know I am all about docker and I am all about world domination with distributed computing, particularly with Python Applications.

This week marks something new for me. I've written plenty of documentation, oodles of Power Point presentations with accompanying in-person training. This is the first for me to start recording myself for video, and it was an exciting start! It feels a bit silly, because there are probably millions of people on YouTube or other video formats by now, but it was a first for me. 

Check it out over here on the NEW Dabble of DevOps Youtube channel!


More Python + Docker Resources

If you want to see a deeper dive on building python applications in Docker containers check out my Develop a Python Application in Docker and Deploy to AWS series.

Please watch, comment, share, and let me know what you think!

Happy teching!

Continue Reading...

3 Easy Ways to Deploy your Docker Web App on AWS

Uncategorized Jun 21, 2019

AWS is shiny

AWS is so, so shiny and awesome, but when I first got started I felt overwhelmed by the sheer amount of stuff. There is so much flexibility, and cli tools, and ways to do anything. Initially, all I wanted to be able to do the things I'd always been able to do, like spin up a server and deploy a docker web application. 

Today I am going to share 3 relatively easy methods to deploy your Docker projects on AWS. None of these solutions requires you to have much more AWS specific knowledge than knowing how to login. Having a bit of knowledge of security groups and virtual private clouds (VPC) will be helpful, but never fear! You can deploy your apps!

AWS EC2 Instance

Using the AWS Elastic Compute Cloud (get it? 2 Cs!) is probably the most straightforward and comfortable way for someone, including someone who knows nothing about the AWS ecosystem, to get started, but this is most certainly not a shiny solution.

Elastic Compute Cloud

The AWS...

Continue Reading...

Configure Nginx to Serve a Flask App

Uncategorized Jun 15, 2019

Deploying your web application with NGINX

I was so stoked the first time I'd managed to build a web application. It was solving an actual problem real people were facing. It was love at first site between me and infinite scroll data tables.  

Then I had to deploy my application, and several other applications on the same server, and I was less than pleased!

Writing all the code was such a big hurtle for me, and then I had to figure out the deployment feature too. Luckily, I worked with a great IT department and they were able to point me towards some great resources. These days I do all my development in docker and keep as a library of templates for deploying my projects.

Flask + NGINX

These days I mostly use Traefik to do my load balancing and proxy passes, but its important to understand the fundamentals before you go throwing around all kinds of mostly optional prettiness. 

In this example I am deploying a Python Flask application...

Continue Reading...

Docker Stacks with Databases

Uncategorized May 31, 2019

Connecting your Python App to a Database

All without installing a thing! Besides docker, of course! When I was first starting getting with docker I had SO MUCH trouble figuring out the connection strings. Once I got it sorted I wrote myself up a series of templates I could refer to. 

There's no real purpose to this post. I just wanted to have all the related templates in one place.

Python + MySQL

Spin up a MySQL and connect to it with python! Read the blog post.

Python + Postgres

If you want performance go for PostgresSQL and python. Read the blog post.

Python + MongoDB

I love MongoDB! It feels like CODE. Read the blog post. 

Python + Redis

If you’re looking for a fantastically fast datastore than simply fire up a Redis datastore ! You can check out the entire blog post here.

Continue Reading...

Setup a Redis + Python Docker Dev Stack

Uncategorized May 24, 2019

Query a Redis DataStore from a Python Application

Redis is one of my favorite technologies to use when I'm building web apps. It's just super. If you haven't used it its sort of like a database, without the structure, and its blazing fast because its in memory. Have you ever set up a web application that took longer than the millisecond of patience people have when using the internet? There's a good bet you could cache quite a few requests and speed the whole thing up!

I used to do all kinds of nonsense where I would save things to temporary files and pick them back up later, or had cron jobs to stuff computations into pickle files. No more! Now I stick all that in a Redis in memory cache, which is way better and super fast!

By the end of this blog post you will have all the tools you need to query a Redis database, with some dummy data, from a python application, all without installing a single thing besides docker onto your computer! 

Learning Curve


Continue Reading...

Setup a MongoDB + Python Docker Dev Stack

Uncategorized May 13, 2019

MongoDB + Python

I only just recently jumped on the MongoDB bandwagon, but I really, really like it. To me, working with MongoDB feels so much more natural. It feels like working with code. MongoDB collections are arrays of objects, as opposed to a SQL DB where we kind of make it look like an object. Compared to SQL databases it is a much newer technology, but it is quickly gaining traction and becoming more and more robust.

With that said, it's nice to have a docker-compose template on hand to quickly spin up a MongoDB + Python Stack! That is exactly what we have here today. If you read my Setup a MySQL + Python Docker Dev Stack post, this will feel very similar.

MongoDB in Docker Compose

First of all, let's examine what just MongoDB looks like in docker compose. I'm going to go over only a very small portion of the potential configuration here. If you need more check out the official MongoDB docker image.

version: '3' services: mongo: image:...
Continue Reading...

Setup a Postgres + Python Docker Dev Stack

Uncategorized May 06, 2019

Query a Postgres DB from a Python Application

By the end of this blog post you will have all the tools you need to query a Postgres database, with some dummy data, from a python application, all without installing a single thing besides docker onto your computer! 

Learning Curve

If you read my Setup a MySQL + Python Docker Dev Stack, you'll know all about my how much trouble I had figuring out that pesky connection string. It was such a nitpicky little detail, but so crucial!

That single lesson, while very frustrating, has held through with every database I have ever connected to. It doesn't matter what kind of database you are connecting to. It could be mysql, postgres, mongodb, a redis cache. You just need to know the user, password, host, and default port! 

Onwards with an Example!

Docker-Compose networking MAGIC

When you spin up a docker-compose stack it kind of acts as if there are IT goblins living in your computer and creating hostnames...

Continue Reading...

50% Complete

DevOps for Data Scientists Weekly Tutorials

Subscribe to the newsletter! You'll get a weekly tutorial on all the DevOps you need to know as a Data Scientist. Build Python Apps with Docker, Design and Deploy complex analyses with Apache Airflow, build computer vision platforms, and more.