Dask is a parallel computing library for Python. I think of it as being like MPI without actually having to write MPI code, which I greatly appreciate!

Dask natively scales Python

Dask provides advanced parallelism for analytics, enabling performance at scale for the tools you love

https://dask.org

One of the cooler aspects of Dask is that you scale across computers/servers/nodes/pods/container/etc. This is why I say it's like MPI.

What we'll be talking about today are:

• Advantages to using Kubernetes
• Disadvantages to using Kubernetes
• Install the Dask Helm Chart
• Scale your Dask Workers Up / Down with kubectl scale
• Modify the Dask Helm Chart to add Extra Packages
• Autoscale your Dask Workers with Horizontal Pod AutoScalers

Benfits to Dask on Kubernetes

Let's talk about some of the (many!) benefits to using Kubernetes!

Customizable Configuration

Another very important aspect of Dask, at least for me, is that I can set it up so that the infrastructure side of things is completely...

Deploy RShiny on AWS EKS with Terraform

Today we will be deploying the rocker/shiny image on Kubernetes with AWS, or EKS.

If you're following along with the deploy RShiny on AWS Series, you'll know that I covered deploying RShiny with a helm chart . Today, I want to go deeper into deploying RShiny on EKS, along with some tips and tricks that I use for my everyday deployments.

An Extremely Brief Introduction to Kubernetes

Kubernetes is kind of a beast to get started with, and people constantly complain that its extremely complicated to get started with. They would be correct, but I'm here to give the 2 minute rundown of what you need to know to deploy your RShiny (or Dash, Flask, Django, Ruby on Rails, etc) application on Kubernetes. This is because Kubernetes is not magical, and it's not even that new. It's a very nice abstraction layer on top of infrastructure concepts that already exist.

Disclaimer! I know I am oversimplifying things here, and I am totally...

If you want a very robust solution to deploy your RShiny applications a managed Kubernetes service on AWS, or EKS, is a great choice! Kubernetes has, for now, fully dominated the DevOps and deployment space in terms of deploying containerized services.

Should I Use Kubernetes to Deploy my RShiny App?

As with all things, there are pros and cons to using Kubernetes.

Cons

• Kubernetes is a beast to get started with.
• You need to have your RShiny App packaged with docker.

Pros

• Kubernetes is incredibly well supported across a range of providers, including AWS, GCP, Azure, Digital Ocean, etc.
• Kubernetes is complicated, but with complicated comes armies of engineers who want to abstract away the complicated into configuration files. So you if you stick with the growing pains it's likely you will get a lot of benefit and things will get less painful over time.
• Kubernetes has been embraced by the "people who deploy stuff" community, which means that there are a lot of resources out for it.
• ...

AWS Elastic Kubernetes Service (EKS) is a fully managed service AWS launched recently. Elastic services in AWS it means that the number of instances actually in use scales up or down based on the demand. This is first of all seriously cool, and second of all can cut down on costs. Fewer requests? Fewer nodes!

I'm just getting started with Kubernetes myself, and going through this walkthrough was a great learning exercise. 

I love deploying applications with docker swarm because it's fairly simple and I already know it, however, Swarm for AWS has some downsides. Firstly, it is not elastic, secondly, in order to get sticky sessions you need to add an additional service such as Traefik. With session affinity you can deploy RShiny and Python Dash applications with no other functionality besides the built in, and that's amazing!

I also personally think the moving towards Kubernetes over Swarm. It even comes installed on my mac version of Docker. Now is a great time to get...