Deploy and Scale your Dask Cluster with Kubernetes
Dask is a parallel computing library for Python. I think of it as being like MPI without actually having to write MPI code, which I greatly appreciate!
Dask natively scales Python
Dask provides advanced parallelism for analytics, enabling performance at scale for the tools you love
https://dask.org
One of the cooler aspects of Dask is that you scale across computers/servers/nodes/pods/container/etc. This is why I say it's like MPI.
What we'll be talking about today are:
- Advantages to using Kubernetes
- Disadvantages to using Kubernetes
- Install the Dask Helm Chart
- Scale your Dask Workers Up / Down with
kubectl scale - Modify the Dask Helm Chart to add Extra Packages
- Autoscale your Dask Workers with Horizontal Pod AutoScalers
Benfits to Dask on Kubernetes
Let's talk about some of the (many!) benefits to using Kubernetes!
Customizable Configuration
Another very important aspect of Dask, at least for me, is that I can set it up so that the infrastructure side of things is completely...
Categories
All CategoriesRecent Posts
Would you like to work together to design and implement your data science and analytics infrastructure? Do you need an AWS Batch or HPC cluster for your High Throughput analysis, or a Data Visualization Application deployed on AWS?
Receive weekly tutorials on only the DevOps you need to know as a Data Scientist.
During these tutorials I will teach you to:
- Build Python Apps with Docker
- Design and Deploy complex analyses with Apache Airflow
- Build computer vision platforms
- Deploy RShiny with Docker and on AWS
- And more!
Apache Airflow Project Template
Build out your Data Science with Apache Airflow.
- Build custom docker images
- Bring up your stack with one command
- Add plugins and custom REST APIs by understanding Airflow best practices
- Deploy to Kubernetes with Helm using a ready made Helm chart.
Deploy Kubernetes on AWS with:
- Terraform - infrastructure as code to give you a repeatable recipe for your Kubernetes deployment
- Detailed documentation
- Get setup up with a configurable cluster that can be deployed for different projects, or environments (dev/stage/prod)
This guide provides you with a birds eye view of which AWS services to use for:
- Exploratory analyses
- Production pipelines
- Data visualization platforms (including RShiny and Dash)
- and Machine Learning Pipelines
These are the services that I use every day to deploy Bioinformatics related infrastructure.
If you're looking for even more in depth guides or project templates check out my store.