Dask is a parallel computing library for Python. I think of it as being like MPI without actually having to write MPI code, which I greatly appreciate!
Dask natively scales Python
Dask provides advanced parallelism for analytics, enabling performance at scale for the tools you love
One of the cooler aspects of Dask is that you scale across computers/servers/nodes/pods/container/etc. This is why I say it's like MPI.
What we'll be talking about today are:
Let's talk about some of the (many!) benefits to using Kubernetes!
Another very important aspect of Dask, at least for me, is that I can set it up so that the infrastructure side of things is completely...
For a few years now I have been on a quest to find a tool I really like for annotating HCS images using a web interface. I've used several tools, including a desktop application called LabelImg, and I have finally found a tool that checks all the boxes called LabelStudio!
Label Studio is an annotation tool for images, audio, and text. Here we'll be concentrating on images as our medium of choice.
I go through the process in this video.
You can, of course, use your own data, but for this tutorial I will be using a publically available C. elegans dataset from the Broad BioImage Benchmark Collection.
mkdir data cd data wget https://data.broadinstitute.org/bbbc/BBBC010/BBBC010_v2_images.zip unzip BBBC010_v2_images.zip
HCS images are often very dark when opened in a system viewer. To use them for the rest of the pipeline we will have to do a two step conversion process, first using bftools to convert from tif -> png, and then using Imagmagick to do a levels...
If you're following along with the deploy RShiny on AWS Series, you'll know that I covered deploying RShiny with a helm chart . Today, I want to go deeper into deploying RShiny on EKS, along with some tips and tricks that I use for my everyday deployments.
Kubernetes is kind of a beast to get started with, and people constantly complain that its extremely complicated to get started with. They would be correct, but I'm here to give the 2 minute rundown of what you need to know to deploy your RShiny (or Dash, Flask, Django, Ruby on Rails, etc) application on Kubernetes. This is because Kubernetes is not magical, and it's not even that new. It's a very nice abstraction layer on top of infrastructure concepts that already exist.
Disclaimer! I know I am oversimplifying things here, and I am totally...
Which solution you go for will depend upon your own needs. Normally, I am using a lot of bioinformatics and/or data science packages and nothing really beats the conda ecosystem for that in my opinion. On the other hand having an image with RShiny already installed is really nice! I also quite like the way that the
rocker/shiny image is configured. You drop your files into a directory and as long as you have the packages you need your shiny app will start up directly!
Here is our usual shiny app mostly stolen from the R shiny docs . ;-)
#!/usr/bin/env Rscript # This example comes from the r-shiny examples github repo. # https://github.com/rstudio/shiny-examples/blob/master/001-hello/app.R library(shiny) # Define UI for app that draws a histogram ---- ui <- fluidPage( # App title ---- titlePanel("Hello Shiny!"), #...
If you want a very robust solution to deploy your RShiny applications a managed Kubernetes service on AWS, or EKS, is a great choice! Kubernetes has, for now, fully dominated the DevOps and deployment space in terms of deploying containerized services.
As with all things, there are pros and cons to using Kubernetes.
If you're deploying applications on AWS one of the easier ways to get started is to simply deploy everything on EC2. If you're not familiar with AWS EC2, EC2 instances are compute instances. They are like your (linux) desktop, except that yous spin them up and kill them on demand with AWS console (or CLI or some other infrastructure automation tool).
If you're looking for a solution that has some built in scale and ability to throw a Load Balancer at it EC2 is fine. It's also nice because it's just a computer. You don't have to find a work around to get you a computer the way you do with Docker (docker run) or Kubernetes (kube exec). You deal with an EC2 instance the way you would any remote server, but its on AWS so you get some additional niceness such as the backups, Elastic IPs, and Load Balancers.
This is slightly more difficult than using LightSail. If you are completely new to AWS and on a tight deadline you may want to start there....
My favorite way to deploy RShiny locally is to simply package it into a docker image and run it. Running any application in docker makes it easily transportable, and is a generally acceptable way of distributing applications in 2019.
This solution does not require rshiny-server or shinyapps.io, not because I have anything against either of those solutions. I just tend to stick to a few favorite deployment methods to keep my head from spinning straight off my body. ;-)
If you're not familiar with docker then I have a FREE course available. The first module is plenty to
get you up and running and can be completed in an hour or two. For the most part, if you can use a command line you can use docker.
Now that we've covered some housekeeping let's get started building your docker image. Like any project, you want to have all your relevant code in a directory. This way it is accessible to the docker build process.
I've been having a blast the last few months learning ApacheAirflow. It's become an indispensable tool in my getting stuff done toolbox.
Follow me on Twitter @jillianerowe, or send me a message with questions, topic suggestions, ice cream recommendations, or general shenanigans.
If your head is spinning over which deployment scenario to choose for your RShiny app look no further! I have a whole series planned out for you on various deployment options and why you should choose each one.
Deployment scenarios are like snowflakes. No two are exactly alike! ;-) You need different level of power and control for different deployment scenarios. Here we are going to talk about RShiny deployments, but it applies to just about everything.
Lightsail is a relatively recent addition to the whole AWS deployment ecosystem. It makes it much simpler and more streamlined to deploy a web application than some of their other, more powerful solutions.
Lightsail would be a good choice for deployment for you if:
If you're looking to start a freelance business in tech, or any business really, then you'll need to find a way to have the clients knocking down your doors. To make this a very short story you mostly do this by putting yourself out there, being helpful, and talking to people.
More specifically, you can start a blog to demonstrate your awesome knowledge (get the word out there) with awesome content (helping people!).
I've been a software engineer for nearly 10 years (how time flies!) and about 6 months ago I started to seriously consider the idea that I wanted to venture off on my own. Be my own boss, be mistress of my own destiny, all that kind of thing. My biggest worry was well how do I find my own clients?
One of the first pieces of advice I got was to start a blog, and it's worked out fairly well for me, so here we are!
I'm not affiliated with any of these products or services. These are simply my opinions.
Well, first of all, think of a name...