Sign in Subscribe

Jillian Rowe

Jillian Rowe

Creating a Custom CellProfiler Docker Image

Creating a Custom CellProfiler Docker Image

Overview If you've ever worked with scientific software you will know that installing them is not necessarily straightforward. I think this is changing quite a bit with tools like conda and docker, but sometimes we need to just sit down and debug an installation. There is hope here,

Setup PyCharm to use a Project Interpreter in a Docker Compose Service

Setup PyCharm to use a Project Interpreter in a Docker Compose Service

Overview If you are anything like me, you heavily rely on a debugger. Likewise, you install all your development environments into a docker container because installing things on your own computer is FOR THE BIRDS. I cannot tell you what a relief it is to have all my development environments

Apache Airflow Tutorial – Part 4 DAG Patterns

Apache Airflow Tutorial – Part 4 DAG Patterns

Overview During the previous parts in this series, I introduced Apache Airflow in general, demonstrated my docker dev stack, and built out a simple linear DAG definition. I want to wrap up the series by showing a few other common DAG patterns I regularly use. In order to follow along,

Apache Airflow Tutorial – Part 3 Start Building

Apache Airflow Tutorial – Part 3 Start Building

Overview If you've read this far you should have a reasonable understanding of the Apache Airflowlayout and be up and running with your own docker dev environment. Well done! This part in the series will cover building an actual simple pipeline in Airflow. Start building by getting the

Apache Airflow Tutorial – Part 1 Introduction

Apache Airflow Tutorial – Part 1 Introduction

What is Apache Airflow? Briefly, Apache Airflow is a workflow management system (WMS). It groups tasks into analyses, and defines a logical template for when these analyses should be run. Then it gives you all kinds of amazing logging, reporting, and a nice graphical view of your analyses. I'

Apache Airflow Tutorial – Part 2 Install with Docker

Apache Airflow Tutorial – Part 2 Install with Docker

Install Apache Airflow With Docker Overview In this part of the series I will cover how to get a nice Apache Airflow instance up and running with docker. You won't need to have anything installed locally besides docker, which is fantastic, because configuring all these pieces individually would

Setting up a Local Spark Development Environment using Docker

Setting up a Local Spark Development Environment using Docker

Every time I want to get started with new tech I figure out how to get a stack up and running that closely resembles a real-world production instance as much as possible. This is a get up and running post. It does not get into the nitty gritty details of