How to Deploy Airflow From Dev to Prod Like A BOSS

Airflow Summit 2022 with Evgeny Shulman

Virtual Event
May 26, 2022
Airflow Summit 2022
01:00 PM

How to Deploy Airflow From Dev to Prod Like A BOSS

Managing Airflow in large-scale environments is tough. You know this and Evgeny knows this.

But, what if you had a guide to make development, testing, and production lifecycles more manageable?

In this presentation, Evgeny will share how he manages Airflow for large-scale environments with friendly deployments at every step.

After attending the session, Airflow engineers will:

  • Understand the advantages of each kind of deployment
  • Know the differences between Deployment and Airflow Executor
  • Deploy how to incorporate all kinds of deployments for their day-to-day needs

Save the date to learn how Databand can help you manage Airflow for large-scale environments!

Detailed Description:

  • Airflow Deployment
  • Why it’s important
  • What’s wrong with saying “I use k8s at my Airflow cluster”. Airflow Executor vs Airflow Deployment
  • Deploying our Airflow Environments for multiple use cases:
  • Docker-compose – for fast, iterative development
  • Kubernetes with Helm Chart – for production
  • CI/CD
  • Local development tips and tricks
  • Our development cycle:
  • Starting with docker-compose
  • Moving to production with Helm + Kubernetes
  • How to incorporate Airflow, as a standard part of our development lifecycle including version management and other moving pieces.
  • Challenges we had to solve
  • DAGs deployment
  • Managing connections and variables
  • What are the options?
  • A short intro into helm and SAAS.
  • Managed Airflow.
Evgeny Shulman
Co-founder and CTO @ Databand

Don’t want to wait until the event to see in action?

Schedule a demo