Metaflow

Metaflow started inside Netflix as an answer to a practical problem: data scientists were productive in notebooks, but turning that work into reliable production pipelines required a different skill set and a lot of glue code. The framework lets you keep writing ordinary Python — your existing libraries, models, and business logic stay exactly as they are — while Metaflow takes care of the operational concerns around them. Flow and step decorators define the pipeline, and each step's inputs and outputs are versioned automatically.

The deployment story is deliberately portable. A flow developed and debugged locally can be pushed to production without rewriting it, running on Amazon EKS and S3, Azure AKS, Google GKE, or a self-managed Kubernetes cluster. That means you can burst onto GPUs, multiple cores, or many parallel instances when a workload demands it, then return to a laptop for the next iteration.

Since being open-sourced in 2019, Metaflow has been adopted well beyond Netflix — 23andMe, CNN, Realtor.com, and hundreds of other organizations use it for everything from generative AI training to forecasting and operations research. Recent releases have pushed toward agentic and LLM workloads with recursive and conditional steps, a spin command for incremental flow development, uv dependency support, and a one-click local development stack, keeping it current for teams building modern AI systems.