The most persistent myth in data engineering is that you need a dedicated, heavy-duty orchestrator to manage a medallion architecture. If I hear one more architect argue that we need a complex DAG to move data from Bronze to Silver, I’m going to assume they’ve never had to debug a zombie process at 3:00 AM on a Sunday.
You don't need a complex orchestration layer. You need a reliable state machine that triggers your jobs when the data lands. Most of the complexity we build into our pipelines—retries, backfills, complex branching—is a direct result of choosing the wrong tool for the wrong scale.
You are likely staring at a familiar problem: you have a landing zone in S3 or ADLS, a Databricks workspace, and a business requirement that says the Silver table needs to be updated within fifteen minutes of the Bronze ingestion. You’re trying to decide whether to pay for the managed overhead of a workflow engine or just hack together some cloud-native glue.
The contenders
First, there’s AWS Step Functions. It’s the ultimate "I don't want to manage a server" play. You define your pipeline in Amazon States Language (ASL), essentially a JSON-based state machine. It integrates with everything in AWS, but it treats your data pipeline like a series of API calls.
Then, there’s Apache Airflow. It’s the industry standard for a reason: it’s Python. If you can write a script, you can write a DAG. But it’s a high-maintenance beast. You’re managing an environment—whether it’s MWAA, Cloud Composer, or a self-hosted Kubernetes cluster—and you’re fighting the "executor" war.
Finally, there’s Databricks Workflows. This is the "keep it in the house" option. It’s built into the platform where your compute already lives. It’s not just a job scheduler; it’s an integrated execution engine that knows the difference between a cluster startup failure and a Delta Lake transaction conflict.
Photo by Kellie Shepherd Moeller on Unsplash
Cost and the hidden tax of complexity
People often look at the price tag of a single execution and ignore the operational tax.
Step Functions looks cheap until you reach scale. At $25 per million state transitions, a complex pipeline with thousands of tasks will bleed your budget dry. More importantly, the cost of debugging a complex ASL JSON file is astronomical. When a TaskFailed event fires, you’re parsing CloudWatch logs and trying to reconstruct state from a visual graph that looks like a bowl of spaghetti.
Airflow is a hidden cost center. If you’re running it on a managed service like MWAA, you’re paying for the base instance price, which is rarely under $200-$300 a month before you even run a single job. If you’re self-hosting on EKS, factor in the engineering time. I’ve seen teams spend 20% of their "data engineering" time just keeping the Airflow scheduler and web server alive. That’s a massive salary tax to pay for the privilege of writing PythonOperator stubs.
Databricks Workflows is arguably the cheapest for medallion pipelines because of its integration with the compute layer. You aren't paying for a separate orchestration cluster. You’re using the existing Databricks compute, and the job orchestration is effectively free or negligible. You stop paying for the orchestrator the moment the job finishes. In a medallion architecture, where you’re often running jobs in sequence, this tight coupling is a feature, not a bug.
Failure modes and the "retry" trap
Let’s talk about real-world failure. In healthcare pipelines, "fail fast" is a requirement, not a suggestion.
In Airflow, the biggest failure mode is the "scheduler heartbeat." When your DAGs get too numerous or complex, the scheduler lags. You wake up to a dashboard showing nothing running, while your actual data is missing its SLA. You’re left digging through airflow-scheduler.log to find out why the heartbeats timed out. It’s a distributed system problem that you now own.
Step Functions has a "max duration" limit of one year, but it struggles with long-running, stateful processes. If your transformation job takes four hours and the underlying Lambda or Batch job hangs, Step Functions will keep the execution open, eating costs and potentially causing downstream concurrency limit issues. Retrying a failed step is easy, but retrying a failed data state (like a partial Delta write) is a nightmare you have to code manually.
Databricks Workflows has a massive advantage here: it understands the Delta Lake state. If a job fails because of a ConcurrentAppendException or a cluster start timeout, the platform handles the retries internally. It’s aware of the underlying compute, so if a node goes down, it doesn't just re-run the task; it re-provisions the cluster. It’s the only one of the three that doesn't treat your data pipeline like an abstract black box.
Photo by Francesca Tosolini on Unsplash
The operations burden and the "Python" illusion
Everyone loves Airflow because they love Python. It’s a trap.
When you write an Airflow DAG, you aren't writing data engineering code; you’re writing infrastructure code. You’re managing dependencies, requirements.txt files, and venv isolation. If you want to upgrade your Spark version, you might have to migrate your entire Airflow environment. It’s a classic case of the abstraction leaking.
Step Functions requires you to become an expert in Amazon States Language. It’s a declarative nightmare. You’ll find yourself writing nested Choice states just to handle a simple conditional logic that would have been a single if statement in any other language. You’re trading Python flexibility for AWS-specific lock-in.
Databricks Workflows allows you to define your pipeline in code—using Databricks Asset Bundles (DABs)—but it doesn't force you to manage the infrastructure. You define your task dependencies in YAML or Python, and the platform handles the deployment and the execution. It’s the only one that feels like a modern developer experience rather than a 2015-era sysadmin chore.
What I'd pick, and why
If you are building a medallion pipeline today, stop looking at Airflow. Unless you have a massive, heterogeneous environment where you need to orchestrate non-Databricks tasks (like triggering a legacy mainframe job or an on-prem file transfer), Airflow is overkill. It’s a platform for general-purpose workflow management, not a specialized tool for Delta Lake pipelines.
Step Functions is for when you are deep in the AWS ecosystem and need to trigger event-driven microservices. It’s not for heavy-duty ETL. If your pipeline involves moving gigabytes or terabytes through a Silver-to-Gold transformation, Step Functions will eventually become a bottleneck for your sanity.
I pick Databricks Workflows every single time for medallion architectures.
The caveat? You are locking yourself into the Databricks ecosystem. If your CTO decides to pivot to Snowflake or BigQuery tomorrow, you’re rewriting your orchestration layer. But let’s be honest: you’re already locked into Delta Lake. If you’re going to be locked in, you might as well use the tools that make your life easier.
Databricks Workflows with Databricks Asset Bundles (DABs) is the current gold standard. It gives you the CI/CD pipeline, the version control, and the observability you need without the "scheduler maintenance" headache of Airflow or the JSON-hell of Step Functions.
Use the right tool for the job. You’re a data engineer, not an Airflow cluster administrator. Spend your time fixing your data quality issues instead of babysitting a scheduler.
