TL;DR — Direct Answer
Airflow is not dying: it remains the most-deployed orchestrator on earth, with managed offerings on every cloud (Composer, MWAA, Astronomer) and a 3.x line that answered most old complaints (DAG versioning, better scheduling, datasets/assets). Most "Airflow pain" is self-inflicted — five antipatterns below cause 80% of it. The honest exceptions: asset-centric lineage-first platforms (Dagster wins) and dynamic, event-driven, bursty workflows (Prefect wins). For scheduled batch data platforms with a team that knows it — Airflow remains the boring, correct choice.
Why the "Airflow is legacy" narrative is wrong
Every cloud sells managed Airflow because demand keeps growing. The hiring market is full of engineers who already know it. The provider ecosystem (hundreds of maintained integrations) is unmatched. And Airflow 3 shipped the things people actually complained about: DAG versioning, a modern React UI, event-driven scheduling, and first-class data assets. Tools with this much gravitational mass don't die; they get boring — and boring is a compliment in orchestration.
The five ways teams actually break Airflow
1. Processing data inside tasks
Airflow is a scheduler, not a compute engine. Pandas transformations inside PythonOperators turn your scheduler into an underpowered, unobservable Spark cluster. Tasks should trigger work (Spark job, dbt run, BigQuery query) and wait — workers stay thin, retries stay cheap.
2. XCom as a data pipe
XCom is for metadata — a path, a row count, a run ID. Teams shoving DataFrames through XCom (serialized into the metadata DB!) create the slow, flaky Airflow they then blame. Pass references; store data in object storage.
3. Dynamic DAG generation from mutable sources
Generating DAGs at parse time from a database or API call means every scheduler heartbeat hits that source, and a slow response stalls all scheduling. Generate from static config in the repo; if the shape is truly dynamic, use dynamic task mapping inside a stable DAG instead.
4. One mega-DAG, or one DAG per table
Both extremes hurt: a 400-task mega-DAG makes every retry a negotiation; 400 single-task DAGs make lineage invisible. Right-size around business domains and use datasets (Airflow's asset triggers) to chain domains without monoliths.
5. No idempotency discipline
Half of "Airflow is unreliable" is tasks that can't be safely re-run. Every task should be a pure function of its logical date: partition-overwrite writes, deterministic inputs, no "append and hope." This is pipeline engineering, not tooling — no orchestrator fixes it.
Where the challengers honestly win
| Situation | Better tool | Why |
|---|---|---|
| Asset/lineage-first platform; strong local dev & testing culture | Dagster | Software-defined assets make lineage the primary abstraction; best-in-class local testing |
| Bursty, event-driven, highly dynamic flows; many ad-hoc runs | Prefect | Python-native dynamic flows without parse-time ceremony |
| dbt-only shop | dbt Cloud / scheduler | An orchestrator on top of one tool is overhead |
| Scheduled multi-system batch platform, mixed team | Airflow | Ecosystem, hiring pool, managed options, 3.x maturity |
We run all three in production for clients (pipeline engagements). The honest summary: tool choice moves outcomes ~20%; the idempotency, thin-task, and testing disciplines above move them ~80% — on any orchestrator.