Airflow, Celery, and Temporal get compared as if they were three brands of the same tool. They are not. Airflow schedules data. Celery distributes tasks. Temporal guarantees workflows. Teams that pick by GitHub stars find this out at 2 a.m., usually in the same week their first LLM pipeline meets real traffic.
We run all three in production across client systems, and a fourth option for small stacks. This is the comparison I wish someone had handed me, with the receipts and the scars.
The one question that decides it
Ignore the feature matrices. Ask: what happens to my state when a worker dies mid-task?
- Airflow: the task is marked failed in the metadata database. The scheduler retries it on its declarative policy. Your data is late; nothing is lost, because Airflow never held your state, only your task status. This is fine, because Airflow's job is batch: DAGs of dependent steps on a schedule, backfills when history changes.
- Celery: by default, the task is gone. The broker handed it to the worker, the worker died, nobody tells you. You turn on
acks_lateand learn about visibility timeouts. Celery holds state in the broker exactly as long as the task is in flight, which is precisely when your worker is most likely to die. - Temporal: the workflow's entire history is an event log. A dead worker changes nothing; another worker replays the history and resumes from the last completed step. That guarantee is the whole product, and you pay for it in operational weight and a stricter programming model.
Everything else in this comparison is that answer wearing different costumes.
Where the 90-second LLM call hurts
AI workloads have a shape older queues were not tuned for: single calls that legitimately take 10 to 90 seconds, cost real money per attempt, and sometimes hang forever. From the demo's own first week, our reasoning model spends about 9 seconds thinking on every answer, and that was the fast case.
- Airflow tolerates long tasks, but its scheduling granularity is coarse. A DAG of hundreds of per-document LLM calls turns the scheduler into your bottleneck. Airflow wants chunky batch steps, not high-fan-out per-item calls.
- Celery is where the trap lives. If your visibility timeout is shorter than your longest LLM call, the broker assumes the worker died and re-delivers the task. Now two workers run the same call, and you pay the token bill twice for the privilege of a race condition. Every Celery-backed LLM system eventually rediscovers this.
- Temporal was built for exactly this shape: activities carry heartbeats and start-to-close timeouts, and a hung call is distinguishable from a slow one. This is the strongest technical argument for it in agentic systems.
Receipts: we run all three
Airflow, for data on a schedule. One of our ingestion platforms runs 15 Airflow DAGs feeding a Postgres knowledge schema. The wins are idempotent, schema-versioned steps and free backfills. Nobody there has ever asked Airflow to do anything in real time, which is why it works.
Celery, for volume with cost isolation. A studio-scale ETL we operate runs five isolated Celery queues: embeddings, AI relevancy, reports, ad hoc, default. Isolation is the point. Embedding backfills cannot starve customer-facing report generation, and per-tenant token counts get persisted for cost attribution. Celery at its best is cheap, fast, and everywhere; you just have to accept that its failure mode is silence, and watch queue depth like it owes you money.
Temporal, because Celery hit a wall. A document-intelligence SaaS we built ran its email ingestion on Celery until the workload outgrew it. We migrated to self-hosted Temporal, and promptly met Temporal's own limit: the roughly 2,000-pending-activity ceiling per workflow. The fix was per-thread keying with sliding-window concurrency, so no single workflow ever holds the whole backlog. Two lessons in one migration: the durability guarantee was worth the move, and every orchestrator has a ceiling you only find under production load.
And the honest fourth option. The live demo on this site runs none of them. It uses ARQ, a small asyncio Redis queue, with heartbeat-supervised workers that exit non-zero when their heartbeat dies and a chunk cap that routes poison documents to a dead-letter queue. Running a Temporal cluster for a demo would be operational cosplay. Match the substrate to the stakes.
The decision table
| You have | Pick | Because |
|---|---|---|
| Batch ingestion on a schedule, dependencies, backfills | Airflow | Data lateness is visible and recoverable; the scheduler is the product |
| High-volume short tasks, workloads that must not starve each other | Celery | Isolated queues + cheap horizontal workers; accept the silence, monitor queue depth |
| Multi-step workflows where a lost step costs money or trust (payments, agent actions, human approvals) | Temporal | Replayable history and signals; the guarantee is worth the cluster |
| A small asyncio service and one queue's worth of work | ARQ (or similar) | A supervised toy beats an unsupervised platform |
Two of these usually coexist. Airflow feeding batches into a Celery- or Temporal-backed serving path is a normal architecture, not a failure to decide. And if the real question is one layer down, between the two task queues themselves, Celery vs ARQ has its own comparison with the configuration scars.
The spine matters more than the model
Model choice gets all the attention, but when an agentic system fails in production, the postmortem is usually about the substrate: a re-delivered task that double-charged a customer, a queue that backed up silently, a workflow that lost its place. The orchestrator is the part of your AI system that decides whether a bad Tuesday costs you a retry or a reputation.
The full operational detail, including the runbooks and architecture decision records behind these systems, is in the operator brief. The playbook for choosing loops worth automating in the first place is in the first article of this series. And the ARQ-backed pipeline is running right now if you want to poke the small end of this spectrum.