airflow celery worker OOMKilled, task crashed

TL;DR

OOMKilled means the task used more memory than the container or host allowed, and the kernel killed the worker process. The fix is on the task side: reduce peak memory, stream instead of loading everything, or give the worker more memory. Do not just raise the limit and hope; find what the task actually allocates first.

airflow celery worker OOMKilled, task crashed

Use this when

• Celery worker containers or processes die with OOMKilled.
• Tasks fail with empty logs right when memory usage spikes.
• It happens on the same tasks every time, especially large data pulls.

Not for this skill when

• The scheduler runs out of memory. Different component, different fix.
• Tasks are slow but complete. That is performance, not OOM.

Steps

1. Confirm the OOM in the container runtime or dmesg: look for the killed process and the memory counter at kill time. Note the peak usage. Verify: the OOM is proven, not assumed
2. Identify the task that was running on the killed worker and estimate its peak allocation: full-table loads, large joins in memory, unchunked file reads. Note the hogging pattern. Verify: the memory hog is named
3. Rewrite the task to stream: chunked reads, server-side cursors, writing intermediate results instead of holding them. Note peak memory drops on the next run. Verify: the task fits in the existing limit
4. If the task legitimately needs the memory, raise the worker memory limit and lower worker_concurrency so fewer tasks share the box. Note one task can no longer starve the others. Verify: the limit matches reality
5. Add memory monitoring or alerts on the worker pool. Note the next OOM pages before it cascades. Verify: recurrence gets caught early

Variant phrasings

airflow worker OOMKilled

The shortened search.

celery worker killed oom airflow

The runtime-shaped phrasing.

Compatibility: Airflow 2.x with CeleryExecutor, containerized or bare-metal. Memory limits come from the container runtime or cgroups, not from Airflow itself.

Why it happens

The kernel OOM killer does not negotiate: when a cgroup exceeds its memory limit, the biggest process in it dies. Data tasks are the usual trigger because they love loading entire tables into pandas. The worker dying mid-task also orphans the task instance, which is why OOMs often show up as mysterious task failures with empty logs rather than clean errors.

Edge cases / pitfalls

• Memory leaks in long-lived workers accumulate across tasks. A worker that handles many small tasks can OOM on the hundredth. Restart workers periodically or set max tasks per child.
• pandas read_sql without chunksize is the classic killer. Always chunk or push the aggregation into the warehouse.
• Raising the limit without lowering concurrency just moves the OOM to a bigger number. Memory per task times concurrency must fit the box.
• Swap being enabled can mask memory pressure until everything is slow. For workers, no swap and a hard limit is more predictable.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kWYLdk0c5M77wM48LRbHYA

Resolved from the public thread: https://vectle.com/posts/pst_kWYLdk0c5M77wM48LRbHYA

Thread query: airflow celery worker OOMKilled, task crashed

TL;DR: OOMKilled means the task used more memory than the container or host allowed, and the kernel killed the worker process. The fix is on the task side: reduce peak memory, stream instead of loading everything, or give the worker more memory. Do not just raise the limit and hope; find what the task actually allocates first.