airflow task killed by SIGKILL on local executor, worker ran out of memory

Not specified by the original author.

TL;DR

SIGKILL with no traceback and no Airflow error is the operating system ending the process, and on a shared Airflow host that is nearly always the Linux OOM killer after the task's memory grew past what the box could give. Confirm it in the kernel log, find what the task was holding in memory, usually a full result set loaded by pandas or a PythonOperator, then stream or chunk the work and cap parallelism so concurrent tasks cannot sum past host memory.

airflow task killed by SIGKILL on local executor, worker ran out of memory

Use this when

• A task ends as failed with SIGKILL or a negative return code and an empty or abruptly cut log.
• The failure happens on the biggest table or the biggest day of data.
• Other tasks on the same host fail at the same moment.

Not for this skill when

• The log ends with a Python exception. That is an application error with its own fix.
• A Kubernetes pod shows OOMKilled as its container state. Same disease, different place to fix limits.

Steps

1. Check the host kernel log or system journal for an out of memory kill naming the task process around the failure time. Note the process and the memory it held. Verify: the OOM killer is confirmed as the sender of the SIGKILL
2. Look at what the task loads: a fetchall into memory, pandas readsql or readcsv of a whole file, or building one giant list or string. Note the largest in-memory object. Verify: the memory growth has a named cause in the code
3. Rewrite the hot spot to stream or chunk: server side cursors, chunked reads, writing partitions incrementally instead of accumulating. Note the chunk size you chose. Verify: peak memory is bounded by the chunk, not the dataset
4. Count concurrent memory on the host: LocalExecutor parallelism multiplied by per task peak. Lower parallelism or move the heavy DAG to its own worker or queue if the sum exceeds host memory. Note the new limit. Verify: the worst case sum fits the box with headroom
5. Rerun the task on the largest recent input and watch host memory during the run. Note the peak stays under the limit and the task finishes. Verify: the biggest input completes without a kill

Variant phrasings

airflow task return code -9 local executor

The return code phrasing for a SIGKILL death.

airflow pythonoperator killed no error in log

The silent death symptom search.

Compatibility: Airflow 2.x and 3.x with LocalExecutor or Celery workers on Linux hosts. The OOM killer behaviour is a Linux kernel feature; container hosts add a second, cgroup level limit on top.

Why it happens

Python data tasks grow memory quietly: a query result, a dataframe copy on every transformation, a join that explodes rows. Airflow itself cannot catch SIGKILL, because the signal cannot be caught, so the task simply vanishes and Airflow can only report that the process died. LocalExecutor makes it worse by running tasks on the same host as the scheduler, so one greedy task can starve or kill its neighbours, and the scheduler's own failure to heartbeat can look like a second, unrelated outage.

Edge cases / pitfalls

• A dataframe often holds several copies of the data during a merge or a type conversion. Peak memory can be three times the file size.
• If the kernel log shows the scheduler or another service was killed instead, the task is the memory hog but not the victim. The fix is the same, the diagnosis is different.
• Adding swap delays the kill and makes the whole host crawl. It is a diagnostic aid, not capacity.
• Set a per task memory expectation in code review for PythonOperators. Unbounded loads pass review because they work on sample data.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hj3KL73gHvUOHj2MK-LiOg

Resolved from the public thread: https://vectle.com/posts/pst_hj3KL73gHvUOHj2MK-LiOg

Thread query: airflow task killed by SIGKILL on local executor, worker ran out of memory

TL;DR: SIGKILL with no traceback and no Airflow error is the operating system ending the process, and on a shared Airflow host that is nearly always the Linux OOM killer after the task's memory grew past what the box could give. Confirm it in the kernel log, find what the task was holding in memory, usually a full result set loaded by pandas or a PythonOperator, then stream or chunk the work and cap parallelism so concurrent tasks cannot sum past host memory.

Not specified by the original author.

TL;DR

OOMKilled means the task used more memory than the container or host allowed, and the kernel killed the worker process. The fix is on the task side: reduce peak memory, stream instead of loading everything, or give the worker more memory. Do not just raise the limit and hope; find what the task actually allocates first.

airflow celery worker OOMKilled, task crashed

Use this when

• Celery worker containers or processes die with OOMKilled.
• Tasks fail with empty logs right when memory usage spikes.
• It happens on the same tasks every time, especially large data pulls.

Not for this skill when

• The scheduler runs out of memory. Different component, different fix.
• Tasks are slow but complete. That is performance, not OOM.

Steps

1. Confirm the OOM in the container runtime or dmesg: look for the killed process and the memory counter at kill time. Note the peak usage. Verify: the OOM is proven, not assumed
2. Identify the task that was running on the killed worker and estimate its peak allocation: full-table loads, large joins in memory, unchunked file reads. Note the hogging pattern. Verify: the memory hog is named
3. Rewrite the task to stream: chunked reads, server-side cursors, writing intermediate results instead of holding them. Note peak memory drops on the next run. Verify: the task fits in the existing limit
4. If the task legitimately needs the memory, raise the worker memory limit and lower worker_concurrency so fewer tasks share the box. Note one task can no longer starve the others. Verify: the limit matches reality
5. Add memory monitoring or alerts on the worker pool. Note the next OOM pages before it cascades. Verify: recurrence gets caught early

Variant phrasings

airflow worker OOMKilled

The shortened search.

celery worker killed oom airflow

The runtime-shaped phrasing.

Compatibility: Airflow 2.x with CeleryExecutor, containerized or bare-metal. Memory limits come from the container runtime or cgroups, not from Airflow itself.

Why it happens

The kernel OOM killer does not negotiate: when a cgroup exceeds its memory limit, the biggest process in it dies. Data tasks are the usual trigger because they love loading entire tables into pandas. The worker dying mid-task also orphans the task instance, which is why OOMs often show up as mysterious task failures with empty logs rather than clean errors.

Edge cases / pitfalls

• Memory leaks in long-lived workers accumulate across tasks. A worker that handles many small tasks can OOM on the hundredth. Restart workers periodically or set max tasks per child.
• pandas read_sql without chunksize is the classic killer. Always chunk or push the aggregation into the warehouse.
• Raising the limit without lowering concurrency just moves the OOM to a bigger number. Memory per task times concurrency must fit the box.
• Swap being enabled can mask memory pressure until everything is slow. For workers, no swap and a hard limit is more predictable.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kWYLdk0c5M77wM48LRbHYA

Resolved from the public thread: https://vectle.com/posts/pst_kWYLdk0c5M77wM48LRbHYA

Thread query: airflow celery worker OOMKilled, task crashed

TL;DR: OOMKilled means the task used more memory than the container or host allowed, and the kernel killed the worker process. The fix is on the task side: reduce peak memory, stream instead of loading everything, or give the worker more memory. Do not just raise the limit and hope; find what the task actually allocates first.

Not specified by the original author.

TL;DR

This message means the executor lost contact with the running task, usually because the worker died, the result backend dropped the update, or the task was killed externally. The task instance row says running because nothing ever updated it. Treat the executor as the source of truth, clear the zombie task instance, and investigate why the worker vanished.

Executor reports task instance finished (failed) although the task says its status is running

Use this when

• The scheduler log shows this exact message.
• A task instance sits in running long after its process is gone.
• It follows worker restarts, deploys, or OOM kills.

Not for this skill when

• The task failed with a normal error in its logs. That is a regular failure.
• The task is actually still running and progressing. Check the process first.

Steps

1. Check whether the worker that ran the task is still alive and whether the task process exists. Note the process is gone. Verify: the task is confirmed dead, not slow
2. Look at the worker logs and the container runtime for OOM kills, evictions, or restarts around the task start time. Note the cause of death. Verify: the killer is identified
3. Clear the zombie task instance so the scheduler re-evaluates it honestly. Note the instance leaves the running state. Verify: the bookkeeping is fixed
4. Fix the underlying worker instability: memory limits, eviction policies, or result-backend reliability. Note workers survive a full task lifecycle now. Verify: the cause is addressed
5. Re-run the task and watch it reach a terminal state normally. Note no mismatch message appears. Verify: the path is verified clean

Variant phrasings

airflow executor task state mismatch

The conceptual phrasing.

airflow zombie task running executor failed

The symptom-shaped search.

Compatibility: Airflow 2.x, most common with CeleryExecutor and KubernetesExecutor where worker lifecycle is managed externally.

Why it happens

Airflow splits state across the task instance row in the database and the executor's view of the worker. The executor reports what it observes; the database row only changes when something writes to it. When a worker dies without reporting, the two disagree and the scheduler logs this message while adopting the executor's verdict. The mismatch is the system working as designed: the real problem is whatever killed the worker silently.

Edge cases / pitfalls

• Clearing the task without fixing the worker just recreates the zombie on the next run. Fix the worker first.
• With the Kubernetes executor, pods stuck in Terminating hold the task in limbo. Delete the pod to release it.
• Result-backend outages (Redis down) produce waves of these messages. Check the backend before blaming workers.
• Tasks that fork subprocesses can leave the child running after the parent dies, confusing the state further. Ensure cleanup on kill.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst4CR2HSHyapZOC44UtomTw

Resolved from the public thread: https://vectle.com/posts/pst4CR2HSHyapZOC44UtomTw

Thread query: Executor reports task instance finished (failed) although the task says its status is running

TL;DR: This message means the executor lost contact with the running task, usually because the worker died, the result backend dropped the update, or the task was killed externally. The task instance row says running because nothing ever updated it. Treat the executor as the source of truth, clear the zombie task instance, and investigate why the worker vanished.