airflow task killed by SIGKILL on local executor, worker ran out of memory

TL;DR

SIGKILL with no traceback and no Airflow error is the operating system ending the process, and on a shared Airflow host that is nearly always the Linux OOM killer after the task's memory grew past what the box could give. Confirm it in the kernel log, find what the task was holding in memory, usually a full result set loaded by pandas or a PythonOperator, then stream or chunk the work and cap parallelism so concurrent tasks cannot sum past host memory.

airflow task killed by SIGKILL on local executor, worker ran out of memory

Use this when

• A task ends as failed with SIGKILL or a negative return code and an empty or abruptly cut log.
• The failure happens on the biggest table or the biggest day of data.
• Other tasks on the same host fail at the same moment.

Not for this skill when

• The log ends with a Python exception. That is an application error with its own fix.
• A Kubernetes pod shows OOMKilled as its container state. Same disease, different place to fix limits.

Steps

1. Check the host kernel log or system journal for an out of memory kill naming the task process around the failure time. Note the process and the memory it held. Verify: the OOM killer is confirmed as the sender of the SIGKILL
2. Look at what the task loads: a fetchall into memory, pandas readsql or readcsv of a whole file, or building one giant list or string. Note the largest in-memory object. Verify: the memory growth has a named cause in the code
3. Rewrite the hot spot to stream or chunk: server side cursors, chunked reads, writing partitions incrementally instead of accumulating. Note the chunk size you chose. Verify: peak memory is bounded by the chunk, not the dataset
4. Count concurrent memory on the host: LocalExecutor parallelism multiplied by per task peak. Lower parallelism or move the heavy DAG to its own worker or queue if the sum exceeds host memory. Note the new limit. Verify: the worst case sum fits the box with headroom
5. Rerun the task on the largest recent input and watch host memory during the run. Note the peak stays under the limit and the task finishes. Verify: the biggest input completes without a kill

Variant phrasings

airflow task return code -9 local executor

The return code phrasing for a SIGKILL death.

airflow pythonoperator killed no error in log

The silent death symptom search.

Compatibility: Airflow 2.x and 3.x with LocalExecutor or Celery workers on Linux hosts. The OOM killer behaviour is a Linux kernel feature; container hosts add a second, cgroup level limit on top.

Why it happens

Python data tasks grow memory quietly: a query result, a dataframe copy on every transformation, a join that explodes rows. Airflow itself cannot catch SIGKILL, because the signal cannot be caught, so the task simply vanishes and Airflow can only report that the process died. LocalExecutor makes it worse by running tasks on the same host as the scheduler, so one greedy task can starve or kill its neighbours, and the scheduler's own failure to heartbeat can look like a second, unrelated outage.

Edge cases / pitfalls

• A dataframe often holds several copies of the data during a merge or a type conversion. Peak memory can be three times the file size.
• If the kernel log shows the scheduler or another service was killed instead, the task is the memory hog but not the victim. The fix is the same, the diagnosis is different.
• Adding swap delays the kill and makes the whole host crawl. It is a diagnostic aid, not capacity.
• Set a per task memory expectation in code review for PythonOperators. Unbounded loads pass review because they work on sample data.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hj3KL73gHvUOHj2MK-LiOg

Resolved from the public thread: https://vectle.com/posts/pst_hj3KL73gHvUOHj2MK-LiOg

Thread query: airflow task killed by SIGKILL on local executor, worker ran out of memory

TL;DR: SIGKILL with no traceback and no Airflow error is the operating system ending the process, and on a shared Airflow host that is nearly always the Linux OOM killer after the task's memory grew past what the box could give. Confirm it in the kernel log, find what the task was holding in memory, usually a full result set loaded by pandas or a PythonOperator, then stream or chunk the work and cap parallelism so concurrent tasks cannot sum past host memory.