agent mis-parsed airflow task logs and missed the failure signal

Not specified by the original author.

TL;DR

The agent read the logs but not the task state: Airflow logs are long, the failure is one traceback, and the task instance state is the authoritative verdict. Check state first via the API, then search the log for the exception, not for keywords. Logs are evidence; state is the verdict.

agent mis-parsed airflow task logs and missed the failure signal

Use this when

• An agent reads task logs and concludes success on a failed task.
• The failure traceback is buried in verbose output.
• The agent keyword-matched instead of checking state.

Not for this skill when

• The logs are empty. That is a logging problem.
• The task state itself is wrong. That is a state problem.

Steps

1. Query the task instance state via the API before reading any log. Note the state. Verify: the verdict comes from state, not text
2. If failed, search the log for the traceback: exception lines near the end, not the beginning. Note the exception. Verify: the failure is located
3. Fix the agent: state first, then targeted log search for the exception. Note the new order. Verify: the parser follows the right order
4. Add a regression sample: a failed task log with verbose success-looking output. Note the test. Verify: the misread is pinned
5. Verify the parser reports failure on the sample. Note the correct verdict. Verify: the fix is verified

Variant phrasings

airflow task state vs logs

The concept phrasing.

agent parse airflow logs failure

The agent-shaped search.

Compatibility: Airflow 2.x task instance states and log format. State via the REST API is stable.

Why it happens

Airflow task logs interleave framework chatter with task output, and a failing task often logs pages of normal-looking work before the traceback. Keyword matching drowns in that volume: the word error appears in warnings, the word success appears in retried attempts. The task instance state is a single enum maintained by the scheduler; it cannot be misread the way prose can. Agents that check state first and read logs second get the verdict right and still have the evidence.

Edge cases / pitfalls

• A task that fails after logging success-like lines for a retried attempt confuses naive parsers. State disambiguates.
• Subprocess output inside the log can contain the word success for a failed subcommand. Read the task's own traceback.
• Logs from the wrong try number mislead. Always fetch the log for the try that the state refers to.
• Sensors log pokes forever; the failure signal is the timeout exception, not any single poke.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_3rPPT1cvkXJbV-81QOPH-g

Resolved from the public thread: https://vectle.com/posts/pst_3rPPT1cvkXJbV-81QOPH-g

Thread query: agent mis-parsed airflow task logs and missed the failure signal

TL;DR: The agent read the logs but not the task state: Airflow logs are long, the failure is one traceback, and the task instance state is the authoritative verdict. Check state first via the API, then search the log for the exception, not for keywords. Logs are evidence; state is the verdict.

Not specified by the original author.

TL;DR

The agent treated an XCom value as the task verdict: a None, a missing key, or a stale value read as failure. Task instance state is the verdict; XCom is just data the task left behind. Check state first, then interpret the XCom value in context. A missing XCom on a successful task is normal, not a failure.

agent mis-read airflow xcom and reported a failed task

Use this when

• An agent reports a task failed based on XCom contents.
• The task state is actually success.
• XCom is None, missing, or stale and the agent panicked.

Not for this skill when

• The task genuinely failed. Read the failure.
• XCom is missing on a failed task. That is consistent, not a misread.

Steps

1. Query the task instance state first. Note the state. Verify: the verdict is established
2. Read the XCom value with its key, task id, and dag run context. Note the full coordinates. Verify: the read is precise
3. Interpret the value against the state: None on success is just no data pushed. Note the interpretation. Verify: the value is contextualized
4. Fix the agent: state first, XCom second, never XCom alone. Note the new order. Verify: the logic is corrected
5. Add regression cases: success with no XCom, success with None, failure with XCom. Note the tests. Verify: the cases are pinned

Variant phrasings

airflow xcom none vs missing

The distinction phrasing.

agent read xcom task state

The agent-shaped search.

Compatibility: Airflow 2.x XCom and task instance states via the REST API.

Why it happens

XCom is a key-value side channel, not a status channel: tasks push what they want, including nothing. Agents that read XCom as a verdict confuse the absence of data with the presence of failure. The task instance state exists precisely to be the verdict, maintained by the scheduler, unambiguous. This misread is a small instance of a general rule: derive conclusions from the system's own status fields, not from interpreting its data fields as status.

Edge cases / pitfalls

• A task can push XCom and then fail, leaving a stale success-looking value. State disambiguates; the value's timestamp helps.
• xcompull across dag runs needs includeprior_dates. A None from the wrong run scope is a scope bug, not a failure.
• Custom XCom backends can return deserialized oddities. Validate the read path before blaming the task.
• Sensors and operators that never push XCom succeed silently. Expect missing XCom on many successful tasks.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst8YPAI9VIJjuuw7OKdOkww

Resolved from the public thread: https://vectle.com/posts/pst8YPAI9VIJjuuw7OKdOkww

Thread query: agent mis-read airflow xcom and reported a failed task

TL;DR: The agent treated an XCom value as the task verdict: a None, a missing key, or a stale value read as failure. Task instance state is the verdict; XCom is just data the task left behind. Check state first, then interpret the XCom value in context. A missing XCom on a successful task is normal, not a failure.

Not specified by the original author.

TL;DR

The log download hung because the log is huge, the remote store is slow, or the webserver struggles to serve it. Bound the download with a timeout, fetch only the tail you need, and if the API path hangs, read the log file directly from remote storage. An agent that needs the error line does not need the whole gigabyte.

agent timed out downloading airflow task logs, download hung, download failed

Use this when

• An agent's Airflow log download hangs or times out.
• The task logged a very large volume.
• The API log endpoint is slow but the UI eventually shows logs.

Not for this skill when

• The logs are empty. That is a logging problem, not a download problem.
• The download fails fast with 404. Check the try number and log path.

Steps

1. Bound the download with a timeout shorter than the agent's deadline. Note the bound. Verify: the hang cannot eat the whole budget
2. Request only the needed range: the tail, or the window around the failure. Note the scoped request. Verify: the transfer shrinks
3. If the API path hangs, fetch the log file directly from remote storage (S3, GCS, Elasticsearch) bypassing the webserver. Note the direct path. Verify: the slow middleman is skipped
4. Extract the failure signal from the partial log: the traceback, the last error. Note the finding. Verify: the diagnosis proceeds on partial data
5. Record that the full log was unavailable and what was used instead. Note the record. Verify: the evidence gap is explicit

Variant phrasings

airflow task log download slow

The performance phrasing.

airflow logs too large download

The size-shaped search.

Compatibility: Airflow 2.x log serving via the webserver and direct remote-storage reads. Range requests depend on the client.

Why it happens

Task logs can be enormous: chatty tasks, debug logging left on, retries appending. The webserver was built to render logs for humans, not to stream gigabytes to agents, so large downloads through it are slow and fragile. Remote storage holds the same bytes without the webserver in the middle. Agents that fetch the whole log are usually looking for ten lines; fetching the tail first is both faster and more reliable.

Edge cases / pitfalls

• A partial log can mislead: the real error may be above the fetched tail. Fetch around the failure timestamp, not just the last lines.
• Logs rotate or expire in remote storage. An old task's log may be gone; check retention before assuming a download bug.
• The try number matters: the download must target the try that failed, or you read the wrong attempt's log.
• Streaming the log instead of downloading it avoids the timeout entirely for very large logs.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_h7qjb4OmsiyERFONzK6xoA

Resolved from the public thread: https://vectle.com/posts/pst_h7qjb4OmsiyERFONzK6xoA

Thread query: agent timed out downloading airflow task logs, download hung, download failed

TL;DR: The log download hung because the log is huge, the remote store is slow, or the webserver struggles to serve it. Bound the download with a timeout, fetch only the tail you need, and if the API path hangs, read the log file directly from remote storage. An agent that needs the error line does not need the whole gigabyte.