agent mis-parsed airflow task logs and missed the failure signal

TL;DR

The agent read the logs but not the task state: Airflow logs are long, the failure is one traceback, and the task instance state is the authoritative verdict. Check state first via the API, then search the log for the exception, not for keywords. Logs are evidence; state is the verdict.

agent mis-parsed airflow task logs and missed the failure signal

Use this when

• An agent reads task logs and concludes success on a failed task.
• The failure traceback is buried in verbose output.
• The agent keyword-matched instead of checking state.

Not for this skill when

• The logs are empty. That is a logging problem.
• The task state itself is wrong. That is a state problem.

Steps

1. Query the task instance state via the API before reading any log. Note the state. Verify: the verdict comes from state, not text
2. If failed, search the log for the traceback: exception lines near the end, not the beginning. Note the exception. Verify: the failure is located
3. Fix the agent: state first, then targeted log search for the exception. Note the new order. Verify: the parser follows the right order
4. Add a regression sample: a failed task log with verbose success-looking output. Note the test. Verify: the misread is pinned
5. Verify the parser reports failure on the sample. Note the correct verdict. Verify: the fix is verified

Variant phrasings

airflow task state vs logs

The concept phrasing.

agent parse airflow logs failure

The agent-shaped search.

Compatibility: Airflow 2.x task instance states and log format. State via the REST API is stable.

Why it happens

Airflow task logs interleave framework chatter with task output, and a failing task often logs pages of normal-looking work before the traceback. Keyword matching drowns in that volume: the word error appears in warnings, the word success appears in retried attempts. The task instance state is a single enum maintained by the scheduler; it cannot be misread the way prose can. Agents that check state first and read logs second get the verdict right and still have the evidence.

Edge cases / pitfalls

• A task that fails after logging success-like lines for a retried attempt confuses naive parsers. State disambiguates.
• Subprocess output inside the log can contain the word success for a failed subcommand. Read the task's own traceback.
• Logs from the wrong try number mislead. Always fetch the log for the try that the state refers to.
• Sensors log pokes forever; the failure signal is the timeout exception, not any single poke.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_3rPPT1cvkXJbV-81QOPH-g