agent timed out when airflow REST API hung on dag run status poll, poll failed

TL;DR

The Airflow API hung instead of answering, so the agent's poll never returned: the webserver is overloaded, the metadata database is slow, or the request queued behind heavy UI traffic. Put a hard timeout on every poll, back off exponentially, and treat repeated hangs as an Airflow health problem, not a run problem. Never poll without a timeout.

agent timed out when airflow REST API hung on dag run status poll, poll failed

Use this when

• Agent polls to the Airflow API hang instead of returning.
• The agent's own timeout fires on the poll, not on the run.
• Polls work sometimes and hang other times.

Not for this skill when

• The API returns errors quickly. Handle the error codes.
• The DAG run itself is stuck. Diagnose the run separately.

Steps

1. Set an explicit timeout on every API poll, shorter than the agent's overall deadline. Note polls cannot hang forever. Verify: the hang is bounded
2. Add exponential backoff with jitter between polls. Note poll pressure drops during outages. Verify: the agent stops adding load
3. After consecutive hung polls, check Airflow health: webserver load, database latency. Note the health signal. Verify: API hangs become a health question
4. Distinguish the two failures: API hung means unknown run state, not failed run. Note the distinction in the agent's reasoning. Verify: unknown is not failed
5. Escalate Airflow health problems with the poll history attached. Note the escalation. Verify: the right team gets the right evidence

Variant phrasings

airflow api timeout polling

The API-shaped search.

airflow rest api slow dag run status

The symptom phrasing.

Compatibility: Airflow 2.x REST API. Timeout and backoff are client-side patterns, API-version independent.

Why it happens

The Airflow webserver serves the UI, the API, and sometimes heavy ad-hoc queries from the same process pool. When the metadata database slows down, API requests queue behind everything else and a status poll that should take milliseconds hangs for minutes. An agent polling without its own timeout inherits that hang and burns its deadline waiting for an answer about a run that might be perfectly healthy. The poll timeout is what keeps an API problem from becoming an agent problem.

Edge cases / pitfalls

• Treating a hung poll as a failed run is the classic misread. Unknown state needs re-polling, not failure handling.
• Aggressive polling during an Airflow slowdown makes the slowdown worse. Back off; the API is telling you it is busy by not answering.
• The experimental API and the stable API have different performance characteristics. Use the stable v1 API for polling.
• Auth token expiry mid-polling looks like hangs on some clients. Refresh credentials on a schedule, not just on 401s.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ljkOnBSItLehK9sZMthgbw