agent timed out when agent tried to poll a long training run and failed after polling timeout error

TL;DR

Poll with a budget and a heartbeat check instead of a single long wait: query the run's status at lengthening intervals, verify liveness from the job system as well as the tracker, and when the budget ends report the run's last known state rather than failing as if the run failed. The timeout should reflect the job's expected duration class, set when the job is submitted, not a global default.

agent timed out when agent tried to poll a long training run and failed after polling timeout error

Use this when

• The run or call fails with this symptom: agent timed out polling a long training run after a polling timeout.
• You are working on an agent supervising long training runs to completion and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A job that genuinely exceeded any reasonable duration because it hung, where the fix is the job's own watchdog.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Record the job's expected duration and the tracker's last metric timestamp when polling starts. Verify: expectations and liveness signals are defined
2. Poll at growing intervals and treat fresh metric timestamps as liveness even when status stays running. Verify: polling distinguishes alive from stalled
3. When the budget expires, report the run as still running with its last progress instead of recording a failure. Verify: timeout produces a status report, not a false failure
4. Resume supervision later from the recorded run id and confirm the pattern survives the run's completion. Verify: the agent observes the true terminal state eventually

Variant phrasings

agent timed out polling long training run polling timeout

The verbatim phrasing.

agent polling timeout run still running liveness fix

The liveness phrasing.

supervising agent gave up on long run heartbeat metrics

The heartbeat phrasing.

Compatibility: Agent polling of long-running training jobs via tracker APIs

Why it happens

A poller's timeout is a statement about the poller's patience, not the job's health, but naive agents record it as the job's failure. Long jobs need supervision that separates the two, using metric freshness as the liveness signal the status field cannot provide.

Edge cases / pitfalls

• Doubling intervals without a cap overshoots completions by hours; cap the interval at a few minutes.
• Cluster schedulers expose queue and run states the tracker lacks; consult both before declaring a stall.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DOZL9VIiVXx7abPV-cS2hg