agent's CI poll loop never saw the run finish: webhook vs polling

TL;DR

A poll loop that never sees the run finish is usually polling the wrong endpoint, missing terminal states, or racing completion. Prefer webhooks: the CI system calls you when the run finishes. Where polling must stay, handle every terminal state explicitly, add a heartbeat, and cap total wait time. An unbounded poll is a bug.

Error

agent's CI poll loop never saw the run finish: webhook vs polling

The agent polled the CI status endpoint for an hour, the run finished in twelve minutes, and the loop kept going until the job timed out.

Steps

1. Diagnose the miss: check what the poll loop requested versus what the CI API returned (wrong run id, cached status, unhandled terminal state like cancelled). Expected output: the exact reason the loop never observed completion.
2. Switch to webhooks where the CI system supports them: register a callback for run completion and have the agent wait on the callback. Expected output: the agent wakes exactly when the run finishes.
3. For polling that must remain: enumerate every terminal state (success, failure, cancelled, timed out, skipped) and treat any of them as "done". Expected output: a state list with no gaps.
4. Add a heartbeat log line per poll interval and a hard cap on total polling time, after which the loop escalates instead of spinning. Expected output: visible progress and a bounded wait.
5. Make status fetches idempotent and cheap: cache the last state, back off between polls, stop polling once terminal. Expected output: a loop that cannot run forever.

When to use

• Poll loops outlive the runs they watch.
• The agent misses completion and the job times out.
• CI offers webhooks you are not using.

When not to use

• The CI API is synchronous and reliable and a bounded poll already works (do not rebuild what works).
• Webhooks cannot reach the agent (no inbound network); then fix the polling properly instead.
• The "never finishes" is actually a stuck CI run (then the run is the problem, not the loop).

Variant phrasings

Agent polled the wrong run id

A surprisingly common miss: the loop watches a stale id while the real run finishes elsewhere. Log the watched id at loop start.

CI status stayed "in progress" after cancellation

Cancelled is a terminal state the loop did not handle. Enumerate all of them.

Webhook arrived but the agent was not listening yet

Race between registration and completion. Register the webhook before triggering the run, and reconcile with one status fetch at startup.

Why it happens

Polling feels simple, so it gets written without a state machine: no terminal-state list, no timeout, no logging. CI APIs have quirks (eventual consistency, cached statuses, states the author never saw), and the naive loop falls into exactly one of them. Webhooks invert the problem: the CI system, which knows when it is done, tells you.

Edge cases

• Webhook delivery can fail or duplicate; make the handler idempotent and keep a reconciliation poll as backup.
• Signed webhooks need verification; verify the signature before acting on the payload.
• Some CI systems only webhook on certain events; confirm completion events are covered, not just start events.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_R8SwYvD474fQlq6-zh8nTw