agent loop detected: agent polls Kubeflow run status forever after run deleted error

TL;DR

Treat a deleted run as a terminal discovery, not a polling target: the agent polls a Kubeflow run that no longer exists, so confirm the deletion from the run list, stop the poll, and report the run as deleted with whatever artifacts survive in storage. Not found on a status poll means the run left the system. Polling cannot bring it back, and the loop delays the only useful work: checking what the run left behind.

agent loop detected: agent polls Kubeflow run status forever after run deleted error

Use this when

• The check or run fails with this symptom: an agent loop polls Kubeflow run status forever after the run is deleted.
• You are working on an agent workflow monitoring Kubeflow pipeline runs to completion and the failure repeats on the same data or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A run that is not found because of a namespace mistake, which should be corrected and re-polled rather than reported as deleted.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. List runs in the namespace and search by the run name and pipeline. Verify: the run is absent from the listing, confirming deletion rather than a lookup error
2. Check the artifact storage for the run outputs. Verify: surviving artifacts are inventoried
3. Stop the poll and record the outcome as deleted, with the last known status. Verify: the workflow exits with a truthful result
4. Add not-found handling to the poller: after a small number of consecutive not-found responses, verify against the run list and exit. Verify: a repeat deletion terminates the poll quickly
5. Note the retention or deletion source if the platform records it. Verify: the team learns why the run vanished

Variant phrasings

agent polls kubeflow run status forever run deleted

The shorter phrasing focused on the named failure.

kubeflow run deleted agent loop error

The symptom phrasing a searcher tries next.

kubeflow poller not found exit fix

The variant naming the failing stage or tool directly.

Compatibility: Kubeflow Pipelines run retention and agent polling workflows

Why it happens

Pollers are written for the states they expect: running, then a terminal state. Deletion by retention, by a user, or by a namespace cleanup removes the run from the state machine entirely, and a poller without a not-found exit waits on a ghost.

Edge cases / pitfalls

• Distinguish deleted from never existed by checking the submission record the workflow stored at launch.
• Artifact storage usually outlives run metadata; check it before declaring the work lost.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_oL57jtmvsFn7clSyhr08kw