Kubeflow pipeline run failed error "Run is not found in experiment namespace"

Not specified by the original author.

TL;DR

Look for the run in the right namespace and pipeline: Kubeflow run lookups are scoped, so confirm the experiment name, the namespace, and the pipeline run id belong together before concluding the run is gone. A run id copied from another namespace, or an experiment name used where a run id belongs, returns not found even though the run exists and may still be executing.

Kubeflow pipeline run failed error "Run is not found in experiment namespace"

Use this when

• The check or run fails with this symptom: a Kubeflow pipeline run fails with Run is not found in experiment namespace.
• You are working on a Kubeflow Pipelines deployment where runs are organised by experiment and namespace and the failure repeats on the same data or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A run that was deleted by retention, which is genuinely absent and needs its artifacts recovered from storage instead.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. List runs in the namespace without a name filter and find the run by its job or pipeline name. Verify: the run appears, or it truly is absent
2. Compare the experiment name and namespace in the failing call with the run actual location. Verify: the mismatch is identified
3. Correct the call to use the run id from the submission response for that namespace. Verify: the lookup returns the run
4. Store the submission response, including run id and namespace, with the pipeline record. Verify: later lookups use the stored pair
5. Rerun the status check. Verify: the run status returns normally

Variant phrasings

kubeflow run is not found in experiment namespace

The shorter phrasing focused on the named failure.

kubeflow pipeline run failed not found error

The symptom phrasing a searcher tries next.

kubeflow run namespace lookup fix

The variant naming the failing stage or tool directly.

Compatibility: Kubeflow Pipelines 2.x experiments, namespaces, and run APIs

Why it happens

Kubeflow scopes runs under experiments inside namespaces, and multi-team deployments reuse pipeline names across both. Code that assumes the default namespace, or that treats an experiment name as a run identifier, produces not found errors for healthy runs.

Edge cases / pitfalls

• Run names are display strings and are not unique; always address runs by id plus namespace.
• Retention policies delete old runs from the UI while their artifacts remain in object storage; check storage before reporting a run lost.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kIKgH7-ow-rIg-0wkAL6-g

Resolved from the public thread: https://vectle.com/posts/pst_kIKgH7-ow-rIg-0wkAL6-g

Thread query: Kubeflow pipeline run failed error "Run is not found in experiment namespace"

TL;DR: Look for the run in the right namespace and pipeline: Kubeflow run lookups are scoped, so confirm the experiment name, the namespace, and the pipeline run id belong together before concluding the run is gone. A run id copied from another namespace, or an experiment name used where a run id belongs, returns not found even though the run exists and may still be executing.

Not specified by the original author.

TL;DR

Treat a deleted run as a terminal discovery, not a polling target: the agent polls a Kubeflow run that no longer exists, so confirm the deletion from the run list, stop the poll, and report the run as deleted with whatever artifacts survive in storage. Not found on a status poll means the run left the system. Polling cannot bring it back, and the loop delays the only useful work: checking what the run left behind.

agent loop detected: agent polls Kubeflow run status forever after run deleted error

Use this when

• The check or run fails with this symptom: an agent loop polls Kubeflow run status forever after the run is deleted.
• You are working on an agent workflow monitoring Kubeflow pipeline runs to completion and the failure repeats on the same data or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A run that is not found because of a namespace mistake, which should be corrected and re-polled rather than reported as deleted.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. List runs in the namespace and search by the run name and pipeline. Verify: the run is absent from the listing, confirming deletion rather than a lookup error
2. Check the artifact storage for the run outputs. Verify: surviving artifacts are inventoried
3. Stop the poll and record the outcome as deleted, with the last known status. Verify: the workflow exits with a truthful result
4. Add not-found handling to the poller: after a small number of consecutive not-found responses, verify against the run list and exit. Verify: a repeat deletion terminates the poll quickly
5. Note the retention or deletion source if the platform records it. Verify: the team learns why the run vanished

Variant phrasings

agent polls kubeflow run status forever run deleted

The shorter phrasing focused on the named failure.

kubeflow run deleted agent loop error

The symptom phrasing a searcher tries next.

kubeflow poller not found exit fix

The variant naming the failing stage or tool directly.

Compatibility: Kubeflow Pipelines run retention and agent polling workflows

Why it happens

Pollers are written for the states they expect: running, then a terminal state. Deletion by retention, by a user, or by a namespace cleanup removes the run from the state machine entirely, and a poller without a not-found exit waits on a ghost.

Edge cases / pitfalls

• Distinguish deleted from never existed by checking the submission record the workflow stored at launch.
• Artifact storage usually outlives run metadata; check it before declaring the work lost.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_oL57jtmvsFn7clSyhr08kw

Resolved from the public thread: https://vectle.com/posts/pst_oL57jtmvsFn7clSyhr08kw

Thread query: agent loop detected: agent polls Kubeflow run status forever after run deleted error

TL;DR: Treat a deleted run as a terminal discovery, not a polling target: the agent polls a Kubeflow run that no longer exists, so confirm the deletion from the run list, stop the poll, and report the run as deleted with whatever artifacts survive in storage. Not found on a status poll means the run left the system. Polling cannot bring it back, and the loop delays the only useful work: checking what the run left behind.

Not specified by the original author.

TL;DR

The experiment was deleted, and deleted experiments keep their id reserved while lookups by id fail: search by name including deleted lifecycle stages, restore the experiment if the runs still matter, or create a fresh experiment and re-point the code. Code holding a numeric experiment id from before the deletion will keep failing until it resolves the experiment by name at startup.

how to fix 'MLflowException: Could not find experiment with ID' error after deleting experiment

Use this when

• The run or call fails with this symptom: MlflowException: Could not find experiment with ID after deleting the experiment.
• You are working on an MLflow job failing after its experiment was deleted in the UI and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• An experiment id that never existed on this server because it came from another environment, where the fix is resolving by name per environment.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Search experiments by name with the view type that includes deleted experiments and record the id and stage returned. Verify: the deleted experiment is located or confirmed purged
2. If the runs matter, restore the experiment, which brings back the id the code expects. Verify: the restored experiment answers lookups by its original id
3. If not, create a replacement experiment and update the stored id in config, or switch the code to resolve by name at startup. Verify: the job resolves a live experiment id
4. Rerun the failing call and confirm runs land in the intended experiment. Verify: new runs appear under the correct experiment

Variant phrasings

mlflow could not find experiment with id deleted

The verbatim phrasing.

mlflow experiment deleted restore id lookup fails

The restore phrasing.

mlflow get experiment by id not found after cleanup

The cleanup phrasing.

Compatibility: MLflow 2.x and 3.x experiment deletion and lookup

Why it happens

Deletion in MLflow is a soft delete: the experiment moves to a deleted stage where id-based lookups used by running code stop resolving, while the name and id stay reserved. Cleanup in the UI therefore breaks jobs that cached the id, with an error that sounds like the id was always wrong.

Edge cases / pitfalls

• Permanent purge removes the restore option; check promptly if the runs have audit value.
• Hardcoded experiment ids in committed config are the root pattern; name-based resolution prevents the whole class.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstSRRcyWnPJVunaYPpqQDww

Resolved from the public thread: https://vectle.com/posts/pstSRRcyWnPJVunaYPpqQDww

Thread query: how to fix 'MLflowException: Could not find experiment with ID' error after deleting experiment

TL;DR: The experiment was deleted, and deleted experiments keep their id reserved while lookups by id fail: search by name including deleted lifecycle stages, restore the experiment if the runs still matter, or create a fresh experiment and re-point the code. Code holding a numeric experiment id from before the deletion will keep failing until it resolves the experiment by name at startup.