MLflow error: 'mlflow.exceptions.RestException: RESOURCE_DOES_NOT_EXIST' when fetching a run failed
Do not retry the fetch blindly: resolve the run id first with a search over the experiment, because a RESOURCE_DOES_NOT_EXIST on get_run almost always means the run was deleted, the id was copied from a different tracking server, or the run lives under a different experiment than the client assumes. Use when an MLflow tracking setup where a run lookup fails. Not for a run that exists but belongs to an experiment.
TL;DR
Do not retry the fetch blindly: resolve the run id first with a search over the experiment, because a RESOURCEDOESNOTEXIST on getrun almost always means the run was deleted, the id was copied from a different tracking server, or the run lives under a different experiment than the client assumes. Confirm the id exists on the server you are actually connected to, then fetch it.
MLflow error: 'mlflow.exceptions.RestException: RESOURCE_DOES_NOT_EXIST' when fetching a run failedUse this when
- The run or call fails with this symptom: mlflow.exceptions.RestException: RESOURCEDOESNOT_EXIST when fetching a run.
- You are working on an MLflow tracking setup where a run lookup fails and the failure repeats on the same run or configuration.
- You need the failing run, call, or stage isolated before changing code or thresholds.
Not for this skill when
- A run that exists but belongs to an experiment the caller has no permission to read, where the fix is access rather than id repair.
- The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.
Steps
- Print the active tracking URI in the failing process and confirm it points at the server that recorded the run. Verify: the URI matches the server where the run was logged
- Search the experiment with search_runs and look for the run id, or the run's start time and tags if the id was transcribed. Verify: the search shows whether the id exists on this server at all
- Check the run's lifecycle stage; a deleted run sits in the trash stage and plain fetches behave as if it never existed. Verify: the run is found in deleted stage or confirmed absent
- Restore the run if it was deleted, or correct the id or server in the caller, then fetch again. Verify: the fetch returns the run with its params and metrics
Variant phrasings
mlflow resource does not exist get run
The get-run phrasing.
mlflow run id not found tracking server
The wrong-server phrasing.
restexception resourcedoesnot_exist mlflow run deleted
The deleted-run phrasing.
Compatibility: MLflow 2.x and 3.x tracking servers and Python client
Why it happens
MLflow identifies runs only by id within one backend store, and the client raises the same not-exist error for a deleted run, a mistyped id, and an id from another server. Retrying the identical fetch cannot distinguish those causes, so the search-first check is what separates them.
Edge cases / pitfalls
- Ids copied from a UI URL can include the experiment path segment; strip everything except the 32 character run id.
- A run logged to a local file store will never appear on a remote server; check where the training job pointed its tracking URI.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_PbfOowY4v36rv-tIDW6FKw