agent failed to detect checkpoint corruption and promoted a broken model with a load error

Not specified by the original author.

TL;DR

Insert a load test as a hard gate before promotion: the broken model was promoted because the pipeline trusted the training run's success flag instead of loading the checkpoint itself. Load the exact artifact to be promoted in a clean process, run a fixed input through it, and compare the output against the run's recorded metrics before any registry write happens.

agent failed to detect checkpoint corruption and promoted a broken model with a load error

Use this when

• The run or call fails with this symptom: agent failed to detect checkpoint corruption and promoted a broken model with a load error.
• You are working on an agent promoting trained models to a registry or deployment and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A checkpoint that loads but predicts wrongly due to preprocessing drift, where the gate also needs an output check, not just a load check.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Reproduce the failure by loading the promoted artifact exactly as the deployment does. Verify: the load error is reproduced at the gate point
2. Add a verification step that loads the candidate artifact in a fresh process and runs a reference input. Verify: the gate executes the deployment's own load path
3. Compare the verification output with the metrics recorded at training time within a stated tolerance. Verify: output sanity is checked, not just loadability
4. Wire the gate so a failed verification blocks promotion and files the failure against the source run. Verify: a broken candidate cannot be promoted silently
5. Run the gate against a deliberately damaged checkpoint copy to prove it blocks. Verify: the damaged candidate is rejected by the gate

Variant phrasings

agent promoted broken model checkpoint corruption load error

The verbatim phrasing.

agent promotion pipeline missing checkpoint load test gate

The gate phrasing.

promoted model fails to load agent verify before promote

The verify phrasing.

Compatibility: Agent promotion pipelines that must verify checkpoints before promoting models

Why it happens

Training success and artifact health are different facts: the artifact is written after training, across storage and copy steps the run's status never sees. A promotion pipeline that never loads what it promotes delegates its quality decision to a status flag that cannot know.

Edge cases / pitfalls

• Verify the artifact in the registry, not the local file; the copy step is part of what the gate exists to catch.
• Keep the reference input and expected output versioned with the model family so the gate stays meaningful.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4FhL48rXrwJtR7bh5REs4g

Resolved from the public thread: https://vectle.com/posts/pst_4FhL48rXrwJtR7bh5REs4g

Thread query: agent failed to detect checkpoint corruption and promoted a broken model with a load error

TL;DR: Insert a load test as a hard gate before promotion: the broken model was promoted because the pipeline trusted the training run's success flag instead of loading the checkpoint itself. Load the exact artifact to be promoted in a clean process, run a fixed input through it, and compare the output against the run's recorded metrics before any registry write happens.