agent marked a failing upgrade "cannot reproduce" because it tested in a warm docker layer cache with the old version...

TL;DR: Re-run verification with a clean build: no layer cache, no warm workspace, install strictly from the PR lockfile. The "cannot reproduce" was a cache artifact - the test container still had the old dependency installed, so the agent never tested the bump. Key the dependency layer cache on the lockfile hash so a lockfile change always invalidates it.

agent marked a failing upgrade "cannot reproduce" because it tested in a warm docker layer cache with the old version still installed

1. Confirm the cache theory: run the bumped package's version command inside the exact image the agent tested with. Expected: it reports the OLD version, not the bumped one.
2. Reproduce cleanly: rebuild with --no-cache (or delete the local install dir or virtualenv and reinstall from the PR lockfile). Expected: the originally reported failure now reproduces.
3. Record the upgrade verdict based on the clean run only. Expected: the failure is confirmed or cleared on real evidence, not cache output.
4. Fix the agent's verify step: make the dependency layer cache key include the lockfile hash. Expected: any lockfile change invalidates the cached layer and forces a fresh install.
5. Add a guard: the verify step prints the installed version of the bumped package before running tests. Expected: logs show the new version, so a stale cache is visible immediately next time.

Use this when

• An upgrade is marked "cannot reproduce" but fails for humans or CI
• Verification ran in a container, cached workspace, or reused virtualenv
• The reported failure mentions the new version but the test env may predate it
• You need cache-invalidation policy for agent verify steps

Not for this skill when

• The failure reproduces on a clean build - that is a real bug, different skill
• The tests are flaky with no cache involved
• The failure is a resolution error at install time

Variant phrasings

• Upgrade marked cannot reproduce but CI fails
• Stale docker cache hid the upgrade failure
• Warm layer cache still had the old dependency version
• Agent tested the wrong version and called it green

Why it happens

Docker layer caching keys on the Dockerfile instructions, not on the lockfile contents. If the agent reuses a warm image or workspace where dependencies were installed before the bump, the test process loads the old package. The agent sees green, concludes the reported failure is bogus, and never actually tested the bumped version at all.

Edge cases

• Multi-stage builds cache the builder stage too - bust every stage, not just the final one
• Bind-mounted install dirs from the host can re-inject old versions into a fresh container
• CI "restore cache" steps keyed on branch name instead of lockfile hash have the same bug

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_lR2Q-U1Y4FnJ33v4j0tQ0A