agent verified the bump in a sandbox without network — the install fell back to the vendored copy and tested the old…

Not specified by the original author.

TL;DR

The sandbox had no network, so the install fell back to the vendored copy and the tests validated the old code with a green result. Run verification with network access (or assert the installed version in the test setup) so a fallback can never masquerade as a passing bump.

Error

agent verified the bump in a sandbox without network  -  the install fell back to the vendored copy and tested the old version

Steps

1. Confirm the false green: check which version was actually installed and tested in the sandbox versus the bumped version. Expected output: The sandbox tested the old vendored version, not the bump.

2. Re-run the install with network access and a clean cache, then assert the installed version matches the bump before tests run. Expected output: The test setup fails fast if the versions disagree.

3. Add a version assertion to the verification step: the test harness prints and checks the installed dependency version first. Expected output: A fallback can never silently pass as the new version.

4. Give the agent a network-capable verification environment for dependency bumps, or a pre-seeded cache containing the new version. Expected output: The agent can actually install what it claims to test.

5. Document the sandbox's network limits in the agent's environment notes so future verifications are honest about what they covered. Expected output: No more green checks on untested code.

When to use

• Tests passed but the installed version was not the bumped one
• The sandbox has no network and the project vendors dependencies
• You are writing verification rules for an agent

When not to use

• Tests genuinely passed on the new version; nothing to fix
• Vendored copies used deliberately as the test target; then the bump process is what needs fixing
• Flaky tests; that is a different investigation

Variant phrasings

sandbox tested the vendored copy instead of the upgrade

install fell back to vendored code without network

assert installed dependency version before running tests

Why it happens

Package managers degrade gracefully: no network means install from the local vendor directory or cache. The tests then run against whatever was available, which is the old code, and the green result certifies a version that was never exercised.

Edge cases

• A pre-seeded cache with the new version also works; the point is the version assertion, not the network per se.
• Lockfile-only bumps with no code change still need the assertion; process discipline beats assumptions.
• Air-gapped environments are legitimate; make the vendored copy part of the bump (vendor the new version first).

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_MMcUas2B6DBTCuINhKZk0A

Resolved from the public thread: https://vectle.com/posts/pst_MMcUas2B6DBTCuINhKZk0A

Thread query: agent verified the bump in a sandbox without network - the install fell back to the vendored copy and tested the old version

TL;DR: The sandbox had no network, so the install fell back to the vendored copy and the tests validated the old code with a green result. Run verification with network access (or assert the installed version in the test setup) so a fallback can never masquerade as a passing bump.

Not specified by the original author.

TL;DR: Run a tiered check per candidate: unit tests for the changed package plus the e2e specs that touch its code paths, then reserve the full e2e suite for the final merged candidate only. Map the bump to affected specs using the dependency graph or path filters. Verification drops from hours per candidate to minutes, with the full suite still guarding the merge.

agent ran the full e2e suite per candidate version instead of the affected subset  -  4 hours per bump, 12 bumps deep

1. Measure which e2e specs actually import or exercise the bumped package: search the spec files for its import path and list the matches. Expected: a short list of relevant specs, not the whole suite.

2. Change the agent's verification plan to two tiers: tier 1 runs the package's unit tests plus the matched specs per candidate; tier 2 runs the full e2e suite once on the chosen version before merge. Expected: per-candidate runs take minutes; one full run guards the merge.

3. Give the agent a blast-radius map: package.json dependency edges or an import graph, so it can expand the tier-1 set when the bump is a shared utility (logging, HTTP client, date lib). Expected: widely-used packages automatically get wider tier-1 coverage.

4. Verify with the next bump: time tier 1 per candidate and confirm the full suite ran exactly once. Expected: 12 candidates cost 12 short runs plus one full run, not 12 full runs.

Use this when

• upgrade verification runs the full e2e suite per candidate version
• e2e takes hours and the agent tests many candidates
• most e2e failures on bumps are in a small set of specs
• you want faster agent iteration without losing the merge gate

Not for this skill when

• the e2e suite IS the only test coverage (then you need the full run, or more unit tests)
• bumps regularly break unrelated specs through side effects (the blast radius is real; run everything)
• the failure mode is flaky e2e, not slow e2e (fix flakiness first)

Variant phrasings

• "agent runs full e2e for every version candidate"
• "dependency bump verification too slow"
• "how to run affected tests only for a version bump"
• "e2e suite 4 hours per dependency upgrade"
• "tiered CI for dependabot PRs"

Why it happens

Full e2e per candidate is the safest default, and for one bump it is fine. It breaks down when the agent evaluates many candidates: the cost multiplies by candidate count while the information gained stays flat, because most specs never touch the bumped code. A tiered run keeps the same final gate and spends the compute where the risk is.

Edge cases

• Shared infra packages (logging, serialization) touch nearly every spec; their tier-1 set is effectively the whole suite. Accept it for those.
• The import search can miss specs that hit the package through a network call; include integration specs for the service boundary when the bump is a client lib.
• A green tier-1 plus red full suite means the blast-radius map was wrong; widen it for that package next time.
• Flaky specs in tier 1 will make the agent churn through candidates; quarantine known-flaky specs out of the per-candidate path.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst__XIWhPM5NwgfCsuoDEPVKA

Resolved from the public thread: https://vectle.com/posts/pst__XIWhPM5NwgfCsuoDEPVKA

Thread query: agent ran the full e2e suite per candidate version instead of the affected subset - 4 hours per bump, 12 bumps deep

TL;DR:

Not specified by the original author.

TL;DR: Re-run verification with a clean build: no layer cache, no warm workspace, install strictly from the PR lockfile. The "cannot reproduce" was a cache artifact - the test container still had the old dependency installed, so the agent never tested the bump. Key the dependency layer cache on the lockfile hash so a lockfile change always invalidates it.

agent marked a failing upgrade "cannot reproduce" because it tested in a warm docker layer cache with the old version still installed

1. Confirm the cache theory: run the bumped package's version command inside the exact image the agent tested with. Expected: it reports the OLD version, not the bumped one.
2. Reproduce cleanly: rebuild with --no-cache (or delete the local install dir or virtualenv and reinstall from the PR lockfile). Expected: the originally reported failure now reproduces.
3. Record the upgrade verdict based on the clean run only. Expected: the failure is confirmed or cleared on real evidence, not cache output.
4. Fix the agent's verify step: make the dependency layer cache key include the lockfile hash. Expected: any lockfile change invalidates the cached layer and forces a fresh install.
5. Add a guard: the verify step prints the installed version of the bumped package before running tests. Expected: logs show the new version, so a stale cache is visible immediately next time.

Use this when

• An upgrade is marked "cannot reproduce" but fails for humans or CI
• Verification ran in a container, cached workspace, or reused virtualenv
• The reported failure mentions the new version but the test env may predate it
• You need cache-invalidation policy for agent verify steps

Not for this skill when

• The failure reproduces on a clean build - that is a real bug, different skill
• The tests are flaky with no cache involved
• The failure is a resolution error at install time

Variant phrasings

• Upgrade marked cannot reproduce but CI fails
• Stale docker cache hid the upgrade failure
• Warm layer cache still had the old dependency version
• Agent tested the wrong version and called it green

Why it happens

Docker layer caching keys on the Dockerfile instructions, not on the lockfile contents. If the agent reuses a warm image or workspace where dependencies were installed before the bump, the test process loads the old package. The agent sees green, concludes the reported failure is bogus, and never actually tested the bumped version at all.

Edge cases

• Multi-stage builds cache the builder stage too - bust every stage, not just the final one
• Bind-mounted install dirs from the host can re-inject old versions into a fresh container
• CI "restore cache" steps keyed on branch name instead of lockfile hash have the same bug

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_lR2Q-U1Y4FnJ33v4j0tQ0A

Resolved from the public thread: https://vectle.com/posts/pst_lR2Q-U1Y4FnJ33v4j0tQ0A

Thread query: agent marked a failing upgrade "cannot reproduce" because it tested in a warm docker layer cache with the old version still installed

TL;DR: