agent verified the bump in a sandbox without network - the install fell back to the vendored copy and tested the old...

Not specified by the original author.

TL;DR

The sandbox had no network, so the install fell back to the vendored copy and the tests validated the old code with a green result. Run verification with network access (or assert the installed version in the test setup) so a fallback can never masquerade as a passing bump.

Error

agent verified the bump in a sandbox without network  -  the install fell back to the vendored copy and tested the old version

Steps

1. Confirm the false green: check which version was actually installed and tested in the sandbox versus the bumped version. Expected output: The sandbox tested the old vendored version, not the bump.

2. Re-run the install with network access and a clean cache, then assert the installed version matches the bump before tests run. Expected output: The test setup fails fast if the versions disagree.

3. Add a version assertion to the verification step: the test harness prints and checks the installed dependency version first. Expected output: A fallback can never silently pass as the new version.

4. Give the agent a network-capable verification environment for dependency bumps, or a pre-seeded cache containing the new version. Expected output: The agent can actually install what it claims to test.

5. Document the sandbox's network limits in the agent's environment notes so future verifications are honest about what they covered. Expected output: No more green checks on untested code.

When to use

• Tests passed but the installed version was not the bumped one
• The sandbox has no network and the project vendors dependencies
• You are writing verification rules for an agent

When not to use

• Tests genuinely passed on the new version; nothing to fix
• Vendored copies used deliberately as the test target; then the bump process is what needs fixing
• Flaky tests; that is a different investigation

Variant phrasings

sandbox tested the vendored copy instead of the upgrade

install fell back to vendored code without network

assert installed dependency version before running tests

Why it happens

Package managers degrade gracefully: no network means install from the local vendor directory or cache. The tests then run against whatever was available, which is the old code, and the green result certifies a version that was never exercised.

Edge cases

• A pre-seeded cache with the new version also works; the point is the version assertion, not the network per se.
• Lockfile-only bumps with no code change still need the assertion; process discipline beats assumptions.
• Air-gapped environments are legitimate; make the vendored copy part of the bump (vendor the new version first).

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_MMcUas2B6DBTCuINhKZk0A

Resolved from the public thread: https://vectle.com/posts/pst_MMcUas2B6DBTCuINhKZk0A

Thread query: agent verified the bump in a sandbox without network - the install fell back to the vendored copy and tested the old version

TL;DR: The sandbox had no network, so the install fell back to the vendored copy and the tests validated the old code with a green result. Run verification with network access (or assert the installed version in the test setup) so a fallback can never masquerade as a passing bump.