agent opened the upgrade PR against main but the package's tests only ran under the agent's branch config - CI never...

TL;DR: Retarget the PR so CI runs the full test suite against the bumped dependency, and make that suite a required check. The bump passed because the tests covering the changed package never ran on the PR - the agent verified against its own branch config instead. Add a verification job that checks out the PR, installs from the PR lockfile, and runs the package tests, then re-run it green before merging.

agent opened the upgrade PR against main but the package's tests only ran under the agent's branch config - CI never tested the bump

1. List which checks actually ran on the PR: gh pr checks [PR-NUMBER]. Expected: the output shows the package's test job missing or skipped, while only the agent's branch-config jobs ran.
2. Compare the PR's workflow runs against a normal main-branch run. Expected: you can point to the specific test suite that runs on main but never ran on the PR.
3. Add a verification workflow that checks out the PR head, installs dependencies from the PR lockfile, and runs the bumped package's test suite. Expected: the new check appears on the PR and exercises the bumped code.
4. Mark that check as required in branch protection and re-run it on the PR. Expected: all required checks green, including the previously skipped suite.
5. Backfill the damage: run the downstream tests that broke post-merge against the merged bump. Expected: the failures reproduce in CI; fix forward or revert before the next release.

Use this when

• An upgrade PR shows green CI but downstream tests break after merge
• The PR's check list does not include the bumped package's test suite
• The agent's branch config differs from main's CI config
• You suspect the bump was never actually exercised by tests

Not for this skill when

• CI did run the right tests and they passed - that is a different failure
• The break is a flaky test unrelated to the bump
• The problem is a dependency resolution error, not a testing gap

Variant phrasings

• CI never tested the dependency bump
• Upgrade PR green but broke main
• Agent PR checks skipped the package tests
• Downstream tests broke after a green upgrade PR

Why it happens

The agent verifies against its own branch's CI config, which is often a minimal smoke job, while main runs the full matrix. The PR checks look green because the failing suite was never in the PR's job list. Merging is then a blind jump: the bump was never exercised against the tests that cover it, so the first real test run happens on main, after the damage is merged.

Edge cases

• Branch protection requires exact check names, so the new job must be registered as required or the agent can still merge without it
• In monorepos the package tests may live in a subdirectory - path filters must include the lockfile or the job will skip again
• The agent may have cached "green" results from the branch run - invalidate caches whenever the lockfile changes

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4kZYRtK49Ldh3vNFk4WH-Q