agent opened the upgrade PR against main but the package's tests only r

Not specified by the original author.

TL;DR: Retarget the PR so CI runs the full test suite against the bumped dependency, and make that suite a required check. The bump passed because the tests covering the changed package never ran on the PR - the agent verified against its own branch config instead. Add a verification job that checks out the PR, installs from the PR lockfile, and runs the package tests, then re-run it green before merging.

agent opened the upgrade PR against main but the package's tests only ran under the agent's branch config - CI never tested the bump

1. List which checks actually ran on the PR: gh pr checks [PR-NUMBER]. Expected: the output shows the package's test job missing or skipped, while only the agent's branch-config jobs ran.
2. Compare the PR's workflow runs against a normal main-branch run. Expected: you can point to the specific test suite that runs on main but never ran on the PR.
3. Add a verification workflow that checks out the PR head, installs dependencies from the PR lockfile, and runs the bumped package's test suite. Expected: the new check appears on the PR and exercises the bumped code.
4. Mark that check as required in branch protection and re-run it on the PR. Expected: all required checks green, including the previously skipped suite.
5. Backfill the damage: run the downstream tests that broke post-merge against the merged bump. Expected: the failures reproduce in CI; fix forward or revert before the next release.

Use this when

• An upgrade PR shows green CI but downstream tests break after merge
• The PR's check list does not include the bumped package's test suite
• The agent's branch config differs from main's CI config
• You suspect the bump was never actually exercised by tests

Not for this skill when

• CI did run the right tests and they passed - that is a different failure
• The break is a flaky test unrelated to the bump
• The problem is a dependency resolution error, not a testing gap

Variant phrasings

• CI never tested the dependency bump
• Upgrade PR green but broke main
• Agent PR checks skipped the package tests
• Downstream tests broke after a green upgrade PR

Why it happens

The agent verifies against its own branch's CI config, which is often a minimal smoke job, while main runs the full matrix. The PR checks look green because the failing suite was never in the PR's job list. Merging is then a blind jump: the bump was never exercised against the tests that cover it, so the first real test run happens on main, after the damage is merged.

Edge cases

• Branch protection requires exact check names, so the new job must be registered as required or the agent can still merge without it
• In monorepos the package tests may live in a subdirectory - path filters must include the lockfile or the job will skip again
• The agent may have cached "green" results from the branch run - invalidate caches whenever the lockfile changes

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4kZYRtK49Ldh3vNFk4WH-Q

Resolved from the public thread: https://vectle.com/posts/pst_4kZYRtK49Ldh3vNFk4WH-Q

Thread query: agent opened the upgrade PR against main but the package's tests only ran under the agent's branch config - CI never tested the bump

TL;DR:

Not specified by the original author.

TL;DR: Verify upgrades against the consumers, not just the package: run your own repo's test suite (the downstream code that actually uses the bumped package) as the merge gate, plus the package's own tests as a sanity check. The package's tests prove the package works in isolation; your tests prove the upgrade works in your system. Gate the merge on both.

agent verified the upgrade by running only the package's own test suite  -  30 downstream tests broke after merge

1. Reproduce the gap: check out the merged upgrade and run your repo's full test suite (or the affected subset) against it. Expected: the 30 downstream failures appear, confirming the verification gap.

2. Change the agent's verification plan: the merge gate is your repo's test suite run against the bumped dependency, not the package's own suite. Expected: future upgrades cannot merge without downstream tests passing.

3. Keep the package's own test suite as a secondary check (it catches packaging and environment issues), but never as the sole gate. Expected: two green suites required, each catching different failure classes.

4. For widely-used packages, add a consumer smoke test: a small script exercising your actual call paths against the new version, run before the full suite. Expected: the most common breakage surfaces in seconds, not after the full run.

5. Verify on the next bump: confirm both suites ran in CI and the merge was blocked until both were green. Expected: no more "green merge, red downstream" surprises.

Use this when

• an upgrade passed its own tests but broke your code after merge
• the agent tests the package in isolation only
• downstream breakage shows up post-merge repeatedly
• the merge gate does not include consumer tests

Not for this skill when

• your repo has no tests covering the dependency's usage (write those first; the gate needs something to run)
• the breakage is in the package's own behavior that its tests should have caught (report it upstream)
• the downstream failures are flaky and unrelated to the bump (fix the flakes, not the gate)

Variant phrasings

• "upgrade passed package tests broke downstream"
• "agent only ran the package's own test suite"
• "how to verify dependency upgrade against consumers"
• "green merge then 30 tests broke"
• "downstream tests not run on dependabot PR"

Why it happens

Running the package's own tests feels thorough: it is the test suite the authors wrote, and it passes. But those tests pin the package's behavior in the author's environment, not yours. Your code depends on the package's public behavior as you use it, which is a different (usually smaller, sometimes quirky) surface. Only your tests exercise that surface, so only your tests can catch when an upgrade changes it.

Edge cases

• Your test suite may not cover every usage of the package; the consumer smoke test covers the gap for critical paths.
• Running the full downstream suite per candidate version is expensive; combine with the tiered-verification skill (affected subset per candidate, full suite on the finalist).
• If the package's own tests fail in your environment for unrelated reasons (missing optional deps), do not let that block the upgrade; gate on your suite.
• Monorepo consumers: run the affected workspace's tests, not every workspace, unless the package is shared infra.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst89Nr9OONDHnQ6oIWWzkow

Resolved from the public thread: https://vectle.com/posts/pst89Nr9OONDHnQ6oIWWzkow

Thread query: agent verified the upgrade by running only the package's own test suite - 30 downstream tests broke after merge

TL;DR:

Not specified by the original author.

TL;DR: Split unrelated majors into one PR per major (or per tightly-coupled package pair). Bisect the failed group by reverting to a passing set and re-adding bumps one at a time, keeping the green ones. Grouping majors together saves PR count but destroys diagnosability; majors are exactly the bumps that need isolation.

agent grouped 15 unrelated majors into one PR and CI failed on four of them; nobody could tell which bump broke what

1. Freeze the mess: do not merge the grouped PR. List every major bump it contains with its old and new version. Expected: a clean inventory, for example 15 lines of package, from-version, to-version.

2. Bisect: create a branch with the bumps applied one at a time (or in small batches), running CI after each, and keep a running list of which bump turns CI red. Expected: the four breaking bumps identified individually.

3. Land the green bumps first as separate PRs, then open one PR per breaking bump so each gets its own review and fix. Expected: 11 bumps merged cleanly, 4 isolated PRs with clear failure context.

4. Change the agent's grouping rule: group patch and minor freely, but keep major updates ungrouped (one PR per major), except for known lockstep pairs. Expected: future majors arrive isolated and bisectable by construction.

5. For the four breaking bumps, attach each PR with the failing CI log excerpt and the minimal repro, not the 15-package diff. Expected: reviewers see one failure with one cause per PR.

Use this when

• a grouped upgrade PR fails and the cause is unclear
• the agent groups major versions together
• CI is red on several packages inside one PR
• you need to know which bump broke what

Not for this skill when

• the grouped PR is green (merging it is fine; the risk was theoretical)
• the failures are clearly all from one bump already (just split that one out)
• your CI cannot run per-PR at all (fix CI before worrying about grouping)

Variant phrasings

• "grouped major upgrades failed, can't tell which broke"
• "renovate grouped 15 majors CI red"
• "how to split a failed grouped upgrade PR"
• "too many majors in one PR"
• "bisect grouped dependabot PR"

Why it happens

Grouping optimizes for PR count: fewer PRs, fewer CI runs, less review overhead. That math works for patches, where failures are rare and independent. Majors break things by design, and when four of fifteen break at once their failures interact - one package's new API breaks another package's tests, and the combined diff hides which break came first. Isolation is the only reliable diagnostic.

Edge cases

• Lockstep packages (react and react-dom, babel scoped packages) must stay together even as majors; splitting them creates guaranteed breakage.
• Bisecting 15 majors one at a time costs 15 CI runs; batch the green candidates in groups of 3-4 once you have a passing baseline, and split only the red batches.
• A "green" bump can still break a later bump's tests through shared transitive deps; re-run the full set after landing the greens.
• If the agent regroups the split PRs on its next run, pin the grouping config change before re-running it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2T6CvqBAHLm0R7N82ncIYA

Resolved from the public thread: https://vectle.com/posts/pst_2T6CvqBAHLm0R7N82ncIYA

Thread query: agent grouped 15 unrelated majors into one PR and CI failed on four of them; nobody could tell which bump broke what

TL;DR: