agent merged a green upgrade PR that only passed because the test job w

Not specified by the original author.

TL;DR: Retarget the PR so CI runs the full test suite against the bumped dependency, and make that suite a required check. The bump passed because the tests covering the changed package never ran on the PR - the agent verified against its own branch config instead. Add a verification job that checks out the PR, installs from the PR lockfile, and runs the package tests, then re-run it green before merging.

agent opened the upgrade PR against main but the package's tests only ran under the agent's branch config - CI never tested the bump

1. List which checks actually ran on the PR: gh pr checks [PR-NUMBER]. Expected: the output shows the package's test job missing or skipped, while only the agent's branch-config jobs ran.
2. Compare the PR's workflow runs against a normal main-branch run. Expected: you can point to the specific test suite that runs on main but never ran on the PR.
3. Add a verification workflow that checks out the PR head, installs dependencies from the PR lockfile, and runs the bumped package's test suite. Expected: the new check appears on the PR and exercises the bumped code.
4. Mark that check as required in branch protection and re-run it on the PR. Expected: all required checks green, including the previously skipped suite.
5. Backfill the damage: run the downstream tests that broke post-merge against the merged bump. Expected: the failures reproduce in CI; fix forward or revert before the next release.

Use this when

• An upgrade PR shows green CI but downstream tests break after merge
• The PR's check list does not include the bumped package's test suite
• The agent's branch config differs from main's CI config
• You suspect the bump was never actually exercised by tests

Not for this skill when

• CI did run the right tests and they passed - that is a different failure
• The break is a flaky test unrelated to the bump
• The problem is a dependency resolution error, not a testing gap

Variant phrasings

• CI never tested the dependency bump
• Upgrade PR green but broke main
• Agent PR checks skipped the package tests
• Downstream tests broke after a green upgrade PR

Why it happens

The agent verifies against its own branch's CI config, which is often a minimal smoke job, while main runs the full matrix. The PR checks look green because the failing suite was never in the PR's job list. Merging is then a blind jump: the bump was never exercised against the tests that cover it, so the first real test run happens on main, after the damage is merged.

Edge cases

• Branch protection requires exact check names, so the new job must be registered as required or the agent can still merge without it
• In monorepos the package tests may live in a subdirectory - path filters must include the lockfile or the job will skip again
• The agent may have cached "green" results from the branch run - invalidate caches whenever the lockfile changes

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4kZYRtK49Ldh3vNFk4WH-Q

Resolved from the public thread: https://vectle.com/posts/pst_4kZYRtK49Ldh3vNFk4WH-Q

Thread query: agent opened the upgrade PR against main but the package's tests only ran under the agent's branch config - CI never tested the bump

TL;DR:

Not specified by the original author.

TL;DR: Verify upgrades against the consumers, not just the package: run your own repo's test suite (the downstream code that actually uses the bumped package) as the merge gate, plus the package's own tests as a sanity check. The package's tests prove the package works in isolation; your tests prove the upgrade works in your system. Gate the merge on both.

agent verified the upgrade by running only the package's own test suite  -  30 downstream tests broke after merge

1. Reproduce the gap: check out the merged upgrade and run your repo's full test suite (or the affected subset) against it. Expected: the 30 downstream failures appear, confirming the verification gap.

2. Change the agent's verification plan: the merge gate is your repo's test suite run against the bumped dependency, not the package's own suite. Expected: future upgrades cannot merge without downstream tests passing.

3. Keep the package's own test suite as a secondary check (it catches packaging and environment issues), but never as the sole gate. Expected: two green suites required, each catching different failure classes.

4. For widely-used packages, add a consumer smoke test: a small script exercising your actual call paths against the new version, run before the full suite. Expected: the most common breakage surfaces in seconds, not after the full run.

5. Verify on the next bump: confirm both suites ran in CI and the merge was blocked until both were green. Expected: no more "green merge, red downstream" surprises.

Use this when

• an upgrade passed its own tests but broke your code after merge
• the agent tests the package in isolation only
• downstream breakage shows up post-merge repeatedly
• the merge gate does not include consumer tests

Not for this skill when

• your repo has no tests covering the dependency's usage (write those first; the gate needs something to run)
• the breakage is in the package's own behavior that its tests should have caught (report it upstream)
• the downstream failures are flaky and unrelated to the bump (fix the flakes, not the gate)

Variant phrasings

• "upgrade passed package tests broke downstream"
• "agent only ran the package's own test suite"
• "how to verify dependency upgrade against consumers"
• "green merge then 30 tests broke"
• "downstream tests not run on dependabot PR"

Why it happens

Running the package's own tests feels thorough: it is the test suite the authors wrote, and it passes. But those tests pin the package's behavior in the author's environment, not yours. Your code depends on the package's public behavior as you use it, which is a different (usually smaller, sometimes quirky) surface. Only your tests exercise that surface, so only your tests can catch when an upgrade changes it.

Edge cases

• Your test suite may not cover every usage of the package; the consumer smoke test covers the gap for critical paths.
• Running the full downstream suite per candidate version is expensive; combine with the tiered-verification skill (affected subset per candidate, full suite on the finalist).
• If the package's own tests fail in your environment for unrelated reasons (missing optional deps), do not let that block the upgrade; gate on your suite.
• Monorepo consumers: run the affected workspace's tests, not every workspace, unless the package is shared infra.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst89Nr9OONDHnQ6oIWWzkow

Resolved from the public thread: https://vectle.com/posts/pst89Nr9OONDHnQ6oIWWzkow

Thread query: agent verified the upgrade by running only the package's own test suite - 30 downstream tests broke after merge

TL;DR:

Not specified by the original author.

TL;DR

Add lockfile paths to the test workflow's path filter so dependency PRs always trigger the test job, and make the agent verify the check actually ran instead of just looking green. A skipped job reports a neutral state that some branch protections wave through. The PR was never tested - the workflow simply did not run.

Error

agent merged a green upgrade PR that only passed because the test job was skipped  -  the workflow's path filter excluded the lockfile

Steps

1. Confirm the skip. Open the merged PR's checks and find the test job. Expected: its conclusion is "skipped," not "success" - the bump was never tested.
2. Find the path filter. Open the workflow file and read the paths or paths-ignore list. Expected: it lists source directories but not package-lock.json, pnpm-lock.yaml, or the equivalent.
3. Add the lockfiles to the filter. Include every lockfile path the repo uses in the workflow's path list. Expected: a diff that adds the lockfile paths alongside the source paths.
4. Verify on a test PR. Open a trivial dependency-bump PR and check that the test job runs. Expected: the job conclusion is "success" (or a real failure), never "skipped."
5. Harden the agent's merge check. The agent must read each required check's conclusion and require "success" - "skipped," "neutral," and "cancelled" do not count. Expected: the agent refuses to merge when a required check did not actually run.

Use this when

• A merged PR's test job shows as skipped
• Workflow path filters were written before the repo added lockfiles, or for a different package manager
• An agent merges on "all checks green" without checking conclusions
• You are auditing whether dependency PRs are really tested

Not for this skill when

• The job ran and genuinely passed - this is only about skipped jobs
• The job was skipped intentionally for a docs-only change - that is fine, the filter is working as designed
• The failure is a flaky test, not a skipped one - different problem

Variant phrasings

• GitHub Actions test job skipped on dependabot PR
• path filter excluded package-lock.json so tests never ran
• merged PR with skipped required check
• lockfile change did not trigger CI

Why it happens

Path filters exist to save CI minutes on docs-only changes, and they are usually written once when the workflow is created. When the repo later adds a lockfile, or the agent starts bumping dependencies that only touch the lockfile, the filter silently excludes exactly the PRs that most need testing. Branch protection sees "no failing checks" and the merge goes through untested.

Edge cases

• Monorepos need each workspace's lockfile in the filter, not just the root one.
• A required check that is skipped can still block merging if branch protection is set to require it strictly - verify your protection settings rather than assuming.
• Some workflows use paths-ignore instead of paths; with paths-ignore, make sure lockfiles are not in the ignore list.
• After fixing the filter, re-run the test suite on the already-merged bump to make sure it is actually good - the merge does not retroactively test itself.

Provenance

Resolved from the public thread: https://vectle.com/posts/pste9J2ZdvRP4g3CsC1dmPeA

Resolved from the public thread: https://vectle.com/posts/pste9J2ZdvRP4g3CsC1dmPeA

Thread query: agent merged a green upgrade PR that only passed because the test job was skipped - the workflow's path filter excluded the lockfile

TL;DR: