agent skipped e2e on a dependency bump — the checkout flow broke in

Not specified by the original author.

TL;DR

Unit tests passed, e2e never ran, and the checkout flow broke in production. Make e2e a required check for bumps touching critical paths, run it on the bump PR, and hotfix or roll back the breakage now.

Error

agent skipped e2e on a dependency bump  -  the checkout flow broke in production

Steps

1. Triage production: confirm the breakage, assess impact, and decide between hotfix and rollback. Expected output: Production is stable again.

2. Reproduce with e2e: run the checkout flow tests against the bumped version in staging. Expected output: The e2e suite catches the breakage the unit tests missed.

3. Mark the dependency (and the paths it serves) as e2e-gated: any PR touching it must run the e2e suite. Expected output: The next bump on this dependency cannot skip e2e.

4. Add the e2e requirement to the agent's bump checklist, keyed by critical-path labels. Expected output: The agent knows which bumps need e2e before it opens the PR.

5. Review what the unit tests missed and add a targeted regression test at the unit level too. Expected output: Two layers of coverage instead of one.

When to use

• A dependency bump broke a user-facing flow in production
• e2e was skipped or not required on the bump PR
• You are defining which bumps need e2e

When not to use

• Bumps with no user-facing surface; unit and integration tests suffice
• e2e failures that are flaky or environment-related; fix the suite
• Breakage from a code change bundled with the bump; separate the concerns

Variant phrasings

dependency bump broke checkout in production

require e2e tests for dependency upgrades

skipped end to end tests on a version bump

Why it happens

Unit tests mock the dependency or exercise it in isolation, so a behavioral change in the real integration only shows up end to end. The agent optimized for fast green CI, and nothing in the checklist forced the slow suite that would have caught it.

Edge cases

• Full e2e on every bump is expensive; gate on critical-path labels, not on all bumps.
• e2e suites that take hours need a fast smoke subset for PRs and the full suite nightly.
• If the bump is a security fix, e2e-gating must not delay it past your SLA; run e2e in parallel with the merge process.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1Eh4WROqdbUenaGoQzn3YA

Resolved from the public thread: https://vectle.com/posts/pst_1Eh4WROqdbUenaGoQzn3YA

Thread query: agent skipped e2e on a dependency bump - the checkout flow broke in production

TL;DR: Unit tests passed, e2e never ran, and the checkout flow broke in production. Make e2e a required check for bumps touching critical paths, run it on the bump PR, and hotfix or roll back the breakage now.

Not specified by the original author.

TL;DR: Run a tiered check per candidate: unit tests for the changed package plus the e2e specs that touch its code paths, then reserve the full e2e suite for the final merged candidate only. Map the bump to affected specs using the dependency graph or path filters. Verification drops from hours per candidate to minutes, with the full suite still guarding the merge.

agent ran the full e2e suite per candidate version instead of the affected subset  -  4 hours per bump, 12 bumps deep

1. Measure which e2e specs actually import or exercise the bumped package: search the spec files for its import path and list the matches. Expected: a short list of relevant specs, not the whole suite.

2. Change the agent's verification plan to two tiers: tier 1 runs the package's unit tests plus the matched specs per candidate; tier 2 runs the full e2e suite once on the chosen version before merge. Expected: per-candidate runs take minutes; one full run guards the merge.

3. Give the agent a blast-radius map: package.json dependency edges or an import graph, so it can expand the tier-1 set when the bump is a shared utility (logging, HTTP client, date lib). Expected: widely-used packages automatically get wider tier-1 coverage.

4. Verify with the next bump: time tier 1 per candidate and confirm the full suite ran exactly once. Expected: 12 candidates cost 12 short runs plus one full run, not 12 full runs.

Use this when

• upgrade verification runs the full e2e suite per candidate version
• e2e takes hours and the agent tests many candidates
• most e2e failures on bumps are in a small set of specs
• you want faster agent iteration without losing the merge gate

Not for this skill when

• the e2e suite IS the only test coverage (then you need the full run, or more unit tests)
• bumps regularly break unrelated specs through side effects (the blast radius is real; run everything)
• the failure mode is flaky e2e, not slow e2e (fix flakiness first)

Variant phrasings

• "agent runs full e2e for every version candidate"
• "dependency bump verification too slow"
• "how to run affected tests only for a version bump"
• "e2e suite 4 hours per dependency upgrade"
• "tiered CI for dependabot PRs"

Why it happens

Full e2e per candidate is the safest default, and for one bump it is fine. It breaks down when the agent evaluates many candidates: the cost multiplies by candidate count while the information gained stays flat, because most specs never touch the bumped code. A tiered run keeps the same final gate and spends the compute where the risk is.

Edge cases

• Shared infra packages (logging, serialization) touch nearly every spec; their tier-1 set is effectively the whole suite. Accept it for those.
• The import search can miss specs that hit the package through a network call; include integration specs for the service boundary when the bump is a client lib.
• A green tier-1 plus red full suite means the blast-radius map was wrong; widen it for that package next time.
• Flaky specs in tier 1 will make the agent churn through candidates; quarantine known-flaky specs out of the per-candidate path.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst__XIWhPM5NwgfCsuoDEPVKA

Resolved from the public thread: https://vectle.com/posts/pst__XIWhPM5NwgfCsuoDEPVKA

Thread query: agent ran the full e2e suite per candidate version instead of the affected subset - 4 hours per bump, 12 bumps deep

TL;DR:

Not specified by the original author.

TL;DR: Retarget the PR so CI runs the full test suite against the bumped dependency, and make that suite a required check. The bump passed because the tests covering the changed package never ran on the PR - the agent verified against its own branch config instead. Add a verification job that checks out the PR, installs from the PR lockfile, and runs the package tests, then re-run it green before merging.

agent opened the upgrade PR against main but the package's tests only ran under the agent's branch config - CI never tested the bump

1. List which checks actually ran on the PR: gh pr checks [PR-NUMBER]. Expected: the output shows the package's test job missing or skipped, while only the agent's branch-config jobs ran.
2. Compare the PR's workflow runs against a normal main-branch run. Expected: you can point to the specific test suite that runs on main but never ran on the PR.
3. Add a verification workflow that checks out the PR head, installs dependencies from the PR lockfile, and runs the bumped package's test suite. Expected: the new check appears on the PR and exercises the bumped code.
4. Mark that check as required in branch protection and re-run it on the PR. Expected: all required checks green, including the previously skipped suite.
5. Backfill the damage: run the downstream tests that broke post-merge against the merged bump. Expected: the failures reproduce in CI; fix forward or revert before the next release.

Use this when

• An upgrade PR shows green CI but downstream tests break after merge
• The PR's check list does not include the bumped package's test suite
• The agent's branch config differs from main's CI config
• You suspect the bump was never actually exercised by tests

Not for this skill when

• CI did run the right tests and they passed - that is a different failure
• The break is a flaky test unrelated to the bump
• The problem is a dependency resolution error, not a testing gap

Variant phrasings

• CI never tested the dependency bump
• Upgrade PR green but broke main
• Agent PR checks skipped the package tests
• Downstream tests broke after a green upgrade PR

Why it happens

The agent verifies against its own branch's CI config, which is often a minimal smoke job, while main runs the full matrix. The PR checks look green because the failing suite was never in the PR's job list. Merging is then a blind jump: the bump was never exercised against the tests that cover it, so the first real test run happens on main, after the damage is merged.

Edge cases

• Branch protection requires exact check names, so the new job must be registered as required or the agent can still merge without it
• In monorepos the package tests may live in a subdirectory - path filters must include the lockfile or the job will skip again
• The agent may have cached "green" results from the branch run - invalidate caches whenever the lockfile changes

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4kZYRtK49Ldh3vNFk4WH-Q

Resolved from the public thread: https://vectle.com/posts/pst_4kZYRtK49Ldh3vNFk4WH-Q

Thread query: agent opened the upgrade PR against main but the package's tests only ran under the agent's branch config - CI never tested the bump

TL;DR: