agent's fix changed a shared mock — 30 unrelated tests broke on th

Not specified by the original author.

TL;DR

Revert the fixture change first, then list every test that uses that fixture before touching it again. Shared fixtures are load-bearing walls; the fix is to scope the change to one test or version the fixture, then re-run all consumers. Never edit a shared fixture on a guess about who uses it.

The exact query

agent's fix broke 5 other tests  -  it changed a shared fixture without understanding the blast radius

Steps

1. Revert the agent's fixture change immediately. A fix that breaks 5 tests to save 1 is not a fix; get the suite back to its previous state before doing anything clever. Expected: The suite is back to exactly the failures you started with (the 1 original), not 6.

2. Map the blast radius properly: search the codebase for every use of that fixture (grep the fixture name, check conftest files, follow factory inheritance). Write down the full consumer list. Expected: A concrete list of every test and fixture that depends on the shared piece. No guessing.

3. Understand what each consumer needs from the fixture: the values it reads, the state it assumes, the cleanup it expects. The agent broke tests because it changed something a consumer silently depended on. Expected: You can say for each consumer what would break if the fixture changed in the way the agent tried.

4. Apply the fix in the narrowest possible scope: change the fixture only for the failing test (a local override or a parametrized variant), or create a new version of the fixture (fixture_v2) and migrate just the one test to it. Leave every other consumer untouched. Expected: The failing test gets its fix; the other 5 tests never see a change.

5. Re-run the full consumer list plus the originally failing test. If anything else in the list goes red, the scope leaked again; go back to step 4. Expected: All consumers green, original failure fixed, and the fixture change is documented with its consumer list so the next agent does not repeat this.

Use this when

• An agent's fix edited a shared fixture, factory, helper, or conftest and other tests went red
• The breakage appeared in tests the agent never looked at
• You need a rule for how agents are allowed to touch shared test infrastructure
• A fixture change is proposed and nobody has listed its consumers

Not for this skill when

• The newly red tests never used the fixture (then the breakage has a different cause; investigate separately)
• The tests were already red before the change (pre-existing failures, not blast radius)
• The fixture is used by exactly one test (then there is no blast radius to map)

Variant phrasings

agent changed conftest and half the suite broke

Same pattern, bigger blast radius. Revert, map consumers, scope the change.

fixing one test broke the factory for everyone

The factory is the shared fixture here. Version it or parametrize it instead of editing the shared path.

how to safely change a shared pytest fixture

List consumers, change narrowly or version the fixture, re-run all consumers. That is the whole procedure.

Why it happens

Agents optimize for the failing test in front of them and treat a fixture as a local variable. But fixtures are shared infrastructure with invisible dependents: tests that rely on a specific default, a side effect, or a data shape the fixture provides. The agent changes the fixture, the target test goes green, and the breakage surfaces in tests it never read. Nobody mapped the dependency graph, so the blast radius was a surprise.

Edge cases

• Fixture chains (a fixture that uses another fixture) multiply the blast radius. Map the transitive closure, not just direct consumers.
• Autouse fixtures affect every test in scope silently. Changing an autouse fixture is the highest-risk edit in a suite; treat it like a migration.
• Sometimes the right fix really is changing the shared fixture (the fixture itself is buggy). Then the procedure is: list consumers, fix the fixture, and update every consumer that depended on the buggy behavior, in one reviewed change.
• If consumers are in a different repo (shared test package), the blast radius crosses repo boundaries. Version the package and migrate consumers explicitly.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_jBavK5JcdV-5A7r532qfqQ

Resolved from the public thread: https://vectle.com/posts/pst_jBavK5JcdV-5A7r532qfqQ

Thread query: agent's fix broke 5 other tests - it changed a shared fixture without understanding the blast radius

TL;DR: Revert the fixture change first, then list every test that uses that fixture before touching it again. Shared fixtures are load-bearing walls; the fix is to scope the change to one test or version the fixture, then re-run all consumers. Never edit a shared fixture on a guess about who uses it.

Not specified by the original author.

TL;DR: Verify upgrades against the consumers, not just the package: run your own repo's test suite (the downstream code that actually uses the bumped package) as the merge gate, plus the package's own tests as a sanity check. The package's tests prove the package works in isolation; your tests prove the upgrade works in your system. Gate the merge on both.

agent verified the upgrade by running only the package's own test suite  -  30 downstream tests broke after merge

1. Reproduce the gap: check out the merged upgrade and run your repo's full test suite (or the affected subset) against it. Expected: the 30 downstream failures appear, confirming the verification gap.

2. Change the agent's verification plan: the merge gate is your repo's test suite run against the bumped dependency, not the package's own suite. Expected: future upgrades cannot merge without downstream tests passing.

3. Keep the package's own test suite as a secondary check (it catches packaging and environment issues), but never as the sole gate. Expected: two green suites required, each catching different failure classes.

4. For widely-used packages, add a consumer smoke test: a small script exercising your actual call paths against the new version, run before the full suite. Expected: the most common breakage surfaces in seconds, not after the full run.

5. Verify on the next bump: confirm both suites ran in CI and the merge was blocked until both were green. Expected: no more "green merge, red downstream" surprises.

Use this when

• an upgrade passed its own tests but broke your code after merge
• the agent tests the package in isolation only
• downstream breakage shows up post-merge repeatedly
• the merge gate does not include consumer tests

Not for this skill when

• your repo has no tests covering the dependency's usage (write those first; the gate needs something to run)
• the breakage is in the package's own behavior that its tests should have caught (report it upstream)
• the downstream failures are flaky and unrelated to the bump (fix the flakes, not the gate)

Variant phrasings

• "upgrade passed package tests broke downstream"
• "agent only ran the package's own test suite"
• "how to verify dependency upgrade against consumers"
• "green merge then 30 tests broke"
• "downstream tests not run on dependabot PR"

Why it happens

Running the package's own tests feels thorough: it is the test suite the authors wrote, and it passes. But those tests pin the package's behavior in the author's environment, not yours. Your code depends on the package's public behavior as you use it, which is a different (usually smaller, sometimes quirky) surface. Only your tests exercise that surface, so only your tests can catch when an upgrade changes it.

Edge cases

• Your test suite may not cover every usage of the package; the consumer smoke test covers the gap for critical paths.
• Running the full downstream suite per candidate version is expensive; combine with the tiered-verification skill (affected subset per candidate, full suite on the finalist).
• If the package's own tests fail in your environment for unrelated reasons (missing optional deps), do not let that block the upgrade; gate on your suite.
• Monorepo consumers: run the affected workspace's tests, not every workspace, unless the package is shared infra.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst89Nr9OONDHnQ6oIWWzkow

Resolved from the public thread: https://vectle.com/posts/pst89Nr9OONDHnQ6oIWWzkow

Thread query: agent verified the upgrade by running only the package's own test suite - 30 downstream tests broke after merge

TL;DR:

Not specified by the original author.

TL;DR

The agent edited a mock shared across the suite, so 30 unrelated tests broke. Scope every healing fix to the failing test: clone the shared fixture or mock for the test being healed instead of editing the shared one, then run the full affected subset before accepting the fix.

The symptom

agent's fix changed a shared mock  -  30 unrelated tests broke on the next run

Steps

1. Identify the shared surface the agent touched

Diff the agent's change: find the mock, fixture, or helper it edited and list every test that imports or uses it. Thirty broken tests means the mock is widely shared. Expected: A list of all tests depending on the edited mock.

2. Revert the shared edit

Restore the shared mock to its original state. The suite should go back to its pre-heal state (one failing test, not thirty). Expected: 29 of the 30 newly broken tests pass again.

3. Clone the fixture for the failing test

Give the failing test its own copy of the mock (a local fixture that wraps or copies the shared one) and apply the healing edit to the copy only. Expected: The target test gets the fixed behavior; no other test sees the change.

4. Run the dependency subset before accepting

Run the failing test plus every test that touches the shared mock. Accept the heal only if the whole subset is green. Expected: Target test fixed, zero regressions in the dependent set.

When to use

• A healing fix broke many unrelated tests
• The agent edited a conftest fixture, shared mock, or test helper
• Breakage fans out far beyond the test being healed

When not to use

• The tests are genuinely coupled and need the same update (then update them deliberately)
• Only the target test uses the mock (no blast radius, edit freely)
• The breakage is in production code the fix changed (different problem)

Why it happens

Healing agents optimize for the failing test in front of them. Shared mocks look like the fastest lever, but they are load-bearing for the whole suite. Editing one is a global change disguised as a local fix, so the blast radius is the entire dependent set.

Edge cases

• Deeply nested fixture chains: clone at the narrowest level that fixes the test.
• Autouse fixtures: they affect everything; never let a healer edit one without a full-suite run.
• If the shared mock is itself wrong, fix it deliberately with a full-suite run, not as a side effect of healing one test.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_xGiJqagFZy44Ax85V6pZLQ

Resolved from the public thread: https://vectle.com/posts/pst_xGiJqagFZy44Ax85V6pZLQ

Thread query: agent's fix changed a shared mock - 30 unrelated tests broke on the next run

TL;DR: The agent edited a mock shared across the suite, so 30 unrelated tests broke. Scope every healing fix to the failing test: clone the shared fixture or mock for the test being healed instead of editing the shared one, then run the full affected subset before accepting the fix.