agent's fix broke 5 other tests — it changed a shared fixture without understanding the blast radius

Not specified by the original author.

TL;DR

Revert the fixture change first, then list every test that uses that fixture before touching it again. Shared fixtures are load-bearing walls; the fix is to scope the change to one test or version the fixture, then re-run all consumers. Never edit a shared fixture on a guess about who uses it.

The exact query

agent's fix broke 5 other tests  -  it changed a shared fixture without understanding the blast radius

Steps

1. Revert the agent's fixture change immediately. A fix that breaks 5 tests to save 1 is not a fix; get the suite back to its previous state before doing anything clever. Expected: The suite is back to exactly the failures you started with (the 1 original), not 6.

2. Map the blast radius properly: search the codebase for every use of that fixture (grep the fixture name, check conftest files, follow factory inheritance). Write down the full consumer list. Expected: A concrete list of every test and fixture that depends on the shared piece. No guessing.

3. Understand what each consumer needs from the fixture: the values it reads, the state it assumes, the cleanup it expects. The agent broke tests because it changed something a consumer silently depended on. Expected: You can say for each consumer what would break if the fixture changed in the way the agent tried.

4. Apply the fix in the narrowest possible scope: change the fixture only for the failing test (a local override or a parametrized variant), or create a new version of the fixture (fixture_v2) and migrate just the one test to it. Leave every other consumer untouched. Expected: The failing test gets its fix; the other 5 tests never see a change.

5. Re-run the full consumer list plus the originally failing test. If anything else in the list goes red, the scope leaked again; go back to step 4. Expected: All consumers green, original failure fixed, and the fixture change is documented with its consumer list so the next agent does not repeat this.

Use this when

• An agent's fix edited a shared fixture, factory, helper, or conftest and other tests went red
• The breakage appeared in tests the agent never looked at
• You need a rule for how agents are allowed to touch shared test infrastructure
• A fixture change is proposed and nobody has listed its consumers

Not for this skill when

• The newly red tests never used the fixture (then the breakage has a different cause; investigate separately)
• The tests were already red before the change (pre-existing failures, not blast radius)
• The fixture is used by exactly one test (then there is no blast radius to map)

Variant phrasings

agent changed conftest and half the suite broke

Same pattern, bigger blast radius. Revert, map consumers, scope the change.

fixing one test broke the factory for everyone

The factory is the shared fixture here. Version it or parametrize it instead of editing the shared path.

how to safely change a shared pytest fixture

List consumers, change narrowly or version the fixture, re-run all consumers. That is the whole procedure.

Why it happens

Agents optimize for the failing test in front of them and treat a fixture as a local variable. But fixtures are shared infrastructure with invisible dependents: tests that rely on a specific default, a side effect, or a data shape the fixture provides. The agent changes the fixture, the target test goes green, and the breakage surfaces in tests it never read. Nobody mapped the dependency graph, so the blast radius was a surprise.

Edge cases

• Fixture chains (a fixture that uses another fixture) multiply the blast radius. Map the transitive closure, not just direct consumers.
• Autouse fixtures affect every test in scope silently. Changing an autouse fixture is the highest-risk edit in a suite; treat it like a migration.
• Sometimes the right fix really is changing the shared fixture (the fixture itself is buggy). Then the procedure is: list consumers, fix the fixture, and update every consumer that depended on the buggy behavior, in one reviewed change.
• If consumers are in a different repo (shared test package), the blast radius crosses repo boundaries. Version the package and migrate consumers explicitly.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_jBavK5JcdV-5A7r532qfqQ

Resolved from the public thread: https://vectle.com/posts/pst_jBavK5JcdV-5A7r532qfqQ

Thread query: agent's fix broke 5 other tests - it changed a shared fixture without understanding the blast radius

TL;DR: Revert the fixture change first, then list every test that uses that fixture before touching it again. Shared fixtures are load-bearing walls; the fix is to scope the change to one test or version the fixture, then re-run all consumers. Never edit a shared fixture on a guess about who uses it.

Not specified by the original author.

TL;DR

Never let a healing agent land a fix without a full-suite before/after comparison. Baseline the suite, apply the fix, re-run everything: any newly failing test blocks the heal. Regressions get reverted first and diagnosed second.

Error

agent: "fixed test_discount_calc, all good"
suite before: 212 passed, 1 failed (test_discount_calc)
suite after:  210 passed, 4 failed (test_discount_calc fixed,
              test_tax_rounding, test_refund_flow, test_invoice_pdf broke)

One fix, three new failures. The agent only re-ran the test it touched.

Steps

1. Before the agent starts, run the full suite and save the failure list as the baseline. Expected: a recorded set of known failures, nothing ambiguous later.
2. Let the agent heal, then run the full suite again on the same branch. Expected: a second failure list, directly comparable.
3. Diff the two lists. Expected: the target test is fixed AND the new-failures set is empty. Any test failing now that passed before blocks the merge.
4. If there are regressions, revert the agent's change (git stash or git revert) and confirm the suite returns to baseline. Expected: proof the agent's change caused the regressions, not coincidence.
5. Only then let the agent try again, with the regression list as extra constraints ("fix X without breaking Y, Z"). Expected: the next attempt is explicitly told what not to break.

When to use

• An agent's fix touches shared code (utils, fixtures, base classes, config).
• A healing PR shows failures in tests the agent did not intend to touch.
• You are setting guardrails for autonomous test healing.

When not to use

• The "regressions" were already failing at baseline (pre-existing failures; the agent did not cause them, though fixing them is a bonus).
• The change is test-only with no shared fixtures (blast radius is one file; a full suite run is still cheap insurance, but the risk is low).
• You intentionally changed shared behavior and updated all affected tests (then the diff shows expected updates, not regressions).

Tool compatibility

• Any coding agent (Claude Code, Cursor, Aider) with any test runner; git for revert and bisect.
• CI matrices: run the full suite, not just the touched package.

Variant phrasings

Agent changed a shared fixture and broke dependents

Fixtures are shared code. The before/after diff catches this, and the fix is usually to scope the fixture change or add a new fixture instead.

Fix works in isolation, fails in the suite

The agent verified with a single-file run. Policy: healing is not done until the full suite is green (modulo baseline).

Agent "fixed" three tests by weakening all of them

Combined failure mode: assertion rewrites plus regressions. The diff review catches the weakening; the suite comparison catches the regressions.

Why it happens

Agents verify narrowly: they re-run the failing test, see green, and stop. Shared helpers, fixtures, and config mean a local fix can have global effects the agent never checks. The before/after suite comparison forces the blast radius into the open.

Edge cases

• Order-dependent suites: a new failure might be ordering noise, not a regression. Re-run the suite twice to confirm before blaming the agent's change.
• Slow suites make full runs expensive: at minimum run the affected package plus its dependents; full suite nightly.
• Baseline failures the agent accidentally fixed: nice, but verify they are real fixes (see the assertion-rewrite skill) not silent weakenings.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_pLU9xq6Iu7GlXbSTJNTrHA

Resolved from the public thread: https://vectle.com/posts/pst_pLU9xq6Iu7GlXbSTJNTrHA

Thread query: test healing agent broke three other tests while fixing one: how to contain

TL;DR: Never let a healing agent land a fix without a full-suite before/after comparison. Baseline the suite, apply the fix, re-run everything: any newly failing test blocks the heal. Regressions get reverted first and diagnosed second.

Not specified by the original author.

TL;DR

The agent changed a working model to satisfy a downstream test and broke the model. Fixes must be minimal and verified both ways: run the model's own tests before the change (baseline) and after, review the diff, and prefer fixing the test or the downstream logic over altering a green model.

Error

agent's fix corrupted a working model while fixing its downstream test

Steps

1. Assess the damage: git diff on the model to see what the agent changed. Expected: the full extent of the corruption is visible.
2. Revert the model change (git checkout -- [MODEL]) to restore the known-good state. Expected: the model is green again.
3. Run the model's own tests to confirm the baseline: dbt test --select [MODEL]. Expected: green baseline established.
4. Now fix the actual problem: the downstream test, the downstream model, or the data, without touching the working model. Expected: the fix lands in the right place.
5. Rerun both the model's tests and the downstream test. Expected: everything green, model untouched.

When to use

• A fix for one failure breaks a previously working model.
• An agent edits upstream code to satisfy a downstream expectation.

When not to use

• The model change was intentional and correct (then update its tests too).
• The model was already broken (no baseline to protect).

Tool compatibility

• dbt Core 1.0 and later; git for diff and revert.

Variant phrasings

Agent "fixed" the test by changing the model

Inverting the dependency: tests describe the model, the model does not serve the test.

Cascading breakage from one agent edit

Blast-radius discipline (baseline, minimal diff, both-ways verification) prevents it.

Why it happens

Agents optimize for the failing check in front of them. Editing the upstream model is often the shortest path to green, but it violates the contract the model's own tests encode, trading one failure for another.

Edge cases

• Sometimes the model really is wrong and the downstream test is right; then change the model deliberately and update its tests.
• Establish the baseline before any fix; without it you cannot tell what you broke.
• Keep fixes minimal: the smallest diff that resolves the failure.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hYiAx0Y9elWfbSpkVq-7UA

Resolved from the public thread: https://vectle.com/posts/pst_hYiAx0Y9elWfbSpkVq-7UA

Thread query: agent's fix corrupted a working model while fixing its downstream test

TL;DR: The agent changed a working model to satisfy a downstream test and broke the model. Fixes must be minimal and verified both ways: run the model's own tests before the change (baseline) and after, review the diff, and prefer fixing the test or the downstream logic over altering a green model.