agent merged duplicate tests while deduplicating the suite: how to detect
Detects when a deduplicating agent merged tests that were not actually duplicates by diffing their assertions and behavior, not just their names. Use when a suite-shrinking pass leaves fewer tests but weaker coverage. Not for intentional test consolidation that a human reviewed and approved with the assertion mapping documented.
TL;DR
An agent "deduplicating" a suite can merge two tests that look alike but assert different things, silently deleting coverage. Detect it by diffing assertions and behavior between the merged tests, not titles. Require every dedup to carry an assertion-preservation check: the merged test must prove everything the originals proved.
Error
agent merged duplicate tests while deduplicating the suite: how to detectAfter the dedup pass the suite is smaller, but a regression slips through that one of the deleted tests used to catch.
Steps
- List what the agent merged: every deleted test and the surviving test it was folded into. Expected output: a merge map, deleted to survivor.
- For each merge, diff the assertions: list what the deleted test asserted that the survivor does not. Expected output: a concrete list of lost assertions per merge.
- Check behavior, not names: run the deleted tests standalone against a build with the relevant bug reintroduced, and confirm the survivor catches it too. Expected output: proof of preserved coverage, or proof of a gap.
- Restore or re-split any merge that lost assertions. Expected output: the lost coverage back in the suite, as separate tests or an expanded survivor.
- Add the assertion-preservation check to the dedup policy so future passes cannot merge without it. Expected output: a standing gate on the dedup lane.
When to use
- A dedup pass shrank the suite and regressions started slipping through.
- You suspect an agent merged tests by title similarity.
- You want a review gate before any automated dedup lands.
When not to use
- A human reviewed and approved the consolidation with the assertion mapping written down.
- The merged tests were byte-identical copies (true duplicates; merging is safe).
- The suite is being rewritten deliberately as part of a refactor with its own review.
Variant phrasings
Agent deleted "redundant" edge-case tests
Edge cases that look redundant often guard different branches. Same assertion-diff check applies.
Agent merged parameterized cases into one
Folding ten parameter sets into one happy path deletes nine cases. Require the merged test to keep every parameter set.
Two tests with the same name in different files got merged
Name collision is not duplication. Scope dedup comparisons within the same behavior, not the same title.
Why it happens
The agent scores dedup by suite size: fewer tests looks like success. Title and structure similarity are cheap signals, and actually verifying behavioral equivalence is work the agent is not asked to do. So near-duplicates get merged and the differences, which were the point, get deleted.
Edge cases
- Tests that differ only in setup data can be safely merged by parameterizing; the gate should allow merges that preserve the data matrix.
- Snapshot tests look identical but cover different states; never merge those without human review.
- Flaky tests merged with stable ones hide the flake inside a bigger test; quarantine flakes before any dedup pass.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_GjNKYB1uC8x4B-W8HEV8xQ