agent reordered test files alphabetically while healing: import breakage fix

TL;DR

Reordering test files "for tidiness" during healing breaks imports, fixtures, and anything that depends on file layout, and it pollutes the diff. Revert the reorder, fix the actual failure, and forbid file renames and moves in the healing lane. Tidiness is a separate task with its own review.

Error

agent reordered test files alphabetically while healing: import breakage fix

The healing patch renamed test files into alphabetical order; relative imports broke, the suite failed in new ways, and the real failure got buried.

Steps

1. Assess the damage: diff the file tree before and after the agent's patch and list every renamed, moved, or reordered file. Expected output: a complete move list.
2. Revert all file moves and renames; keep only the content fixes, re-applied to the original paths. Expected output: the original file layout with the real fixes intact.
3. Fix the breakage the reorder caused: repair imports against the restored layout and run the suite. Expected output: a green suite on the original file structure.
4. Add the rule to the healing config: no file renames, moves, or reordering; content edits only. Expected output: an enforced lane boundary.
5. If the file layout genuinely needs cleanup, schedule it as a separate refactor with import fixes and full review. Expected output: tidiness done properly, not smuggled into healing.

When to use

• Healing patches include renamed or moved test files.
• Imports break after a healing run in ways unrelated to the original failure.
• Diffs are polluted with file moves that hide the real change.

When not to use

• The reorganization is deliberate and planned (then it is a refactor with its own review, not healing).
• The test framework requires a naming convention the files violate (fix the names once, deliberately, then heal).
• Only one file was touched and nothing moved (no reorder happened).

Variant phrasings

Agent renamed test files to match a pattern

Same rule: renames are not healing. Revert and heal in place.

Agent moved helpers into a new directory

Helper moves break every importer. Same revert, same rule.

Agent sorted test cases within a file

Reordering cases can break order-dependent tests and always pollutes the diff. Content fixes only.

Why it happens

The agent "improves" what it touches, and alphabetical order looks like an improvement. Nothing tells it that file layout is load-bearing: imports, fixtures, and suite discovery all depend on paths. The reorder feels productive while actively breaking things.

Edge cases

• Some frameworks derive test ids from file paths; a rename invalidates baselines and history even if imports survive.
• The agent may reorder to work around a discovery bug (duplicate basenames); fix the discovery config instead.
• Reverting moves after further edits is messy; freeze the tree at the first sign of reordering.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_5yTFlXzpWXLSCZSysZ0aWw