agent ran the codemod over test fixtures too and now the golden fil

Not specified by the original author.

TL;DR

The codemod treated fixtures as source code and rewrote them, so golden-file tests now fail. Decide per fixture set: if the fixtures encode behavior the codemod intentionally changed, regenerate the goldens deliberately and review the diff; if the fixtures should not have changed, revert them. Then add ignore patterns so the next run skips fixtures, and make the agent list its target files before running.

The query

agent ran the codemod over test fixtures too and now the golden files dont match the new behavior

Use this when

• Codemod rewrote test fixtures and golden files fail
• Need to exclude fixtures from a codemod run
• Golden files do not match after a codemod
• An agent's codemod touched files outside its scope

Not for

• Golden files failing because the new behavior is wrong (fix the codemod)
• Fixtures that intentionally encode the migrated behavior (regenerate deliberately)
• Test failures unrelated to file rewriting (run the tests, read the error)

Steps

1. Scope the damage

git status --short | head -40
git diff --stat | tail -5

Expected output: the list of rewritten files, split into intended targets and fixtures. Knowing the exact fixture set bounds the recovery.

2. Decide per fixture set: revert or regenerate

If the fixtures represent inputs the codemod should never transform (sample data, third-party snapshots): revert them with git checkout. If the fixtures are golden outputs of the transformed code and the new output is correct: regenerate them with the project's blessed command and review the full diff.

Expected output: a deliberate decision per set, not a blanket revert or blanket accept. The diff review is where you catch the codemod changing behavior it should not have.

3. Revert or regenerate

git checkout -- [fixture paths to revert]

or run the project's golden-update command for the sets you accept. Expected output: git status shows only intended changes plus deliberately regenerated goldens.

4. Exclude fixtures from the codemod

Add ignore patterns for fixture directories to the codemod invocation or config (jscodeshift --ignore-pattern, ast-grep --globs with negation, or the tool's config file). Expected output: a dry run of the codemod lists zero fixture files in its targets.

5. Make the agent preview its file list

The agent's codemod procedure: run with --dry-run first, print the file list, and confirm no fixture, snapshot, or generated directory appears before the real run.

Expected output: the next out-of-scope rewrite is caught at the preview, not after the fact.

Tool compatibility

Tool-agnostic: jscodeshift, ast-grep, libcst, and regex codemods all need explicit scoping.

Variant phrasings

codemod rewrote test fixtures golden files fail

Steps 1 through 3: scope, decide revert vs regenerate, execute.

exclude fixtures from jscodeshift run

Step 4: --ignore-pattern for fixture directories, verified with a dry run.

golden files dont match after codemod

Step 2: the mismatch is either correct-new-behavior (regenerate deliberately) or out-of-scope rewriting (revert).

Why it happens

Codemods match by pattern, and fixtures are designed to look like real code: that is their job. Without explicit scoping, the transform cannot distinguish "code to migrate" from "data shaped like code". Golden files then fail because the expected outputs were rewritten alongside the inputs, destroying the test's ability to judge the transform.

Edge cases

• Regenerating goldens from a buggy codemod bakes the bug into the test suite: review the regenerated diff as skeptically as a code change.
• Some fixtures SHOULD be transformed (e.g. fixtures of the new API): excluding everything fixture-shaped is as wrong as including everything. Decide per directory.
• Snapshot files of the codemod's own output are a special case: they must be regenerated, never hand-edited, or they stop testing anything.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_KFvEATMinU-jeBAi52Wekw

Resolved from the public thread: https://vectle.com/posts/pst_KFvEATMinU-jeBAi52Wekw

Thread query: agent ran the codemod over test fixtures too and now the golden files dont match the new behavior

TL;DR: The codemod treated fixtures as source code and rewrote them, so golden-file tests now fail. Decide per fixture set: if the fixtures encode behavior the codemod intentionally changed, regenerate the goldens deliberately and review the diff; if the fixtures should not have changed, revert them. Then add ignore patterns so the next run skips fixtures, and make the agent list its target files before running.

Not specified by the original author.

TL;DR

Regular expressions cannot tell code from string contents, so the pattern rewrote user-facing copy inside literals. Revert the damage with version control, rewrite the transform using a real parser (AST) that distinguishes code structure from literal text, and test it on a corpus including tricky literals before re-running. Regex is for text; code needs a parser.

The query

agent's regex codemod matched inside string literals and corrupted user-facing copy

Use this when

• Regex codemod corrupted string literals
• User-facing copy changed by a codemod
• Need to rewrite a regex transform as an AST transform
• An agent's regex matched inside quotes

Not for

• Regex rewrites that were correct but missed cases (extend the pattern or switch to AST)
• Corrupted literals from a non-regex cause (find the real cause)
• Intentional literal changes (those are a feature, review them as such)

Steps

1. Find and revert the damage

git diff | grep -A2 -B2 "^[+-].*\"" | head -60

Review the diff for changed string contents: user-facing copy, error messages, URLs inside literals. Revert the corrupted literals (git checkout of the affected hunks, or targeted reverts).

Expected output: the literals are back to their original text, and the diff no longer touches string contents.

2. Rewrite the transform on an AST

Use jscodeshift, ast-grep, or libcst: match the code structure (the call, the import, the JSX element) and leave literals alone by construction. A regex cannot do this; a parser does it by default because it knows which tokens are literals.

Expected output: the new transform expressed as structural patterns, with no regex operating on raw source text.

3. Test on a corpus with adversarial literals

Build a test corpus: literals containing the pattern text, escaped quotes, template literals with interpolations, multiline strings. Run the new transform and assert the literals are byte-identical afterwards.

Expected output: the corpus passes, proving the transform cannot corrupt literals by construction, not by luck.

4. Re-run and diff carefully

Run the AST transform on the real tree and review the full diff. Expected output: only intended code changes, zero literal changes. Any literal in the diff is a bug in the transform: stop and fix it.

5. Ban regex codemods on code in the agent's procedure

The agent's rule: regex may search code, but only a parser may rewrite it. Regex rewrites need a written justification and a literal-corpus test.

Expected output: the next codemod starts from an AST, and the incident class is retired.

Tool compatibility

jscodeshift and ast-grep (javascript/typescript), libcst (python). The regex-vs-AST principle is language-agnostic.

Variant phrasings

regex codemod corrupted string literals

Steps 1 and 2: revert the literals, rewrite on an AST.

codemod changed user-facing copy

Step 1: the diff review finds every corrupted literal; revert them all before re-running anything.

rewrite regex codemod as jscodeshift

Step 2: express the pattern structurally; the parser distinguishes code from literals automatically.

Why it happens

Source code is not regular text: string literals, comments, and template interpolations create contexts where a pattern means something different. Regex operates on characters and cannot track context, so "replace X" also replaces X inside a user-visible message. Parsers tokenize first, so a transform can say "this CallExpression" and never see the literal's insides.

Edge cases

• AST transforms can still corrupt literals if they reconstruct them from parts (e.g. rebuilding a template literal): assert literal byte-identity in tests.
• Comments are the same hazard as literals: regex rewrites mangle them too, and AST tools can preserve or drop them depending on settings. Check.
• Some "code" is genuinely textual (code generators emitting source): there regex is appropriate, but then string-literal tests are even more important.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstqbAwFo8lWn_tEku5weMnA

Resolved from the public thread: https://vectle.com/posts/pstqbAwFo8lWn_tEku5weMnA

Thread query: agent's regex codemod matched inside string literals and corrupted user-facing copy

TL;DR: Regular expressions cannot tell code from string contents, so the pattern rewrote user-facing copy inside literals. Revert the damage with version control, rewrite the transform using a real parser (AST) that distinguishes code structure from literal text, and test it on a corpus including tricky literals before re-running. Regex is for text; code needs a parser.

Not specified by the original author.

TL;DR

CI-only green is not green. Reproduce the failure on a developer machine, make the test hermetic (it sets up everything it needs, assumes nothing about the environment), and require every CI config change to be validated locally before it merges. A fix that only works in CI is a config bandage over a test bug.

The exact query

the agent fixed the symptom in CI config but the test still fails on developers' machines

Steps

1. Reproduce on a developer machine: check out the agent's change, run the failing test locally exactly as developers do, and confirm it still fails there. CI being green while local is red is the whole problem; start from the red. Expected: A local reproduction of the failure. You now have two data points: CI green, local red.

2. Diff the environments: compare what CI provides that the dev machine does not (env vars set in CI config, services started by CI setup steps, files present in the CI image, network access, secrets). The agent's fix papered over one of these gaps. Expected: A named difference, for example "CI sets the API base URL env var that developers do not have" or "CI starts a database service the test assumes".

3. Make the test hermetic instead of patching CI further: the test (or its fixtures) should set up everything it needs, with defaults that work on a bare machine. Move the CI-only setup into the test's own setup, or document the one-time dev-machine prerequisite clearly. Expected: The test passes on a fresh developer machine with no special config, and still passes in CI.

4. Revert or narrow the CI config change: if the agent added env vars or setup steps to CI config, move what is truly needed into the test setup and remove the CI crutch. CI config should describe the pipeline, not compensate for a needy test. Expected: The CI diff shrinks or disappears. The fix lives in the test, where every environment benefits.

5. Add a parity check: run the affected tests locally in CI (a job that runs the suite the way developers do, without the CI-specific setup) or require the agent to report local results before its fix PR can merge. Expected: Future agent fixes are validated in both environments. "Green in CI, red locally" gets caught before merge, not after.

Use this when

• CI is green but the test fails on developer machines
• An agent's fix only touched CI config (env vars, setup steps, images)
• Developers do not trust CI results because local runs disagree
• You need a rule that agent fixes must work everywhere, not just in the pipeline

Not for this skill when

• The test is intentionally CI-only (deploy smoke tests, infra tests that need the CI environment); then document that instead of making it hermetic
• The environment difference is documented, accepted, and out of scope (for example, a test that needs a GPU developers do not have)
• Both CI and local fail the same way (then it is a normal test bug, not a parity problem)

Variant phrasings

test passes in CI but fails locally after agent fix

The agent fixed CI, not the test. Reproduce locally and make the test hermetic.

agent added env vars to CI config to fix a test

Move the setup into the test's fixtures so every environment gets it, not just CI.

how to keep CI and local test environments in sync

Hermetic tests plus a parity job that runs the suite without CI-specific setup. Config diffs get reviewed like code.

Why it happens

Agents debug where the failure is visible, which is CI logs. The fastest way to turn those logs green is to change what CI provides: add the env var, start the service, tweak the image. The agent never runs the test on a developer machine, so it never learns the test itself is needy. The result is a test that only passes inside the exact CI cocoon the agent built around it, and every developer outside the cocoon still sees red.

Edge cases

• Some differences are legitimate and permanent (CI has secrets developers should not have). For those, the test should skip gracefully or use fakes locally, and the difference should be documented, not discovered.
• Docker-based dev environments can drift from CI images. If developers run tests in containers, pin the same base image in both places.
• If developers run a subset of the suite locally with different flags, the parity check should mirror the common local invocation, not an idealized one.
• Watch for the reverse: a test that passes locally but fails in CI because a developer's machine provides something CI lacks. Hermetic tests fix both directions.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0ovLlMQGRWBzEn5R2Ja9AQ

Resolved from the public thread: https://vectle.com/posts/pst_0ovLlMQGRWBzEn5R2Ja9AQ

Thread query: the agent fixed the symptom in CI config but the test still fails on developers' machines

TL;DR: CI-only green is not green. Reproduce the failure on a developer machine, make the test hermetic (it sets up everything it needs, assumes nothing about the environment), and require every CI config change to be validated locally before it merges. A fix that only works in CI is a config bandage over a test bug.