agent dry-ran the codemod and the output looked right, then the real run rewrote files the dry-run had skipped

TL;DR: The dry-run and the real run are not running the same command. Capture the file list from each, diff them, and look for the usual culprits: different working directories, different ignore rules, or the transform branching on the dry-run flag itself. Unify file discovery and flags, then verify both modes on a clean checkout.

agent dry-ran the codemod and the output looked right, then the real run rewrote files the dry-run had skipped

1. Re-run the dry-run and capture exactly which files it claims it would change. Save the list to a file.

Expected: a concrete file list from dry-run mode, not a memory of what it looked like.

2. Run the real command against a scratch copy of the repo (never the working tree first) and capture which files it actually changed, for example with git status before and after.

Expected: a second concrete file list you can diff against the first.

3. Diff the two lists and the two invocations. Check: working directory, ignore files respected in one mode but not the other, file discovery (git-tracked files vs glob), and any flag that differs between the two commands.

Expected: you find at least one concrete difference - the modes diverged for a reason, not by magic.

4. Check whether the transform itself branches on dry-run. Search the transform for the dry-run flag and see if any logic depends on it.

Expected: either the transform is flag-agnostic (good) or you found the branch that changes behavior (fix it so the flag only controls writing, never matching).

5. Unify the invocation: same working directory, same explicit file list or same discovery command, same flags apart from dry-run. Re-run both modes on a clean checkout and confirm the lists now match.

Expected: dry-run file list and real-run changed-file list are identical.

Use this when

• A codemod dry-run and real run touch different files
• The real run rewrote files the dry-run skipped, or skipped files the dry-run listed
• You are debugging jscodeshift, ast-grep, or custom transform invocations
• CI's codemod check passes but the applied run surprises you

Not for this skill when

• The transform is non-idempotent and the surprise is from a second pass, not from dry-run vs real divergence - that is an idempotency bug, separate skill
• Both modes agree but the transform output itself is wrong - that is a transform logic bug
• The file lists match and only the content differs - diff the outputs, the discovery is fine
• You never captured the dry-run list and are going from memory - re-run it and capture it properly first

Variant phrasings

• jscodeshift dry run different files than real run
• codemod dry-run skipped files that real run changed
• ast-grep dry run vs apply file list mismatch
• codemod preview lied about which files change

Why it happens

Dry-run and real run are two separate executions that only look like one command. Anything that differs between the executions - the working directory, which ignore files are honored, whether files are discovered via git or via glob, environment variables, or transform code that checks the dry-run flag - changes the file set. The most common culprit is file discovery: dry-run evaluated against git-tracked files while the real run globbed everything, or vice versa.

Edge cases

• Some runners pass the dry-run flag into the transform, and transforms sometimes use it to skip expensive matching. The flag must control writing only, never which files match.
• Running the real command in the working tree before verifying destroys the evidence. Always verify against a scratch copy first.
• Concurrency differences between modes can change ordering but should never change the file set. If the set changes, it is discovery or flags, not threads.
• After unifying, keep the exact commands in the runbook. The next divergence will come from someone retyping the command from memory.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_rkfbSfVj8dzIZf4FN3e8hw