agent's audit passed dry-run rules but failed production policy: config error

TL;DR

The dry-run and production use different policy configs, so the dry-run proves nothing. Make both runs load the same policy version from the same source, add a parity check that fails the run when they differ, and re-validate.

Error

agent's audit passed dry-run rules but failed production policy

Steps

1. Diff the dry-run policy config against the production one: version, thresholds, rule files. Expected: the specific divergence is visible.
2. Point both runs at the same policy source (same versioned file or same policy service). Expected: one source of truth.
3. Add a startup parity check: the agent compares the loaded policy version against the expected production version and refuses to run on mismatch. Expected: divergence fails fast and loudly.
4. Re-run the dry-run against the corrected config. Expected: dry-run verdicts now predict production verdicts.
5. Version and deploy policy changes to dry-run and production together. Expected: the two can never drift again.

When to use

• Dry-run passes but production fails on the same inputs.
• Policy config differs between environments.

When not to use

• The rule logic is wrong in both environments (fix the rule).
• Production data differs from dry-run data (a data problem, not config).

Tool compatibility

• Custom audit agents; versioned policy configs across environments.

Variant phrasings

Dry-run green, production red

Same failure, terse wording.

Policy config drift between environments

Names the cause directly.

Why it happens

Dry-run environments accumulate their own config edits over time. Without a parity mechanism, the dry-run silently becomes a test of a policy nobody runs.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qYy0zpnlEqthBSGEzqvLWw