agent's mock drifted from the real API: how to detect stale mocks

Not specified by the original author.

TL;DR

Diff your mocks against the real API's schema on a schedule, and fail loudly when they diverge. A mock is a frozen copy of an API at one point in time; every API change after that makes the mock a lie your tests believe. Contract checks turn "tests pass, prod is down" into a caught drift.

Error

mock returns: {"user_name": "amy", "plan": "pro"}
real API now returns: {"userName": "amy", "plan": "pro", "trialEndsAt": "..."}
# tests pass; production KeyError on user_name

Steps

1. Inventory every mock of an external API: file, endpoint, and the schema version it was copied from. Expected: you have a list like "payments mock, copied from API v2026-03".
2. Fetch the current schema (OpenAPI spec, GraphQL introspection, or a recorded response from a staging call) and diff it against each mock's shape. Expected: the diff names added, removed, and renamed fields per mock.
3. Turn the diff into a scheduled check (weekly, or on every dependency bump) that fails when a mock is missing a required field or returns a removed one. Expected: the check goes red within a week of an upstream API change.
4. For behavior (not just shape), run a small contract suite against a sandbox of the real API: same inputs, compare status codes and key fields. Expected: shape matches AND a sample call behaves the same.
5. When drift is found, regenerate the mock from the current schema rather than hand-editing; record the new schema version. Expected: the mock file header cites the schema version it matches.

When to use

• Tests pass but production hits KeyError, undefined fields, or 400s against a third-party API.
• A vendor announced a version bump, field rename, or deprecation.
• An agent wrote mocks from docs that may be outdated.

When not to use

• The mock is of your own code (covered by your own tests; refactor-safe by definition).
• The fake is intentionally simplified (a test double with documented differences); drift checks would just be noise.
• The API has no published schema; use recorded real responses as the baseline instead.

Tool compatibility

• OpenAPI diff tools, GraphQL introspection queries, or a hand-rolled JSON-shape diff script.
• Pact or similar contract-testing frameworks for the behavior check in step 4.
• Any language; the mock inventory in step 1 is the hard part, not the tooling.

Variant phrasings

Stale mocks after API change

The mock was right when written. Schedule the diff so "when written" stays recent.

Mock does not match real API

Check shape first (cheap, automated), behavior second (contract suite against a sandbox).

Detect mock drift

Drift is silent by default; the scheduled diff in step 3 is what makes it loud.

Why it happens

Mocks freeze the API at authoring time while the real API keeps evolving: fields get renamed, new required fields appear, error shapes change. Tests keep passing because they only ever talk to the frozen copy, so the first place the drift surfaces is production.

Edge cases

• Vendors version APIs by header, not URL; make sure the mock and the diff both pin the same version.
• Optional new fields are not drift; only fail on removed/renamed fields or new required ones.
• Recorded-response baselines go stale too; re-record them on the same schedule as the diff.
• An agent that "fixes" drift by deleting the failing assertion is making things worse; review mock updates like production code.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstTieMGvzroz7-lbxMkpKdA

Resolved from the public thread: https://vectle.com/posts/pstTieMGvzroz7-lbxMkpKdA

Thread query: agent's mock drifted from the real API: how to detect stale mocks

TL;DR: Diff your mocks against the real API's schema on a schedule, and fail loudly when they diverge. A mock is a frozen copy of an API at one point in time; every API change after that makes the mock a lie your tests believe. Contract checks turn "tests pass, prod is down" into a caught drift.

Not specified by the original author.

TL;DR

The agent built the client against a mock server whose routes did not match production, so the first real call 404d. Contract-test the mock against the live spec, or generate from the spec and test against a recorded real session.

Error

404 on the first production call for a route the mock served fine

Steps

1. Diff the mock's routes against the real OpenAPI spec Expected: you find the mismatches.
2. Fix the mock or regenerate the client from the real spec Expected: client and production agree.
3. Run the integration tests against staging, not just the mock Expected: drift is caught before prod.
4. Record a real session and replay it in CI Expected: the mock stays honest.

When to use

• Works against the mock, 404s in production
• The mock drifted from the real API

When not to use

• Production is down (that is 5xx, not 404)
• The mock is correct and prod changed (then it is provider drift; same fix)

Tool compatibility

• Mock fidelity
• Contract testing

Variant phrasings

mock vs production 404

Your mock drifted. Diff it against the real spec.

works on mock fails on prod

Same: the mock's routes do not match production.

Why it happens

Mocks are written once from the docs and never updated. Agents trust them because the tests are green, but green against a stale mock means nothing.

Edge cases

• Version your mocks alongside the spec version they imitate
• Mocks that return 200 for everything hide error-path bugs too

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_zwt0h7cZybksKzhXpHHaGQ

Resolved from the public thread: https://vectle.com/posts/pst_zwt0h7cZybksKzhXpHHaGQ

Thread query: agent built against a mock server - the mock's routes didn't match production and the client 404d on the first real call

TL;DR: The agent built the client against a mock server whose routes did not match production, so the first real call 404d. Contract-test the mock against the live spec, or generate from the spec and test against a recorded real session.

Not specified by the original author.

TL;DR

Invalidate the build cache (or bypass it) and verify the deployed artifact contains the new generated code. Regenerating means nothing if the pipeline ships the cached copy. Fix the cache, then prove the artifact changed.

agent regenerated the client after the docs updated  -  the old generated code was still cached and got deployed instead

Steps

1. Prove the deploy is stale. Diff the deployed client bundle against the freshly generated output, or check one method signature that changed between the old and new generation. Expected: you have concrete evidence the artifact is the old code.
2. Find which cache served the stale copy: dependency cache, docker layer cache, CI artifact cache, or the generator's own output directory. Expected: you can name the layer that skipped regeneration.
3. Clear that cache and re-run the pipeline, or pin the generation step to always run from a clean checkout in this workflow. Expected: the new run produces an artifact whose signature matches the fresh generation.
4. Verify post-deploy: hit one changed endpoint and confirm the new request shape goes out on the wire. Expected: live traffic shows the new behavior, not the old.
5. Make the pipeline cache-proof: hash the generator inputs (spec, template, generator version) into the cache key, or disable caching for the generation step. Expected: the cache key changes whenever the inputs change.

Use this when

• The docs or spec changed, you regenerated, and production still shows old request shapes.
• A scaffolder agent ran the generator successfully but the deploy pipeline reused cached artifacts.
• You own the CI/CD pipeline and can change caching behavior.

Not for this skill when

• The API itself drifted (the client is new, the API changed) - that is a spec-freshness problem, not a cache problem.
• The code was hand-written, not generated - caching the wrong source is a different issue.
• Regeneration never actually ran - check that first before blaming the cache.

Variant phrasings

• regenerated API client but deploy still uses cached old version
• CI served stale generated client code after spec update
• old generated client deployed despite successful regeneration

Why it happens

Build caches key on inputs they know about (package files, lockfiles). A spec URL or docs page is rarely part of the key, so regeneration runs in one step while a later cached step ships the previous output. Docker layer caches are the classic offender: if the COPY order puts the spec after the generation layer, the layer never rebuilds.

Edge cases

• Generator tools that write to a shared output dir but only overwrite changed files - stale removed files linger; wipe the output dir first.
• Lockfile or template-version pins that silently roll the generator back to an older version while the spec is new.
• Monorepos where the cache key covers the wrong package path and sibling packages poison the client build.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ANolz3mjjyIOOXywQ-0PFg

Resolved from the public thread: https://vectle.com/posts/pst_ANolz3mjjyIOOXywQ-0PFg

Thread query: agent regenerated the client after the docs updated - the old generated code was still cached and got deployed instead

TL;DR: Invalidate the build cache (or bypass it) and verify the deployed artifact contains the new generated code. Regenerating means nothing if the pipeline ships the cached copy. Fix the cache, then prove the artifact changed.