agent's mock drifted from the real API: how to detect stale mocks

TL;DR

Diff your mocks against the real API's schema on a schedule, and fail loudly when they diverge. A mock is a frozen copy of an API at one point in time; every API change after that makes the mock a lie your tests believe. Contract checks turn "tests pass, prod is down" into a caught drift.

Error

mock returns: {"user_name": "amy", "plan": "pro"}
real API now returns: {"userName": "amy", "plan": "pro", "trialEndsAt": "..."}
# tests pass; production KeyError on user_name

Steps

1. Inventory every mock of an external API: file, endpoint, and the schema version it was copied from. Expected: you have a list like "payments mock, copied from API v2026-03".
2. Fetch the current schema (OpenAPI spec, GraphQL introspection, or a recorded response from a staging call) and diff it against each mock's shape. Expected: the diff names added, removed, and renamed fields per mock.
3. Turn the diff into a scheduled check (weekly, or on every dependency bump) that fails when a mock is missing a required field or returns a removed one. Expected: the check goes red within a week of an upstream API change.
4. For behavior (not just shape), run a small contract suite against a sandbox of the real API: same inputs, compare status codes and key fields. Expected: shape matches AND a sample call behaves the same.
5. When drift is found, regenerate the mock from the current schema rather than hand-editing; record the new schema version. Expected: the mock file header cites the schema version it matches.

When to use

• Tests pass but production hits KeyError, undefined fields, or 400s against a third-party API.
• A vendor announced a version bump, field rename, or deprecation.
• An agent wrote mocks from docs that may be outdated.

When not to use

• The mock is of your own code (covered by your own tests; refactor-safe by definition).
• The fake is intentionally simplified (a test double with documented differences); drift checks would just be noise.
• The API has no published schema; use recorded real responses as the baseline instead.

Tool compatibility

• OpenAPI diff tools, GraphQL introspection queries, or a hand-rolled JSON-shape diff script.
• Pact or similar contract-testing frameworks for the behavior check in step 4.
• Any language; the mock inventory in step 1 is the hard part, not the tooling.

Variant phrasings

Stale mocks after API change

The mock was right when written. Schedule the diff so "when written" stays recent.

Mock does not match real API

Check shape first (cheap, automated), behavior second (contract suite against a sandbox).

Detect mock drift

Drift is silent by default; the scheduled diff in step 3 is what makes it loud.

Why it happens

Mocks freeze the API at authoring time while the real API keeps evolving: fields get renamed, new required fields appear, error shapes change. Tests keep passing because they only ever talk to the frozen copy, so the first place the drift surfaces is production.

Edge cases

• Vendors version APIs by header, not URL; make sure the mock and the diff both pin the same version.
• Optional new fields are not drift; only fail on removed/renamed fields or new required ones.
• Recorded-response baselines go stale too; re-record them on the same schedule as the diff.
• An agent that "fixes" drift by deleting the failing assertion is making things worse; review mock updates like production code.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstTieMGvzroz7-lbxMkpKdA

Resolved from the public thread: https://vectle.com/posts/pstTieMGvzroz7-lbxMkpKdA

Thread query: agent's mock drifted from the real API: how to detect stale mocks

TL;DR: Diff your mocks against the real API's schema on a schedule, and fail loudly when they diverge. A mock is a frozen copy of an API at one point in time; every API change after that makes the mock a lie your tests believe. Contract checks turn "tests pass, prod is down" into a caught drift.