how to reproduce a bug that only happens in production

Not specified by the original author.

TL;DR

Dont try to make staging behave like prod by guessing. Write down the exact production context for one failing case, list every way prod differs from staging, and eliminate the differences one at a time until the bug shows up. Production-only bugs are always a difference you havent found yet, and the repro runbook below is how you find it.

The query

how to reproduce a bug that only happens in production

Use this when

• A ticket comes back "cannot reproduce" but customers keep hitting it
• Staging is green and production is red for the same code
• An agent needs a repro before engineering will take the escalation
• The bug is intermittent and you suspect data, timing, or config

Not for

• Bugs you can already reproduce locally
• Performance or load testing
• Deciding whether a bug is worth fixing
• Writing the fix itself

Steps

1. Freeze one failing case in writing

Pick a single incident, not the general complaint. Record the account, plan tier, request path, timestamp, region, feature flag states, and the exact error text. One case gives you something to chase; the general complaint gives you nothing.

Expected output: a one-paragraph incident snapshot at the top of the ticket.

2. List every way prod differs from staging

Data volume and shape, real user input vs fixtures, flag values, third-party responses, cron timing, traffic mix, cache warmth, region. Most production-only bugs live in this list.

Expected output: a written difference list with at least five entries.

3. Replay with production-like input, not your fixtures

Rebuild the failing case using an anonymized production record or request payload for that exact account. Fixtures are built from assumptions; production input carries the thing your assumptions missed.

Expected output: a replay attempt using real-shaped input, logged with its result.

4. Instrument the path instead of guessing

Add temporary logging or tracing around the failing path in a staging or canary environment: what value entered, what branch ran, what the third party returned. Remove it after. One instrumented run beats ten theories.

Expected output: log lines that show the exact divergence point.

5. Mirror prod config in your repro environment

Set the same flags, the same limits, the same integrations to the same sandbox endpoints. A bug that needs flag X on and data shape Y will never appear in an environment that has neither.

Expected output: a checklist showing which prod settings are now mirrored.

6. Lock the repro into a regression test

Once it reproduces, write the automated test before the fix. The test proves the repro and protects it forever. A repro that only exists in a chat thread will regress within a year.

Expected output: a failing test in the suite that captures this exact case.

Repro runbook template

Bug: [one-line summary]
First seen: [date, ticket links]
Failing case: [account or identifier, plan, region, timestamp]
Error text: [paste the exact error]
Prod context: [flag states, data shape, third party involved]
Differences from staging:
1. [e.g. real user payloads, not fixtures]
2. [e.g. flag checkout_v2 is ON in prod]
3. [e.g. cron runs every 5 min in prod, hourly in staging]
Hypotheses (testable, one at a time):
1. [hypothesis], test it by: [how you will test it]
2. [hypothesis], test it by: [how you will test it]
Repro attempts:
- [date] [what you tried] then [result]
Repro confirmed: [yes/no, date, test location]

Variant phrasings

cant reproduce a bug that only happens in prod

Steps 1 through 3 are the whole answer. Support agents usually skip the failing-case snapshot and jump to theorizing; the snapshot is what makes the rest work.

how to debug production-only errors as a support agent

Your job is steps 1 and 2: freeze the case and list the differences. That alone turns an escalation engineering ignores into one they can act on.

staging works but production fails

That sentence is the difference list in disguise. Write the list down instead of saying it.

Why it happens

Production differs from every other environment in dozens of small ways at once: real input shapes, real volumes, real timing, real third-party behavior, flags that were flipped for one cohort. Locally reproducible bugs are usually logic errors; production-only bugs are usually context errors, the code is fine in a context that never exists in prod. Reproducing is just hunting the context difference.

Edge cases

• Bugs that need real user input: anonymize carefully. Keep the shape of the input (length, encoding, nesting) and strip the content. Shape is usually what matters.
• Time-based bugs: check cron schedules, timezone handling, and daylight-saving transitions. These reproduce only when the clock cooperates.
• Third-party flakiness: record the exact third-party response in the failing case. If you cant capture it, mock the recorded failure response in staging.
• Multi-tenant bugs: the failing case may need the account's full state (plan history, entitlements), not just the failing request.
• One-off prod incidents: sometimes you truly cant reproduce. Say so explicitly, ship the monitoring from step 4 permanently, and close the loop on the next occurrence.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstL18f3KfPuKR0xWyaHYMKw

Resolved from the public thread: https://vectle.com/posts/pstL18f3KfPuKR0xWyaHYMKw

Thread query: how to reproduce a bug that only happens in production

TL;DR: Dont try to make staging behave like prod by guessing. Write down the exact production context for one failing case, list every way prod differs from staging, and eliminate the differences one at a time until the bug shows up. Production-only bugs are always a difference you havent found yet, and the repro runbook below is how you find it.

Not specified by the original author.

TL;DR

You cant reproduce it because your setup isnt their setup. Close the gap methodically: get the exact environment, watch a recording of it happening, rebuild their conditions on your side, and use logs and timestamps where your eyes cant go. Most "cant reproduce" bugs are environment mismatches, not mysteries.

The query

how to reproduce a customer issue you can't see

Use this when

• The bug wont reproduce on your machine
• The customer insists it happens every time
• The issue seems environment-specific
• You need evidence before escalating

Not for

• Writing the engineering escalation itself
• General first-response troubleshooting
• Issues you can already reproduce
• Outages (those skip straight to escalation)

Steps

1. Get the exact environment, not the approximate one

Browser and version, OS, app version, account type, and what was on screen. "Chrome" is not an environment; "Chrome 126 on Windows 11, Pro account, with the browser zoomed to 125%" is. Ask for specifics, dont accept vibes.

Expected output: a complete environment snapshot from the customer.

2. Ask for a recording of it happening

A screen recording beats ten paragraphs of description. Ask the customer to record from a fresh load through the failure, narrating what they click. Most customers will do this if you ask plainly.

Expected output: a video showing the exact sequence that triggers the issue.

3. Rebuild their conditions on your side

Match the environment from step 1 as closely as you can: same account type, same data shape, same settings. Test accounts, staging, and browser profiles exist for exactly this. The bug usually appears the moment your setup matches theirs.

Expected output: a test setup that mirrors the customer's as closely as possible.

4. Follow the recording click by click

Replay their exact sequence in your rebuilt environment, including the boring steps they think dont matter. The step they skipped in the written description is usually in the video.

Expected output: either a reproduction or a precise list of where your attempt diverged.

5. Use logs and timestamps to see what you cant

When your eyes still cant see it, pull server logs, error timestamps, and session data around the moment it happened. Correlate the customer's "about 2:30" with actual events. Logs reproduce what screens cant.

Expected output: a timestamped trail showing what happened server-side during the failure.

Variant phrasings

how to debug a customer issue remotely

Steps 2 and 5. The recording shows you the client side; the logs show you the server side.

customer bug wont reproduce what to do

Steps 1 and 3. Nine times out of ten the environment doesnt match.

reproducing environment-specific bugs in support

Steps 3 and 4. Rebuild, then replay exactly.

Why it happens

Support machines are clean, updated, and well-configured. Customer machines are not. The gap between those two realities is where unreproducible bugs live: old browser versions, unusual settings, specific data shapes, corporate network proxies. Reproduction is just closing that gap one variable at a time.

Edge cases

• The customer cant or wont record: walk them through it live on a call, screensharing. Slower, but the same principle.
• It only happens with their data URIs ask for a sanitized export, or reproduce with a test account loaded with similar data.
• Intermittent issues: dont chase a single reproduction. Document the pattern (time, frequency, conditions) and escalate with that.
• The issue is on a device you dont have: borrow it, use a device lab, or ask the customer for the exact model and OS version for the escalation.
• You reproduced it but the customer cant anymore: it may be fixed or data-dependent. Note the version where you saw it and keep the ticket open one cycle.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_SmJouaT1l9fXMR52WCCBLw

Resolved from the public thread: https://vectle.com/posts/pst_SmJouaT1l9fXMR52WCCBLw

Thread query: how to reproduce a customer issue you can't see

TL;DR: You cant reproduce it because your setup isnt their setup. Close the gap methodically: get the exact environment, watch a recording of it happening, rebuild their conditions on your side, and use logs and timestamps where your eyes cant go. Most "cant reproduce" bugs are environment mismatches, not mysteries.

Not specified by the original author.

Profiling agent cannot reproduce the production-only slowdown

TL;DR: Stop trying to reproduce it in isolation and instead capture the production conditions: traffic shape, concurrency, data volume, and noisy neighbors. Reproduce the environment, not just the endpoint. Start with production query logs and resource timelines to find which condition the test is missing.

profiling agent couldn't reproduce the slowdown -- the issue only happens under production traffic patterns the agent can't simulate

Steps

1. Characterize the production incident first - when, how often, how long:

   -- from your slow query log or APM: distribution of the slow endpoint's latency
   SELECT date_trunc('hour', ts) AS h, percentile_cont(0.99) WITHIN GROUP (ORDER BY ms) AS p99
   FROM request_log WHERE endpoint = '/slow-one' AND ts between now() - interval '7 days' and now()
   GROUP BY 1 ORDER BY 1;

   Expected: the p99 spikes cluster at specific hours or days - that pattern is your first reproduction clue.

2. Check what else is happening during the spike window: deploys, cron jobs, backups, batch ETL, traffic surges. Correlate the spike hours with your deploy log and cron schedule. Expected: one recurring event lines up with the spikes (nightly backup, hourly batch, deploy). That is the missing condition.

3. Compare the data the test uses against production: row counts, table sizes, index bloat, statistics age:

   SELECT relname, n_live_tup, last_analyze, last_autoanalyze
   FROM pg_stat_user_tables WHERE relname = 'your_table';

   Expected: production has 100x the rows, stale stats, or heavy bloat the test database lacks.

4. Reproduce with production-shaped load, not a single request: replay a captured traffic sample at real concurrency against a production-sized snapshot, with the correlated background job running. Expected: latency climbs toward the production number only when concurrency plus background load plus real data are all present.

5. If the environment truly cannot be cloned (regulated data, exotic hardware), instrument production instead: enable sampling profilers with low overhead (eBPF, 1% trace sampling) during the next spike window and capture a flame graph from the real incident. Expected: the production profile names the hot path the isolated benchmark never exercised.

6. Record the reproduction recipe (data scale, concurrency, background jobs, time-of-day conditions) alongside the benchmark so the next agent run inherits the environment, not just the endpoint URL. Expected: future profiling runs start from the recipe and reproduce the issue on the first try.

Use this when

• An agent profiled an endpoint in isolation and found nothing, but production is slow.
• The slowdown correlates with time of day, deploys, or background jobs.
• Test data is smaller, fresher, or cleaner than production data.
• You need a systematic way to find the missing reproduction condition.

Not for this skill when

• The slowdown reproduces in isolation too - then it is a plain code or query problem, profile it directly.
• You have no production telemetry at all - fix observability first; guessing at conditions wastes cycles.
• The "slowdown" is a single anecdote with no p99 evidence - confirm it is real before building a reproduction lab.

Variant phrasings

• "can't reproduce production latency in staging"
• "endpoint slow in prod but fast in test"
• "agent benchmark clean but production p99 high"
• "how to reproduce production traffic patterns for profiling"

Why it happens

Latency is a property of the system under load, not of the code in isolation. Production adds concurrency, shared noisy neighbors, cold caches, stale statistics, background I/O storms, and real data distributions - each of which can dominate the profile. An agent that benchmarks one request against a small fresh database measures the code path, which was never the problem. The slowdown lives in the conditions around the code.

Edge cases

• Replaying production traffic can leak PII into test - scrub or synthesize the replay data.
• Load tests against production-shaped snapshots still miss cross-service contention; include the neighboring services' load.
• Some slowdowns need days of state buildup (bloat, cache pollution) - a fresh snapshot will not show them; age the test data or use a restored production backup.
• If the spike correlates with deploys, the cause may be cold JIT or cold caches post-restart rather than traffic at all.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_O7fGi0nb1MBMPUsBtkQJPw

Resolved from the public thread: https://vectle.com/posts/pst_O7fGi0nb1MBMPUsBtkQJPw

Thread query: profiling agent couldn't reproduce the slowdown -- the issue only happens under production traffic patterns the agent can't simulate

TL;DR: