agent couldn't reproduce the failure: it only failed on the merge queue

TL;DR

A failure that only happens in the merge queue is usually about what the queue adds: the rebase onto latest target, sibling PRs batched together, and heavier load. Reproduce it by replicating those conditions: rebase the branch onto the target tip, run the affected tests plus their dependents together, and approximate the queue's parallelism. If the agent only ever tests the branch in isolation, it will never see it.

Error

agent couldn't reproduce the failure: it only failed on the merge queue

Local runs and branch CI are green; the merge queue fails the same test every time, and the agent declares it unreproducible.

Steps

1. Rebase the branch onto the current target tip exactly as the queue does, and re-run the failing test. Expected output: either a reproduction or proof the rebase alone is not the trigger.
2. Identify the batch: find which other PRs were in the queue batch when it failed, and apply their changes too. Expected output: a reproduction environment matching the failed batch.
3. Run the failing test together with its dependents and neighbors, not in isolation; order-dependence and shared state hide in isolation. Expected output: the failure, if it is interaction-based.
4. Approximate queue load: run with the queue's parallelism and resource limits, since timing flakes surface under load. Expected output: timing-sensitive failures reproduced.
5. Once reproduced, diagnose normally; then add the scenario to the pre-merge checks so the queue is not the first place it is seen. Expected output: a regression test or check that catches it on the branch.

When to use

• A test fails only in the merge queue.
• The agent reports "cannot reproduce" after testing the branch alone.
• Batch interactions or rebase effects are suspected.

When not to use

• The failure reproduces on the branch (ordinary bug; fix it there).
• The queue failure is infra (runner issues); the test is fine.
• The queue is the only place the test runs (then it is not queue-specific, it is just untested elsewhere).

Variant phrasings

Failure only happens when two PRs land together

Batch interaction. Reproduce with both applied; the fix may belong in either PR.

Test passes alone, fails with the full suite in the queue

Order or resource dependence. Bisect the suite interaction; require order-independence going forward.

Rebase onto target breaks the test

The target moved under the branch. The branch needs updating, not the test needs healing.

Why it happens

The merge queue is a different execution context from branch CI: newer base, batched changes, different parallelism and load. The agent reproduces in the branch context, which lacks all three, and concludes the failure is phantom. It is not phantom; it is contextual.

Edge cases

• The queue batch that failed may no longer exist; reconstruct it from queue history or bisect against the target tip.
• Flaky-under-load failures reproduce probabilistically; run the replicated setup multiple times before declaring it unreproducible.
• If the queue uses different config (env, flags) than branch CI, diff the configs; the trigger may be a setting, not code.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_gH1vH-WuTDtcUqheXVm0pQ