agent labeled a test flaky that only fails on main - PRs never run it, so the flake is a merge-queue-only regression

TL;DR

The test only fails on main because PRs never run it: it is a merge-queue-only regression, not flakiness. Confirm the coverage gap (which suites run where), bisect the merged commits since it last passed on main, and fix the regression. Labeling it flaky hides a real break.

The symptom

agent labeled a test flaky that only fails on main  -  PRs never run it, so the flake is a merge-queue-only regression

Steps

1. Confirm the coverage gap

Check which suites run on PRs vs main. The test runs only on main; PRs never execute it. Expected: The gap documented: main-only test.

2. Bisect the merge range

Find the last green main run and bisect the merged commits since. The culprit is a merged change, not randomness. Expected: The breaking commit identified.

3. Fix the regression

Address the actual break the commit introduced for this test's scenario. Expected: Test green on main.

4. Close the coverage gap

Either run the test on PRs too, or accept main-only status explicitly with a fast main-failure alert. Do not leave it in the gap unlabeled. Expected: The test is covered where it runs.

When to use

• Test fails on main but PRs are green
• The test does not run on PRs
• Someone labeled it flaky without a bisect

When not to use

• The test fails on PRs too (then investigate normally)
• The main failure is genuinely intermittent (bisect first to be sure)
• Main is green and the label is historical (re-evaluate)

Why it happens

If PRs never run a test, regressions in its scenario merge silently and only surface on main. 'Fails on main, passes on PRs' looks like flakiness if you squint, but the pattern is deterministic: every main run fails. It is a regression with a coverage gap, not a flake.

Edge cases

• Merge-queue batching: bisect the batch, not just the branch.
• Main-only infrastructure differences: rule those out during the bisect.
• Scheduled main runs: use their history as the bisect bounds.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_shs1l8xyNw6x8rhOywGgMQ