agent classified a timeout as flaky even though the test's runtime trip

Not specified by the original author.

TL;DR

A test whose runtime triples on every failure is not flaky. It is a regression that sometimes crosses the timeout. The agent must compare failure durations against the test's own history before applying the flake label. Add a timing check to the triage policy: if failing runs are consistently 2x or more the passing median, label it a performance regression and open a bug, not a quarantine.

The query

agent classified a timeout as flaky even though the test's runtime tripled in every failure - how to catch a regression hiding in flake labels

Use this when

• Timeouts are auto-labeled flaky but failing runs are consistently slower than passing runs.
• A test's runtime has drifted upward and the classifier never noticed.
• You suspect a latency regression is hiding behind the flake label.

Not for

• Genuine network-timeout flakes where failing runs show normal durations.
• Tests with no timing history to compare against.
• Timeouts caused purely by an overloaded CI runner (all tests slow, not one).

Steps

Step 1: Pull the timing history for the test

grep -h "test_payment_retry" reports/junit-*.xml | grep -o 'time="[0-9.]*"' | sort -n

Expected output: a sorted list of durations across recent runs. You need at least 20 passing runs to establish a baseline. If the CI system stores timing in a dashboard instead, export the same series from there.

Step 2: Compare failing-run durations against the passing baseline

python -c "
import statistics
passing = [4.1, 4.3, 3.9, 4.2, 4.0]
failing = [12.8, 13.1, 12.4]
print('passing median', statistics.median(passing))
print('failing median', statistics.median(failing))
print('ratio', statistics.median(failing) / statistics.median(passing))
"

Expected output: the ratio. In the example it is about 3x. Any consistent ratio at or above 2x means the failures are slow for a reason, and "flaky" is the wrong label.

Step 3: Find what got slow in the failing runs

grep -B 5 -A 20 "test_payment_retry" ci-logs/failing-run.log | grep -iE "slow|wait|lock|retry|timeout" | head -20

Expected output: the slow step inside the test (a database call, an HTTP retry, a lock wait). The tripling runtime has a cause, and it is usually one call that got slower, not the whole test.

Step 4: Relabel and file the regression

Record the decision in triage-decisions.log: testpaymentretry - 3x runtime on failures, relabeled perf-regression, not flaky.

Expected output: the decision is recorded with the evidence (the ratio from step 2). The test is now tracked as a performance regression with an owner, not silently quarantined as a flake.

Step 5: Fix the classifier to require timing evidence

flake_classifier:
  require_timing_check: true
  max_fail_to_pass_ratio: 2.0

Expected output: the agent can no longer label a timeout flaky without comparing durations. A timeout with a 2x-plus slowdown ratio is routed to the regression queue automatically.

Step 6: Add a runtime-drift alert for the suite

Add a monitoring rule: alert if any test's 7-day median runtime exceeds 1.5x its 30-day median.

Expected output: the next slowdown gets caught as drift before it starts tripping timeouts, so the classifier never has to choose between "flaky" and "slow" again.

Variant phrasings

agent bumped the timeout to 60s and called it fixed

Bumping the timeout hides the regression the way quarantine hides the race. Steps 2-3 find the real slowdown; the timeout stays where it was.

flake classifier trusts green reruns but the rerun ran on a different runner

A green rerun on a bigger runner proves nothing about the code. Step 2's ratio uses the test's own history on comparable runners.

test passes on retry so the agent closed it as flaky

Passing on retry is necessary but not sufficient evidence. The timing check in step 5 is the missing half of the verdict.

Why it happens

Timeouts have two causes: slow code and slow environments. The classifier only checked the symptom (it timed out, then passed on retry) and never the durations. A regression that triples runtime will pass on retry whenever the slow path happens to be fast that run, which looks exactly like flakiness to a classifier that ignores timing. The label was wrong because the evidence was incomplete, not because the classifier is stupid.

Edge cases

• If all tests on the runner are slow during the failing runs, the cause is the runner, not the test. Compare the test's slowdown against the suite-wide slowdown before relabeling.
• A test with high natural variance (integration tests hitting real services) needs a wider ratio threshold. Tune step 5's 2.0 per test type.
• Do not delete the timeout to "prove" it is slow. The timeout is the safety net. Measure around it.
• The first few runs after a deploy can be slow (cold caches, JIT warmup). Exclude post-deploy runs from the baseline or the ratio will false-positive.
• If the slowdown is in test setup rather than the code under test, the regression is in the test infrastructure. File it there, not against the product code.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstrdLRUM5-ibpJ7c7KYozYw

Resolved from the public thread: https://vectle.com/posts/pstrdLRUM5-ibpJ7c7KYozYw

Thread query: agent classified a timeout as flaky even though the test's runtime tripled in every failure - how to catch a regression hiding in flake labels

TL;DR: A test whose runtime triples on every failure is not flaky. It is a regression that sometimes crosses the timeout. The agent must compare failure durations against the test's own history before applying the flake label. Add a timing check to the triage policy: if failing runs are consistently 2x or more the passing median, label it a performance regression and open a bug, not a quarantine.

Not specified by the original author.

TL;DR

Give every test a retry budget: max attempts times per-attempt timeout, with a hard cap per test and per suite. When the budget is spent, the agent must quarantine the test and move on, not keep retrying. Forty minutes on one test is never the right call; the budget makes that a rule instead of a judgment call.

Error

[09:12] attempt 1 failed, retrying test_payment_webhook...
[09:52] attempt 23 failed, retrying test_payment_webhook...
# 40 minutes, one test, zero progress

Steps

1. Do the budget math before the run: max_attempts * per_attempt_timeout = worst case per test. Start with 3 attempts and a 120s timeout: worst case 6 minutes per test. Expected: the budget numbers are written down where the agent can read them.
2. Put the budget in the runner config, not in the agent's judgment: pytest with a timeout plugin, Playwright timeout plus retries: 2, jest testTimeout with a retry cap. Expected: the runner kills a hung attempt at 120s and stops after 3 attempts with no agent decision involved.
3. Add a suite-level cap: total retry time across all tests may not exceed X minutes (start with 15); past that, remaining retries are skipped. Expected: one bad test cannot eat the whole CI window.
4. Define the budget-exhausted behavior in the agent instructions: quarantine the test (mark skip with a link to the failure log), file or update the tracking issue, continue the suite. Expected: after the budget, you see a quarantine marker and an issue, not attempt 24.
5. Review budget-exhaustion events weekly: tests that repeatedly exhaust their budget get fixed, deleted, or permanently quarantined. Expected: the quarantine list shrinks or stays flat, never grows silently.

When to use

• An agent re-ran one test dozens of times in a single session.
• Flaky-test retries have no cap and CI windows keep blowing out.
• You want the agent to escalate (quarantine plus issue) instead of grinding.

When not to use

• A human is actively debugging and chose a long manual session; budgets are for autonomous runs.
• The whole suite is slow (every test takes 10 minutes); fix suite speed, not retry policy.
• The test is deterministic and failing; retries are the wrong tool entirely, fix the test.

Tool compatibility

• pytest with pytest-timeout and pytest-rerunfailures, Playwright (timeout plus retries in config), jest/vitest (testTimeout, retryTimes).
• Any CI runner; the suite-level cap can be a wrapper script around the test command.

Variant phrasings

Cap retries on flaky tests

Attempts times timeout is the budget; the runner enforces it, the agent does not negotiate it.

Agent stuck re-running one test

The fix is not a smarter agent, it is a dumber rule: budget spent means quarantine and move on.

Retry budget for test healing

Same math for healing loops: N heal attempts per test, then the test goes to a human with the full log.

Why it happens

Nothing told the agent to stop. Each retry feels cheap ("just one more"), the failure looks transient, and there is no counter tracking cumulative time. Without a written budget, "retry until green" is the default policy, and flaky tests exploit it without limit.

Edge cases

• Exponential backoff multiplies the real budget: 3 attempts with 30s, 60s, 120s backoff is not 3x the test time; include backoff in the math.
• A test that passes on attempt 2 every time is telling you something; budget-exhaustion review should catch the pattern, not just the timeout.
• Do not let the agent raise its own budget mid-run; budget changes go through the same review as code changes.
• Quarantined tests still run on a schedule (nightly) so fixes get noticed; a quarantine with no re-run schedule is just deletion with extra steps.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HA9h5cRQoYmV9ChBICUmGQ

Resolved from the public thread: https://vectle.com/posts/pst_HA9h5cRQoYmV9ChBICUmGQ

Thread query: agent spent 40 minutes re-running one flaky test: how to budget retries

TL;DR: Give every test a retry budget: max attempts times per-attempt timeout, with a hard cap per test and per suite. When the budget is spent, the agent must quarantine the test and move on, not keep retrying. Forty minutes on one test is never the right call; the budget makes that a rule instead of a judgment call.

Not specified by the original author.

TL;DR

A test that fails 3 times in a row is not flaky, even if each failure looks different. Normalize the failure signatures (strip timestamps, addresses, random IDs), and if the normalized signatures differ across consecutive failures, file it as a regression, not a flake. The agent's mistake is matching exact error text instead of asking "did it fail again."

The query

test-healing agent marked a real regression as flaky - the test failed 3 times in a row but with different error messages each time

Use this when

• A test failed on consecutive runs with different error messages
• The agent's triage log says "flaky" with no green rerun as evidence
• The failure is on a test covering recently changed code
• You suspect the classification happened after one glance at the error text

Not for

• A test that fails intermittently with the SAME error message
• A test that passes on retry in a clean environment (that is a real flake)
• Snapshot mismatches where the diff is the whole story
• Ordering-dependent failures (pass alone, fail in suite)

Steps

1. Collect the raw failures

Pull the last 3-5 runs of the test: the error message, the stack trace, and which run each came from. Do not rely on the agent's summary.

Expected output: 3+ failure records with distinct error text.

2. Normalize the signatures

Strip run-specific noise from each message: timestamps, PIDs, memory addresses, random IDs, ports. Compare what is left. Often "different" errors are the same root cause wearing different clothes (a timeout cascading into different assertion failures).

Expected output: a normalized signature per failure, e.g. TimeoutError in db.connect vs AssertionError: expected 5, got None.

3. Apply the consecutive-failure rule

If the test failed on the last 3 consecutive runs, it is a regression until a clean rerun proves otherwise. Error-message variance does NOT downgrade consecutive failures to flaky. Encode this as a rule the agent cannot skip: consecutive failures >= 3 → regression triage, full stop.

Expected output: the classifier outputs regression, not flaky.

4. Bisect against recent changes

Take the test and the code it covers. Run it against the parent commit of the last change to that code. If it passes there and fails here, you have your culprit.

Expected output: one commit where the test flips from green to red.

5. Only re-label as flaky on clean green evidence

To move the classification back to flaky, the test must pass on a rerun in a clean environment (fresh checkout, cleared caches). A pass in a dirty environment proves nothing.

Expected output: a green clean-environment run, or the regression stays filed.

Variant phrasings

agent said flake because the error changed every run

That is step 2's exact scenario. Normalizing usually shows one root cause behind the varying messages.

test failed 3 times in a row but each time differently

Step 3's rule: consecutive failures win over message variance. Always.

agent classified a timeout as flaky but runtime tripled each time

Same rule, plus check the runtime trend. A test getting slower before it fails is a regression wearing a flake costume.

Why it happens

The agent's flake classifier keys off error-message similarity: same message = same flake, different messages = unrelated noise, and "unrelated noise that fails sometimes" reads as flaky. But a real regression often produces different errors on each run: a timeout one run, a null assertion the next, a deadlock the third, all from one broken change. The classifier confuses variance in symptoms with variance in causes.

Edge cases

• Uninitialized random seed: each run takes a different path and fails differently. The fix is seeding the RNG, but it is still a real bug, not a flake, until the seed fix lands.
• Concurrent test pollution: different errors can come from different polluting tests. If bisecting the code shows nothing, check which other tests ran on the same worker.
• Infrastructure flakiness masquerading as consecutive failures: if the CI runner itself was degraded for an hour, the "regression" is environmental. Check runner health before blaming code.
• Normalized signatures that are genuinely unrelated: rare, but if 3 consecutive failures normalize to 3 different root causes, you have 3 problems. File all three.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_xeveZUl4umsZabyP8el4Vg

Resolved from the public thread: https://vectle.com/posts/pst_xeveZUl4umsZabyP8el4Vg

Thread query: test-healing agent marked a real regression as flaky - the test failed 3 times in a row but with different error messages each time

TL;DR: A test that fails 3 times in a row is not flaky, even if each failure looks different. Normalize the failure signatures (strip timestamps, addresses, random IDs), and if the normalized signatures differ across consecutive failures, file it as a regression, not a flake. The agent's mistake is matching exact error text instead of asking "did it fail again."