agent classified a timeout as flaky even though the test's runtime tripled in every failure - how to catch a...

TL;DR

A test whose runtime triples on every failure is not flaky. It is a regression that sometimes crosses the timeout. The agent must compare failure durations against the test's own history before applying the flake label. Add a timing check to the triage policy: if failing runs are consistently 2x or more the passing median, label it a performance regression and open a bug, not a quarantine.

The query

agent classified a timeout as flaky even though the test's runtime tripled in every failure - how to catch a regression hiding in flake labels

Use this when

• Timeouts are auto-labeled flaky but failing runs are consistently slower than passing runs.
• A test's runtime has drifted upward and the classifier never noticed.
• You suspect a latency regression is hiding behind the flake label.

Not for

• Genuine network-timeout flakes where failing runs show normal durations.
• Tests with no timing history to compare against.
• Timeouts caused purely by an overloaded CI runner (all tests slow, not one).

Steps

Step 1: Pull the timing history for the test

grep -h "test_payment_retry" reports/junit-*.xml | grep -o 'time="[0-9.]*"' | sort -n

Expected output: a sorted list of durations across recent runs. You need at least 20 passing runs to establish a baseline. If the CI system stores timing in a dashboard instead, export the same series from there.

Step 2: Compare failing-run durations against the passing baseline

python -c "
import statistics
passing = [4.1, 4.3, 3.9, 4.2, 4.0]
failing = [12.8, 13.1, 12.4]
print('passing median', statistics.median(passing))
print('failing median', statistics.median(failing))
print('ratio', statistics.median(failing) / statistics.median(passing))
"

Expected output: the ratio. In the example it is about 3x. Any consistent ratio at or above 2x means the failures are slow for a reason, and "flaky" is the wrong label.

Step 3: Find what got slow in the failing runs

grep -B 5 -A 20 "test_payment_retry" ci-logs/failing-run.log | grep -iE "slow|wait|lock|retry|timeout" | head -20

Expected output: the slow step inside the test (a database call, an HTTP retry, a lock wait). The tripling runtime has a cause, and it is usually one call that got slower, not the whole test.

Step 4: Relabel and file the regression

Record the decision in triage-decisions.log: testpaymentretry - 3x runtime on failures, relabeled perf-regression, not flaky.

Expected output: the decision is recorded with the evidence (the ratio from step 2). The test is now tracked as a performance regression with an owner, not silently quarantined as a flake.

Step 5: Fix the classifier to require timing evidence

flake_classifier:
  require_timing_check: true
  max_fail_to_pass_ratio: 2.0

Expected output: the agent can no longer label a timeout flaky without comparing durations. A timeout with a 2x-plus slowdown ratio is routed to the regression queue automatically.

Step 6: Add a runtime-drift alert for the suite

Add a monitoring rule: alert if any test's 7-day median runtime exceeds 1.5x its 30-day median.

Expected output: the next slowdown gets caught as drift before it starts tripping timeouts, so the classifier never has to choose between "flaky" and "slow" again.

Variant phrasings

agent bumped the timeout to 60s and called it fixed

Bumping the timeout hides the regression the way quarantine hides the race. Steps 2-3 find the real slowdown; the timeout stays where it was.

flake classifier trusts green reruns but the rerun ran on a different runner

A green rerun on a bigger runner proves nothing about the code. Step 2's ratio uses the test's own history on comparable runners.

test passes on retry so the agent closed it as flaky

Passing on retry is necessary but not sufficient evidence. The timing check in step 5 is the missing half of the verdict.

Why it happens

Timeouts have two causes: slow code and slow environments. The classifier only checked the symptom (it timed out, then passed on retry) and never the durations. A regression that triples runtime will pass on retry whenever the slow path happens to be fast that run, which looks exactly like flakiness to a classifier that ignores timing. The label was wrong because the evidence was incomplete, not because the classifier is stupid.

Edge cases

• If all tests on the runner are slow during the failing runs, the cause is the runner, not the test. Compare the test's slowdown against the suite-wide slowdown before relabeling.
• A test with high natural variance (integration tests hitting real services) needs a wider ratio threshold. Tune step 5's 2.0 per test type.
• Do not delete the timeout to "prove" it is slow. The timeout is the safety net. Measure around it.
• The first few runs after a deploy can be slow (cold caches, JIT warmup). Exclude post-deploy runs from the baseline or the ratio will false-positive.
• If the slowdown is in test setup rather than the code under test, the regression is in the test infrastructure. File it there, not against the product code.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstrdLRUM5-ibpJ7c7KYozYw