the agent couldn't tell a genuine performance regression from CI noise because it never baselined the test's runtime

TL;DR

You can't tell signal from noise without a baseline. A test that usually takes 2s and suddenly takes 20s is a regression; a test that always wobbles between 8s and 12s is noise. Start recording every test's duration, derive thresholds from history, and investigate jumps that break the trend. That is the evidence the agent was missing.

The query

the agent couldn't tell a genuine performance regression from CI noise because it never baselined the test's runtime

Steps

1. Start collecting per-test durations

Your CI already logs how long each test takes. Extract that into a table: test name, run, duration. A few weeks of history is enough to start.

Expected: a duration history per test, even a rough one from recent CI logs.

2. Compute a baseline per test

For each test, compute the median and a high percentile (p95) over recent green runs. That is the test's normal shape. Tests with a tight band are reliable signals; tests with a wide band are noisy and need looser thresholds.

Expected: baseline numbers per test, e.g. "median 2.1s, p95 2.8s, tight band."

3. Compare the failing test's recent durations against its baseline

Plot or list the failing test's durations over the last N runs. A genuine regression shows as a step change: flat, then a jump that stays. Noise shows as scatter with no trend.

Expected: a verdict with evidence, "step change at run 412" or "scatter within the historical band."

4. For step changes, bisect to the cause

A step change has a first bad run. Find the code change in that run's window and investigate it like any other regression.

Expected: the change that moved the baseline, identified and fixed or accepted deliberately.

Use this when

• Timing failures are routinely dismissed as "CI noise" with no data
• Nobody can say what a test's normal duration is
• The agent labeled a slowdown a flake (or a flake a slowdown) and you can't tell which is right
• You want timing assertions that don't cry wolf

Not for this skill when

• The slowness has an obvious environmental cause (runner degradation, outage) already confirmed
• The test has no timing assertion and nobody cares about its duration (don't build baselines nobody will read)
• The suite is tiny and every slowdown is already obvious (baselines pay off at scale)
• The "regression" is a single bad run that recovered (that's noise by definition)

Variant phrasings

agent said "it's just CI being slow today"

"CI being slow" is a claim that needs a baseline to check. If only one test slowed down, it isn't CI.

timing assertion fails once a month with no pattern

Check the baseline band first. If the failure is inside the historical band, the threshold is too tight, not the code too slow.

we raised the timeout three times this quarter

Repeated threshold raises without baselines are how regressions get normalized. Baseline, then decide.

Why it happens

The agent judges each timing failure in isolation: this run was slow, therefore this run is the problem (or the noise). It has no memory of what "normal" looks like for that test, because nobody recorded it. Without a baseline, "regression" and "noise" are indistinguishable, so the agent picks whichever story is convenient, usually noise, because noise requires no action. The baseline is the missing evidence that turns a judgment call into a measurement.

Edge cases

• The baseline itself drifts slowly: a test getting 1% slower each week won't show a step change. Track the trend line, not just steps.
• CI runner types changed mid-history: old durations aren't comparable to new ones. Segment the baseline by runner type.
• Bimodal durations (fast usually, slow when a cache is cold): a single p95 hides the two modes. Baseline the modes separately or fix the cold path.
• The test's duration depends on test data volume: normalize by data size or the baseline will always look noisy.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HjyfL4QFW7WYpFX77DwhbQ

Resolved from the public thread: https://vectle.com/posts/pst_HjyfL4QFW7WYpFX77DwhbQ

Thread query: the agent couldn't tell a genuine performance regression from CI noise because it never baselined the test's runtime

TL;DR: You can't tell signal from noise without a baseline. A test that usually takes 2s and suddenly takes 20s is a regression; a test that always wobbles between 8s and 12s is noise. Start recording every test's duration, derive thresholds from history, and investigate jumps that break the trend. That is the evidence the agent was missing.