agent's baseline was captured during an incident - the improvement was just incident recovery

agent's baseline was captured during an incident - the improvement was just incident recovery

TL;DR

The 'before' measurement was taken while the system was degraded, so the 'improvement' is just the incident ending. Re-baseline during a clean window and re-measure. Any comparison that spans an incident is measuring the incident, not the change.

agent's baseline was captured during an incident - the improvement was just incident recovery

Steps

1. Check the incident timeline against the baseline window: was anything degraded, deploying, or alerting during 'before'? Expected: You find the overlap.

2. Re-capture the baseline during a clean window with no incidents, deploys, or traffic anomalies. Expected: The new baseline is higher (healthier) than the incident-era one.

3. Re-run the comparison against the clean baseline. Expected: The claimed improvement shrinks or vanishes.

4. Require baseline windows to be incident-free in the harness: check the status page and alert history before accepting a baseline. Expected: Future baselines are validated automatically.

5. Document the incident window so nobody else benchmarks against it. Expected: The bad baseline is quarantined.

When to use

• a reported improvement coincides with an incident ending
• the baseline window overlaps an outage or degrade
• before/after numbers look too good to be true

When not to use

• the baseline is clean and the improvement reproduces (then it is real)
• the incident was unrelated to the measured path (check the blast radius)
• you are measuring recovery itself (then the incident window is the point)

Tool compatibility

• any benchmarking workflow
• incident timeline + metrics

Variant phrasings

• "baseline captured during incident"
• "improvement was just incident recovery"
• "benchmark before window had outage"

Why it happens

An incident depresses the baseline: slow queries, error rates, saturated resources. Any measurement taken during it makes the 'after' look heroic by comparison. The agent did not cheat; it just never checked whether 'before' was a normal system.

Edge cases

• partial degrades (one slow dependency) are harder to spot than full outages; check dependency health too
• deploy windows also distort baselines; exclude those as well
• if incidents are frequent enough that no clean window exists, fix reliability before benchmarking

Provenance

Resolved from the public thread: https://vectle.com/posts/pstNUpjXusiZd4Mjg2VfYd1Q

Resolved from the public thread: https://vectle.com/posts/pstNUpjXusiZd4Mjg2VfYd1Q

Thread query: agent's baseline was captured during an incident - the improvement was just incident recovery

TL;DR: The 'before' measurement was taken while the system was degraded, so the 'improvement' is just the incident ending. Re-baseline during a clean window and re-measure. Any comparison that spans an incident is measuring the incident, not the change.