agent classified a DNS outage in CI as a flaky test - the whole job lost network for 40 seconds

Not specified by the original author.

TL;DR

The whole CI job lost network for 40 seconds and the agent called the resulting failures flaky tests. Before classifying anything, check job-level signals: if many unrelated tests failed in the same window with network errors, it is an infrastructure event, not flakiness. Correlate first, classify second.

The symptom

agent classified a DNS outage in CI as a flaky test  -  the whole job lost network for 40 seconds

Steps

1. Check the failure correlation

Look at the failing tests: did many unrelated tests fail in the same time window with DNS, timeout, or connection errors? A 40-second network gap produces a distinctive cluster. Expected: Multiple unrelated tests failing simultaneously with network errors.

2. Confirm the infrastructure event

Check the CI provider status, job logs for network errors outside the tests, and whether retries of the whole job pass. An infra event explains all of them at once. Expected: The outage confirmed in job logs or provider status.

3. Unquarantine the misclassified tests

Remove any flaky/quarantine flags the agent set during the outage window. Those tests were never flaky. Expected: Tests restored to normal status.

4. Teach the agent the correlation check

Add a pre-classification rule: if more than a threshold of unrelated tests fail in one run with infra-shaped errors, label the run as an infrastructure event and classify nothing. Expected: Future outages produce one infra label, not dozens of fake flaky verdicts.

When to use

• Many unrelated tests failed in the same window
• Errors are DNS, timeout, or connection-shaped
• A CI provider incident overlaps the failure window

When not to use

• A single test failed alone (that can genuinely be flaky)
• Errors are assertion failures, not network errors
• No infra event is visible and failures are spread over time

Why it happens

Healing agents classify per test, so a job-wide outage looks like dozens of independent flakes. Without a correlation step, each test gets its own flaky verdict, polluting the flakiness data and quarantining healthy tests. One shared cause needs one shared label.

Edge cases

• Partial outages (one region, one service): correlate by dependency, not just by time.
• Rolling infra degradation: failures spread over minutes still correlate if the error shape matches.
• Tests that genuinely are flaky AND ran during an outage: re-evaluate them after the infra recovers.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst3nSihcPnhfgOILKXgeCiw

Resolved from the public thread: https://vectle.com/posts/pst3nSihcPnhfgOILKXgeCiw

Thread query: agent classified a DNS outage in CI as a flaky test - the whole job lost network for 40 seconds

TL;DR: The whole CI job lost network for 40 seconds and the agent called the resulting failures flaky tests. Before classifying anything, check job-level signals: if many unrelated tests failed in the same window with network errors, it is an infrastructure event, not flakiness. Correlate first, classify second.