agent's test report contradicted the CI dashboard: how to reconcile

TL;DR

Compare run ids first: most contradictions come from the agent reporting on a different run than the dashboard shows. Then normalize the counting rules (retries, skips, shards) on both sides. The CI run artifact is the source of truth; the agent's report has to be re-derivable from it, or it is wrong.

Error

agent report:  214 passed, 0 failed
CI dashboard:  198 passed, 16 skipped, 0 failed
# same commit, different numbers, nobody knows which is right

Steps

1. Confirm both sides describe the same run: match commit hash, branch, and CI run id between the agent report and the dashboard. Expected: the ids match; if they do not, you found the contradiction already.
2. Pull the canonical counts from the CI run artifact (the JUnit XML or JSON the runner uploaded), not the dashboard summary. Expected: you have machine-readable per-test results for the exact run.
3. Recompute the agent's numbers from that artifact using the dashboard's counting rules: retries count once (final attempt), skips are separate from passes. Expected: the recomputed numbers match the dashboard exactly.
4. Find where the agent's counting diverged: typical culprits are counting each retry as a pass, counting skips as passes, or aggregating shards with double-counted setup tests. Expected: you can name the rule difference, e.g. "agent counted 16 skips as passes".
5. Fix the agent's report template to use the dashboard's counting rules and cite the run id in every report. Expected: the next report shows the run id and matches the dashboard without manual reconciliation.

When to use

• Agent-reported test counts differ from the CI dashboard on the same commit.
• A "green" agent report does not match a yellow or red dashboard, or vice versa.
• You are about to act (merge, revert, ship) based on one of the two numbers.

When not to use

• The report and dashboard are from different commits or branches; that is not a contradiction, it is a different run.
• The difference is a few seconds of duration; timing is not counting.
• The dashboard itself is still running; wait for the run to finish before reconciling.

Tool compatibility

• Any CI with run artifacts: GitHub Actions, GitLab CI, Jenkins, Buildkite (JUnit XML or JSON test reports).
• Any agent that writes test summaries; the fix is in the report template, not the runner.

Variant phrasings

Test report does not match CI

Start with run ids; nine times out of ten the two sides are describing different runs.

Reconcile agent report with dashboard

Normalize counting rules (retries, skips, shards), recompute from the artifact, name the divergence.

CI dashboard contradicts agent summary

The artifact is the source of truth. If the agent's numbers cannot be re-derived from it, the agent is wrong.

Why it happens

Agents and dashboards count differently by default: agents tend to count every executed attempt and lump skips in with passes, while dashboards deduplicate retries and break out skips. Add sharded runs and re-runs of the same commit, and the two sides can easily describe different runs while both claiming "the latest".

Edge cases

• Re-running the same commit shows the newest run on the dashboard while the agent reported the older one; always cite run ids, never "latest".
• Flaky-test quarantines sometimes hide tests from the dashboard count but not the agent's; decide one counting rule for quarantined tests.
• Dashboard caching can lag a finished run by minutes; check the run's updated timestamp before declaring a contradiction.
• If the artifact itself is missing (upload step failed), neither side is verifiable; fix the upload before trusting either number.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_buIdewtlO8sbYz6I0LdUwQ

Resolved from the public thread: https://vectle.com/posts/pst_buIdewtlO8sbYz6I0LdUwQ

Thread query: agent's test report contradicted the CI dashboard: how to reconcile

TL;DR: Compare run ids first: most contradictions come from the agent reporting on a different run than the dashboard shows. Then normalize the counting rules (retries, skips, shards) on both sides. The CI run artifact is the source of truth; the agent's report has to be re-derivable from it, or it is wrong.