how to split tests by timing data across CI shards

TL;DR

Record how long each spec file takes, then feed those durations to a timing-aware splitter so every shard gets roughly equal work. Even splits by file count fail because a few slow specs dominate. Re-record timings regularly because durations drift as the suite changes.

The query

how to split tests by timing data across CI shards

Use this when

• One CI shard takes twice as long as the others
• You split by file count and the wall-clock time did not improve
• New specs keep landing in the same shard and unbalancing it

Not for

• Deciding the shard count (that is a cost and parallelism decision)
• Parallelizing inside one machine (that is workers, not shards)
• Flaky test retries

Steps

1. Collect per-spec timings. Run the suite with a reporter that writes each spec file and its duration to a JSON file, and save it as a CI artifact.

Expected output: a JSON timing file with one duration per spec, produced by a real run.

1. Pick a timing-aware splitter. Use your runner's built-in sharding with timing input, or a small script that bin-packs specs into N groups by duration.

Expected output: a chosen splitting method and the shard assignment for the current suite.

1. Verify the balance. Sum the durations per shard and check the slowest shard is within about 10 percent of the fastest.

Expected output: a table of per-shard totals showing even distribution.

1. Handle new and renamed specs. Specs with no timing history get a default estimate (the median duration) so they do not all pile into one shard.

Expected output: a fallback estimate rule in the splitter config.

1. Refresh timings on a schedule. Re-record after big suite changes and at least weekly, because durations drift.

Expected output: a scheduled job or workflow step that regenerates the timing file.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_fa3ShjLLRUs6vbOkG7dmaA