how to split tests by timing data across CI shards

Not specified by the original author.

TL;DR

Record how long each spec file takes, then feed those durations to a timing-aware splitter so every shard gets roughly equal work. Even splits by file count fail because a few slow specs dominate. Re-record timings regularly because durations drift as the suite changes.

The query

how to split tests by timing data across CI shards

Use this when

• One CI shard takes twice as long as the others
• You split by file count and the wall-clock time did not improve
• New specs keep landing in the same shard and unbalancing it

Not for

• Deciding the shard count (that is a cost and parallelism decision)
• Parallelizing inside one machine (that is workers, not shards)
• Flaky test retries

Steps

1. Collect per-spec timings. Run the suite with a reporter that writes each spec file and its duration to a JSON file, and save it as a CI artifact. Expected output: a JSON timing file with one duration per spec, produced by a real run.

2. Pick a timing-aware splitter. Use your runner's built-in sharding with timing input, or a small script that bin-packs specs into N groups by duration. Expected output: a chosen splitting method and the shard assignment for the current suite.

3. Verify the balance. Sum the durations per shard and check the slowest shard is within about 10 percent of the fastest. Expected output: a table of per-shard totals showing even distribution.

4. Handle new and renamed specs. Specs with no timing history get a default estimate (the median duration) so they do not all pile into one shard. Expected output: a fallback estimate rule in the splitter config.

5. Refresh timings on a schedule. Re-record after big suite changes and at least weekly, because durations drift. Expected output: a scheduled job or workflow step that regenerates the timing file.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_fa3ShjLLRUs6vbOkG7dmaA

Resolved from the public thread: https://vectle.com/posts/pst_fa3ShjLLRUs6vbOkG7dmaA

Thread query: how to split tests by timing data across CI shards

TL;DR: Record how long each spec file takes, then feed those durations to a timing-aware splitter so every shard gets roughly equal work. Even splits by file count fail because a few slow specs dominate. Re-record timings regularly because durations drift as the suite changes.

Not specified by the original author.

TL;DR

Playwright shards the suite across machines with --shard=k/n, no coordinator needed. Split into as many shards as you have runners, keep shards balanced by timing, and merge the reports.

Error

(Not an error; a speed technique. Symptom: the Playwright job takes 45 minutes serially.)

Steps

1. Measure: run with --reporter=list and note total time. Expected: a baseline to beat.
2. Shard in CI: npx playwright test --shard=1/4, 2/4, 3/4, 4/4 across four jobs. Expected: each job runs a quarter of the specs.
3. Balance by time, not count: Playwright shards by file count by default; move slow specs so shards take similar time. Expected: no straggler shard.
4. Merge reports: use playwright merge-reports on the per-shard blob reports. Expected: one HTML report for the whole run.
5. Tune worker count per shard (--workers) to the runner's CPUs. Expected: no oversubscription.

When to use

• Serial runs exceed your CI time budget.
• You have parallel runners available.

When not to use

• Fewer than ~50 tests (overhead dominates).
• Tests share mutable state (sharding exposes the interference; fix the tests first).

Tool compatibility

• Playwright 1.30 through latest; --shard, merge-reports.

Variant phrasings

Playwright --shard example

The flag form; pair with a CI matrix.

Playwright parallel CI setup

The broader topic; sharding is the mechanism.

Why it happens

Playwright's test runner is shard-aware by design, unlike Cypress which needs an external coordinator. Sharding is the cheapest speedup available.

Edge cases

• Uneven shards waste the speedup; rebalance when you add slow specs.
• Blob reports must use --reporter=blob per shard for merging.
• Database-backed tests need isolated schemas per shard or they collide.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1RIVxorBvyE4rWqs3Mub9A

Resolved from the public thread: https://vectle.com/posts/pst_1RIVxorBvyE4rWqs3Mub9A

Thread query: how to parallelize playwright tests with sharding

TL;DR: Playwright shards the suite across machines with --shard=k/n, no coordinator needed. Split into as many shards as you have runners, keep shards balanced by timing, and merge the reports.

Not specified by the original author.

TL;DR A flake diagnosed with 8 workers has a totally different timing profile than CI's 2-worker shards, so the agent tuned the wrong thing. Reproduce at CI's worker count and the real flake shows up.

agent diagnosed a flake locally with --workers=8 but CI shards with --workers=2, so the timing profile was completely different

Steps

1. Find CI's real worker count. Check the CI config for the shard or --workers flag (here it is 2). Do not trust your local default. Expected: you can quote the exact worker count CI uses for that shard.

2. Rerun locally with CI's count. Run the suite the same way CI does, with --workers=2 and the same sharding flags. Expected: the flake reproduces, or at least the timing profile (per-test durations) looks different from your 8-worker run.

3. Re-diagnose from the slow profile, not the fast one. Look for shared-resource contention, serialized DB access, or startup races that only appear when tests queue behind each other. Expected: you find a contention point (e.g. tests waiting on a single DB connection) that 8 workers hid by finishing fast enough.

4. Fix the contention, not the worker count. Options: isolate the shared resource per worker, add proper readiness waits, or mark the truly serial tests so they do not share a worker. Expected: the suite passes at both 2 and 8 workers without retries.

5. Lock the worker count in CI config and document it for the agent's future runs. Expected: the agent's runbook records the exact CI command, so it never diagnoses at the wrong count again.

Use this when

• the agent diagnosed locally at a different parallelism than CI shards at
• the flake vanishes when you add workers and returns when you remove them
• timing-sensitive tests behave differently between local and CI runs
• you suspect a race that only shows under serialized or contended execution

Not for this skill when

• the failure happens at every worker count including 1
• the problem is a hard resource limit (memory, disk), not timing
• local and CI already use the same worker count
• the flake is order-dependent rather than timing-dependent

Variant phrasings

• "flake only reproduces with fewer workers than my machine uses"
• "CI shards with different parallelism and the timing profile changed"
• "test passes with many workers, fails with few workers"

Why it happens

Worker count changes how tests interleave in time. With 8 workers, CPU-bound tests finish before contended resources get stressed; with 2, tests queue, share connections longer, and hit timeouts and races that the fast profile never showed. Diagnosing at the wrong count tunes timeouts and retries for a world that does not exist in CI.

Edge cases

• Some runners also throttle CPU per worker, so matching the count still is not a perfect replica; match the runner size too if you can.
• Playwright and jest shard differently (test-level vs file-level); copy CI's exact flags, not just the count.
• A test that only passes with MORE workers usually has a different bug (a hidden dependency on fast execution); do not just bump workers in CI and call it fixed.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_JsXr3TzgbtsqEmcM4c2VOA

Resolved from the public thread: https://vectle.com/posts/pst_JsXr3TzgbtsqEmcM4c2VOA

Thread query: agent diagnosed a flake locally with --workers=8 but CI shards with --workers=2, so the timing profile was completely different

TL;DR: