how to speed up a 40-minute CI pipeline

TL;DR

A 40-minute pipeline is slow because of serial work, cold caches, and oversized test suites, not because CI is inherently slow. Profile where the minutes actually go, then parallelize the test suite, cache dependencies and Docker layers, and split the pipeline so fast checks gate merges while slow checks run after. Most teams cut 50-70% with caching plus parallelism alone.

Error / query

how to speed up a 40-minute CI pipeline

Use this skill when

• End-to-end pipeline time is 30+ minutes and developers context-switch while waiting
• You do not know which jobs consume the time
• Builds redo dependency installs and Docker builds from scratch every run
• The test suite runs serially on one runner

Not for this skill when

• The pipeline is fast but flaky (reliability problem, different skill)
• A single step hangs rather than being slow (debug the hang, not the average)
• You are choosing a CI provider (vendor decision, not optimization)
• Queue time, not run time, is the bottleneck (runner capacity, different fix)

Steps

Step 1: Break down where the 40 minutes go

echo "List every job with its duration from the last 5 runs (CI UI or API)."
echo "Rank them; the top 2-3 jobs are your targets. Do not optimize the 2-minute lint job."

Expected: a ranked duration table. Typically one test job and one build job dominate; everything else is noise.

Step 2: Parallelize the test suite

echo "Split tests by timing data across N runners (e.g. pytest -n auto, jest --shard, go test -parallel)."
echo "Target: the longest shard, not the total test count."

Expected: wall-clock test time drops roughly with the shard count. Balance shards by historical duration, not by file count, or one slow shard eats the gain.

Step 3: Cache dependencies between runs

echo "Cache the package manager directory keyed on the lockfile hash (actions/cache, GitLab cache:, Buildkite cache)."
echo "Verify the cache actually restores: check the step log for a cache hit."

Expected: install steps drop from minutes to seconds on unchanged dependencies. A cache that never hits (wrong key, oversized cache) is just overhead, so confirm the hit.

Step 4: Cache Docker layers and avoid rebuilding what did not change

docker pull [registry]/[image]:latest || true
docker build --cache-from [registry]/[image]:latest --build-arg BUILDKIT_INLINE_CACHE=1 -t [image]:ci .

Expected: unchanged layers show CACHED and the image build takes a fraction of the cold time. Keep dependency layers before source-copy layers in the Dockerfile.

Step 5: Restructure the pipeline into fast-gate and slow-follow

echo "Run lint + unit tests on every push (merge gate, target under 10 min)."
echo "Move integration, e2e, and perf tests to post-merge or nightly."

Expected: developers get signal in minutes; slow suites still run but no longer block every iteration. Measure the new p50/p95 pipeline time for a week to confirm the gain holds.

Variant phrasings

"ci takes too long developers waiting"

Same playbook. Steps 1-2 usually recover the most time for the least work.

"how to make github actions faster"

Cache dependencies, use larger runners for the heavy job, parallelize with a matrix, and skip unchanged paths with path filters.

"test suite too slow in ci"

Shard it (step 2) and quarantine the slowest 5% of tests for optimization; a few pathological tests often dominate.

Why it happens

Pipelines accrete: every team adds a job, nobody removes one, caches are set up once and silently break, and test suites grow serially. The 40 minutes are usually 10 minutes of real work repeated inefficiently plus queueing between serial stages. Profiling first matters because the bottleneck is rarely where people guess.

Edge cases and pitfalls

• Parallelism has a ceiling: beyond it, runner contention and test-database conflicts make things slower, not faster.
• Caches keyed wrong (e.g. on branch name instead of lockfile hash) poison builds with stale dependencies; prefer exact-hash keys.
• Skipping CI on "docs only" changes is fine, but path filters that are too broad skip real checks; review them quarterly.
• Bigger runners cost more; compare the dollar cost of faster runners against developer wait time before upsizing everything.
• Do not delete coverage to gain speed; move slow suites out of the merge gate instead of dropping them.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_QSaE3knFLE-Bx1Hv9iDQrQ