pytest coverage "Required test coverage of 90% not reached" failing only in CI: how to find the gap

CI measures a different set of files, or runs a different set of tests, than your laptop. The gap is not missing tests, it is a measurement mismatch: a subset run, a shard, or source files that exist in CI but not locally. Diff the measured file sets and the executed test sets before writing a single new test.

pytest coverage "Required test coverage of 90% not reached" failing only in CI: how to find the gap

Steps

1. Generate the coverage report in CI as an artifact (or print the missing-lines table) and compare the per-file numbers against your local report. Find which files lose coverage in CI.

   Expected: You have the list of files whose CI coverage is lower than local.

2. Check whether CI runs fewer tests: look for -k filters, deselections, sharding, or a fail-fast that aborts the run before the later test files execute.

   Expected: You know exactly which tests CI skips that your laptop runs.

3. Check whether CI measures more files: generated code, platform-specific modules, or vendored sources that exist in the CI checkout but not in your local tree will drag the percentage down.

   Expected: You find the files CI measures that your local run never sees.

4. Align the measurement: include or omit the same source paths in both configs, and make sure CI runs the full test set that the threshold was calibrated against.

   Expected: Local and CI now measure the same files over the same tests, and the numbers agree.

5. If a real gap remains after alignment, write tests for the uncovered lines the report names. Rerun CI to confirm the threshold passes with the aligned config.

   Expected: The CI gate passes and the report shows which lines the new tests cover.

Use this when

• coverage threshold passes locally but fails in CI
• CI and local coverage percentages disagree
• the coverage gate fails after sharding or subsetting tests

Not for this skill when

• coverage is below the threshold in both environments (write more tests)
• the coverage tool itself crashes or errors
• the threshold was just raised and nothing changed (calibrate it first)

Variant phrasings

• pytest coverage threshold failing only in CI
• required test coverage not reached CI but passes locally
• coverage mismatch CI vs local how to find gap

Why it happens

Coverage percentage is measured files divided by executed lines over total lines. CI and local can disagree on both halves: CI may run a subset of tests (shards, filters, fail-fast), which lowers the numerator, or it may measure extra files (generated code, platform modules), which raises the denominator. Either way the percentage drops without a single line of product code changing. Writing new tests before checking the measurement just adds coverage to files CI may not even be counting.

Edge cases

• Coverage config files can differ by environment: check for separate CI config, environment-variable-gated settings, and rc files the CI template injects.
• Parallel runs need their coverage data combined before the report; an uncombined shard reports its own slice as the whole.
• Omit generated and vendored code from measurement explicitly rather than covering it; thresholds should measure code humans maintain.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_fOJb6CwAFcdEBIHbT4rDXQ