playwright test.describe.skip only in CI: flaky skip detection

TL;DR

Skips that fire locally but not in CI (or the reverse) almost always come from the skip condition being evaluated at a different time or with different environment values in each place. Evaluate the condition once, from the same source of truth in both environments, and put the skip where Playwright evaluates it per test. Then verify with a dry list run before the real suite.

Error

playwright test.describe.skip only in CI: flaky skip detection

Suite reports 12 skipped locally, 0 skipped in CI, and the CI run fails on tests that were supposed to be skipped.

Steps

1. Reproduce the mismatch with the CI environment locally: run the suite with the same env vars CI sets (copy them from the CI job log or workflow file). Expected output: the local run now matches CI, which proves the skip condition reads environment, not code.
2. Find every skip call: search the suite for test.skip, test.describe.skip, test.fixme, and test.fail. Expected output: a short list of skip sites, each with its condition visible.
3. Move the condition inside the test body or describe callback so it evaluates at run time in both environments: call test.skip inside the test with a boolean computed from process.env at that moment. Expected output: the skip decision is made per test, identically everywhere.
4. Make CI export the exact variable the condition reads, with the same name and value the local run uses; do not rely on CI-only defaults. Expected output: env dumps from both environments show the same key and value.
5. Verify with npx playwright test --list and compare skipped counts between local and CI. Expected output: identical skip counts and identical skipped test titles in both.

When to use

• Skipped counts differ between local and CI runs.
• Tests meant to be skipped fail in CI.
• Skip conditions read environment variables or CI detection helpers.

When not to use

• Tests fail rather than skip (that is a test bug, not a skip bug).
• Tests are excluded by --grep or --shard (that is filtering, not skipping).
• You want a test to never run anywhere (delete it or skip unconditionally).

Variant phrasings

test.skipIf only skips locally

A custom skipIf helper that checks isCI from a helper module can disagree with Playwright's own CI detection. Read the same flag Playwright uses and pass it explicitly.

Skip annotation lost under --grep

Grep runs before skip evaluation in some reporters, so skipped tests can look unskipped in filtered runs. Confirm with an unfiltered --list run.

Skipped tests reported as passed

Some reporters count skips as passed. Check the raw JSON reporter's skipped field rather than the headline numbers.

Why it happens

Playwright evaluates describe-level skips when the file loads, but the values they depend on (env vars, CI detection, feature flags fetched at runtime) can differ between a developer shell and a CI container. The skip logic is fine; its inputs are not the same in both places.

Edge cases

• test.skip() with no condition inside a describe skips the whole block permanently; a leftover unconditional skip from debugging is the most common cause of CI skipping everything.
• Parallel workers each load test files separately; a skip condition with side effects (writing a file, hitting a service) can behave differently per worker.
• Serial mode changes evaluation order: a skip that depends on a previous test's side effect works in serial and breaks in parallel.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZCwZM14-3smBMqIDz-UHZQ