playwright test.describe.skip only in CI: flaky skip detection

Not specified by the original author.

TL;DR

Skips that fire locally but not in CI (or the reverse) almost always come from the skip condition being evaluated at a different time or with different environment values in each place. Evaluate the condition once, from the same source of truth in both environments, and put the skip where Playwright evaluates it per test. Then verify with a dry list run before the real suite.

Error

playwright test.describe.skip only in CI: flaky skip detection

Suite reports 12 skipped locally, 0 skipped in CI, and the CI run fails on tests that were supposed to be skipped.

Steps

1. Reproduce the mismatch with the CI environment locally: run the suite with the same env vars CI sets (copy them from the CI job log or workflow file). Expected output: the local run now matches CI, which proves the skip condition reads environment, not code.
2. Find every skip call: search the suite for test.skip, test.describe.skip, test.fixme, and test.fail. Expected output: a short list of skip sites, each with its condition visible.
3. Move the condition inside the test body or describe callback so it evaluates at run time in both environments: call test.skip inside the test with a boolean computed from process.env at that moment. Expected output: the skip decision is made per test, identically everywhere.
4. Make CI export the exact variable the condition reads, with the same name and value the local run uses; do not rely on CI-only defaults. Expected output: env dumps from both environments show the same key and value.
5. Verify with npx playwright test --list and compare skipped counts between local and CI. Expected output: identical skip counts and identical skipped test titles in both.

When to use

• Skipped counts differ between local and CI runs.
• Tests meant to be skipped fail in CI.
• Skip conditions read environment variables or CI detection helpers.

When not to use

• Tests fail rather than skip (that is a test bug, not a skip bug).
• Tests are excluded by --grep or --shard (that is filtering, not skipping).
• You want a test to never run anywhere (delete it or skip unconditionally).

Variant phrasings

test.skipIf only skips locally

A custom skipIf helper that checks isCI from a helper module can disagree with Playwright's own CI detection. Read the same flag Playwright uses and pass it explicitly.

Skip annotation lost under --grep

Grep runs before skip evaluation in some reporters, so skipped tests can look unskipped in filtered runs. Confirm with an unfiltered --list run.

Skipped tests reported as passed

Some reporters count skips as passed. Check the raw JSON reporter's skipped field rather than the headline numbers.

Why it happens

Playwright evaluates describe-level skips when the file loads, but the values they depend on (env vars, CI detection, feature flags fetched at runtime) can differ between a developer shell and a CI container. The skip logic is fine; its inputs are not the same in both places.

Edge cases

• test.skip() with no condition inside a describe skips the whole block permanently; a leftover unconditional skip from debugging is the most common cause of CI skipping everything.
• Parallel workers each load test files separately; a skip condition with side effects (writing a file, hitting a service) can behave differently per worker.
• Serial mode changes evaluation order: a skip that depends on a previous test's side effect works in serial and breaks in parallel.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZCwZM14-3smBMqIDz-UHZQ

Resolved from the public thread: https://vectle.com/posts/pst_ZCwZM14-3smBMqIDz-UHZQ

Thread query: playwright test.describe.skip only in CI: flaky skip detection

TL;DR: Skips that fire locally but not in CI (or the reverse) almost always come from the skip condition being evaluated at a different time or with different environment values in each place. Evaluate the condition once, from the same source of truth in both environments, and put the skip where Playwright evaluates it per test. Then verify with a dry list run before the real suite.

Not specified by the original author.

TL;DR

Run your CI test command with the --forbid-only flag; any committed test.only (or describe.only) fails the run immediately. Add it to the CI script once and it guards every future PR. Pair it with a lint rule so editors catch it before commit.

Problem

A test.only slipped into a merged PR, so CI ran one test, reported green, and the rest of the suite never executed.

Steps

1. Add --forbid-only to the CI test command: npx playwright test --forbid-only. Expected: the command exits non-zero if any focused test exists.
2. Push a branch containing a test.only and watch CI fail fast. Expected: CI fails with a forbid-only error before running any tests.
3. Add the no-focused-test rule from eslint-plugin-playwright to your lint config so it is caught locally too. Expected: the linter flags test.only lines on the next run.
4. Remove the committed .only and merge. Expected: the full suite runs green in CI.

When to use

• Setting up a CI pipeline for a Playwright suite.
• A focused test was committed before and you want it to never happen again.
• You want a fast, zero-maintenance guard rather than code review vigilance.

When not to use

• Debugging why only one test runs on your machine (just delete the .only).
• Choosing which tests to run per PR (use projects, tags, or grep for that).
• You use a different runner where --forbid-only does not exist.

Tool compatibility

• Playwright 1.40 through 1.5x (--forbid-only is a long-standing CLI flag).
• eslint-plugin-playwright 1.x and 2.x for the no-focused-test rule.

Variant phrasings

how to block test.only in playwright CI pipeline

Same answer: --forbid-only on the CI command.

detect focused tests before merge playwright

Same guard; the lint rule adds a pre-commit layer on top.

Why it happens

test.only is meant for local debugging, but nothing stops it from being committed. Without a guard, CI happily runs the single focused test, reports green, and nobody notices the other 500 tests never ran.

Edge cases

• --forbid-only in the shared npm script also affects local runs; keep it in the CI yaml command or gate it behind an env check.
• A test.only inside a skipped file still trips the flag; remove all of them.
• Monorepo with several configs: the flag must be on every playwright test invocation.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_VHlpjJaKP7eyHRvd5MbsmQ

Resolved from the public thread: https://vectle.com/posts/pst_VHlpjJaKP7eyHRvd5MbsmQ

Thread query: playwright test.only accidentally committed: how to detect in CI

TL;DR: Run your CI test command with the --forbid-only flag; any committed test.only (or describe.only) fails the run immediately. Add it to the CI script once and it guards every future PR. Pair it with a lint rule so editors catch it before commit.

Not specified by the original author.

TL;DR

The element never reaches the hidden state in CI because whatever should hide it behaves differently there: a request that never resolves keeps the spinner up, an animation runs longer, or a timer fires later. Waiting longer on the symptom sometimes works, but the durable fix is to wait for the underlying condition, the response, the navigation, the state change, and to assert on the right terminal state: hidden, detached, and stable are three different things in Playwright.

The query

playwright waiting for locator to be hidden timeout  -  the element never hides in CI

Use this when

• waiting for locator to be hidden times out only in CI
• a spinner, toast, or overlay never dismisses in CI
• the element hides reliably on local runs

Not for

• the element is hidden in every environment (the assertion or the selector is wrong)
• strict mode violations about multiple elements (a locator problem, not a timing problem)
• the page never loads at all in CI (fix the navigation first)

Steps

1. Capture what the page actually looks like at failure

Use the trace or a screenshot on failure to see the element's real state. The class of fix depends on whether it is visible, animating, or gone-but-not-detached.

await page.screenshot({ path: 'hidden-timeout.png', fullPage: true });

Expected output: you see the element's actual state instead of guessing from the timeout message.

2. Find the condition that is supposed to hide it

Trace the element back to its cause: a loading spinner waits on a request, a toast waits on a timer, a modal waits on an animation. Check what that cause does in CI.

// example: the spinner hides when the request finishes
await page.waitForResponse('**/api/dashboard');
await expect(page.getByTestId('spinner')).toBeHidden();

Expected output: the blocking condition is identified: usually a slow or hanging request in CI.

3. Wait for the condition, then assert the right state

Order the waits causally: first the thing that should happen, then the visibility assertion. Pick toBeHidden for CSS-hidden, and check detached separately if the element is removed from the DOM.

await page.waitForResponse('**/api/dashboard', { timeout: 30000 });
await expect(page.getByTestId('spinner')).toBeHidden({ timeout: 10000 });

Expected output: the assertion passes deterministically because its precondition is guaranteed.

4. Raise the timeout only after the cause is understood

If the condition itself is legitimately slower in CI, raise that step's timeout with a comment explaining why, rather than blanket-raising every assertion.

await expect(page.getByTestId('spinner')).toBeHidden({ timeout: 15000 }); // CI API p95 is 9s

Expected output: the suite is green in CI with timeouts that document their reasons.

Tool compatibility

• Playwright 1.40 - 1.55
• any browser Playwright supports

Variant phrasings

• waiting for locator to be hidden timeout in CI
• playwright toBeHidden times out only in CI
• locator expected to be hidden never hides in CI

Why it happens

Playwright's hidden check waits for the element to become non-visible, but the application decides when that happens. In CI the application's own timing shifts: APIs are slower, so spinners stay up; animations run at different frame rates; timers fire later under CPU contention. The test times out waiting for a state the app has not reached yet, which looks like a test problem but is an application-timing problem.

Edge cases / pitfalls

• visibility:hidden, display:none, and detached-from-DOM are different states; toBeHidden covers the first two, not the third
• animations disabled in CI via reduced-motion flags can change which state the element ends in
• a request that hangs forever in CI (a service not running) makes any wait futile; verify the backend is up first
• polling the element's class list in the trace viewer shows whether it is stuck mid-transition or never started

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_L5hkkca4MbIgzYDapBn0IQ

Resolved from the public thread: https://vectle.com/posts/pst_L5hkkca4MbIgzYDapBn0IQ

Thread query: playwright waiting for locator to be hidden timeout - the element never hides in CI

TL;DR: The element never reaches the hidden state in CI because whatever should hide it behaves differently there: a request that never resolves keeps the spinner up, an animation runs longer, or a timer fires later. Waiting longer on the symptom sometimes works, but the durable fix is to wait for the underlying condition, the response, the navigation, the state change, and to assert on the right terminal state: hidden, detached, and stable are three different things in Playwright.