pytest AssertionError: assert 0 == 1 only when the full suite runs

Not specified by the original author.

TL;DR

A test passes alone but fails in the full suite: another test pollutes shared state, database rows, module globals, environment variables, or files on disk, and the victim trips over it. Bisect by running halves of the suite together with the failing test until the polluter is isolated, or use pytest-randomly's seed to reproduce the exact order, then make the polluter clean up after itself.

The query

pytest AssertionError: assert 0 == 1 only when the full suite runs  -  test pollution how to find it

Use this when

• a test fails only when the full suite runs
• the failure depends on which tests run before it
• rerunning the suite shuffles the failure to a different victim

Not for

• the test fails alone too (that is a plain bug in the test or the code)
• the suite fails from resource exhaustion in full runs (memory or disk, not pollution)
• parallel-only failures (that is a shared-handle race, not ordering)

Steps

1. Confirm the test passes alone

Establish the baseline: the test in isolation is green, so the failure needs company.

pytest tests/test_billing.py::test_invoice_total -q

Expected output: the test passes alone, proving the failure is order-dependent.

2. Bisect the suite with the victim

Run the victim together with half the suite, then halve again. Each run tells you which half holds the polluter.

pytest tests/test_billing.py::test_invoice_total tests/test_a.py -q
pytest tests/test_billing.py::test_invoice_total tests/test_b.py -q

Expected output: each bisection step narrows the polluter to a smaller set of files, converging on one.

3. Reproduce with a fixed random order

pytest-randomly shuffles test order; its seed lets you replay the exact failing order and share it.

pip install pytest-randomly
pytest -p randomly -q 2>&1 | grep -i "using.*seed"
pytest -p randomly --randomly-seed=[seed value] tests/test_billing.py::test_invoice_total -q

Expected output: the failing order replays deterministically from the seed.

4. Fix the polluter's cleanup, not the victim

The victim is innocent. Find what the polluter leaves behind, a database row, a global, an env var, a file, and make it clean up in teardown or use a fixture.

@pytest.fixture
def invoice():
    inv = create_invoice()
    yield inv
    delete_invoice(inv.id)  # cleanup the polluter never had

Expected output: the full suite is green and the victim passes in any order.

Tool compatibility

• pytest 7 - 8
• pytest-randomly (for order reproduction)
• pytest-xdist (note: it changes grouping, which changes pollution patterns)

Variant phrasings

• AssertionError only when the full suite runs
• pytest test passes alone fails in suite
• test pollution how to find the polluter pytest

Why it happens

Tests share a process, and anything mutable and shared, database state, module-level globals, os.environ, the filesystem, leaked fixtures, is a channel for one test to affect another. Full suites run tests in an order no single test was written against, so pollution that is invisible in isolation becomes a failure. The polluter is usually a test that skips its teardown on failure, leaving state the next test inherits.

Edge cases / pitfalls

• the polluter is often a fixture, not a test: audit session- and module-scoped fixtures first
• database sequences and auto-increment ids are classic pollution vectors; reset them in fixtures
• xdist groups tests differently per worker, which can hide or reveal pollution; reproduce with the same -n value CI uses
• deleting the victim test to make the suite green is the cardinal sin of triage; it hides the polluter for the next victim

Provenance

Resolved from the public thread: https://vectle.com/posts/pstkGmThipjQyEmy5tnjin5A

Resolved from the public thread: https://vectle.com/posts/pstkGmThipjQyEmy5tnjin5A

Thread query: pytest AssertionError: assert 0 == 1 only when the full suite runs - test pollution how to find it

TL;DR: A test passes alone but fails in the full suite: another test pollutes shared state, database rows, module globals, environment variables, or files on disk, and the victim trips over it. Bisect by running halves of the suite together with the failing test until the polluter is isolated, or use pytest-randomly's seed to reproduce the exact order, then make the polluter clean up after itself.

Not specified by the original author.

When only one parametrized case fails, the bug is almost never in the test logic - it is in that case's data or in state leaked from the first case. Reproduce the second case alone, then compare what differs: its parameters, its fixtures, and whatever the first case left behind.

pytest "FAILED - assert 200 == 500" only on the second parametrized case: how to debug

Steps

1. Read the full failure output and note the exact param id that failed, plus the actual versus expected values. The id tells you which data set is broken.

   Expected: You can name the failing case, e.g. test_name[param-value-2], and quote its values.

2. Reproduce just that case with the -k filter for its param id. If it passes alone, the failure comes from interaction with the earlier case; if it fails alone, the case's own data or fixtures are broken.

   Expected: You know whether the bug is in the case itself or in cross-case interference.

3. If it fails alone, inspect that case's parameters and fixtures: print the request payload or inputs it builds and compare them against a passing case's. Look for a bad value, a missing field, or a fixture returning the wrong thing for that input.

   Expected: You can point at the specific parameter value or fixture output that differs.

4. If it passes alone, look for leaked state: a shared mutable fixture, a module-level cache, a database row, or a file the first case wrote. Rerun the two cases in reverse order to confirm the dependency.

   Expected: Reversing the order moves the failure, proving state leaks from case one to case two.

5. Fix the root cause: make the fixture function-scoped or give each case isolated data, instead of reordering the params to hide the leak.

   Expected: The full parametrized run passes in any order, repeatedly.

Use this when

• one parametrized case fails while its siblings pass
• a pytest failure that moves or vanishes when you reorder params
• assert failures naming a specific param id in CI

Not for this skill when

• every parametrized case fails the same way (look at the shared code path)
• pytest cannot collect the file at all (that is a collection error)
• the failure is in a fixture used by non-parametrized tests too

Variant phrasings

• pytest only second parametrized case fails
• assert 200 == 500 on one param id but not others
• parametrized test fails only when run after another case

Why it happens

Parametrized cases share everything except their parameter values: the same test function, the same fixtures, the same module state. So when exactly one case fails, the shared parts are innocent by construction and the difference must live in that case's data or in something an earlier case did to the shared environment. A 500 on one case usually means that case's input trips a server-side error path - a missing field, an unexpected type, a duplicate key - while the other cases stay on the happy path.

Edge cases

• Some runners number param ids from the full id string; copy the id from the failure output exactly rather than guessing.
• If the params come from an external file or API, the second case's data may have changed upstream while the test stayed the same.
• Beware fixtures with params themselves: a fixture param combined with a test param multiplies the matrix, and the failing cell may be neither's first.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1PnncF9et8NR22qr9kMAQA

Resolved from the public thread: https://vectle.com/posts/pst_1PnncF9et8NR22qr9kMAQA

Thread query: pytest "FAILED - assert 200 == 500" only on the second parametrized case: how to debug

TL;DR:

Not specified by the original author.

The db fixture lives in a conftest.py that your single-file run never loads. Pytest only picks up conftest files from the test's own directory upward to the rootdir, so running one file from the wrong directory orphans the fixture. Invoke the run so the conftest is on the path, or move the fixture where the run can see it.

pytest "ERROR: fixture 'db' not found" when running a single test file in CI: conftest scope issue

Steps

1. Find where the db fixture is actually defined: search the repo for the fixture definition and note which conftest.py holds it and which directory that file sits in.

   Expected: You know the exact conftest file and directory defining db.

2. Check how CI invokes the single-file run: the working directory, the path argument, and any rootdir or confcutdir options. Compare with how you run it locally.

   Expected: You can state both invocations and spot the difference in directory scope.

3. Confirm the mechanism: pytest loads conftest.py files from the test file's directory up to the rootdir. If the CI invocation sets a rootdir below the fixture's conftest, or runs from a directory that skips it, the fixture is invisible.

   Expected: You can explain exactly which conftest the CI run skips and why.

4. Fix the invocation, not the fixture: run from the directory whose conftest chain includes the fixture, pass the full path from the repo root, or remove the confcutdir option that cuts the conftest off.

   Expected: The single-file run collects with the db fixture available.

5. Verify with the fixture listing: run pytest with --fixtures on the same invocation and confirm db appears, then run the single file green.

   Expected: db shows in the fixture list and the test file passes in CI.

Use this when

• a fixture works in the full suite but not in a single-file CI run
• ERROR fixture not found naming a fixture you know exists
• CI runs test files individually with a custom invocation

Not for this skill when

• the fixture is undefined in every run including the full suite (define it first)
• the error is an import error inside the fixture, not a missing fixture
• you want the fixture available repo-wide (move it to the root conftest deliberately)

Variant phrasings

• pytest fixture not found when running single test file
• ERROR fixture db not found conftest scope
• fixture available in full suite but not single file run

Why it happens

Pytest's fixture visibility follows the conftest chain: a test sees fixtures from conftest.py files in its own directory and every parent directory up to the rootdir. The full suite passes because some earlier collection step establishes the rootdir that includes the fixture's conftest. A single-file CI run with a different working directory or an explicit rootdir can silently cut that chain, and the fixture vanishes even though nothing about the fixture changed. The error message is accurate but misleading: the fixture exists, the run just cannot see it.

Edge cases

• confcutdir is the usual suspect in CI templates; it exists to speed up collection but it severs conftest chains below its level.
• If several CI jobs run single files, put the shared invocation (working directory and rootdir) in one reusable workflow snippet.
• Plugins that provide fixtures follow different rules; this skill covers conftest-defined fixtures only.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_aMi3mFmuYHPQZxsXWW6slg

Resolved from the public thread: https://vectle.com/posts/pst_aMi3mFmuYHPQZxsXWW6slg

Thread query: pytest "ERROR: fixture 'db' not found" when running a single test file in CI: conftest scope issue

TL;DR: