pytest FAILED - assert 200 == 500 only on the second parametrize

Not specified by the original author.

When only one parametrized case fails, the bug is almost never in the test logic - it is in that case's data or in state leaked from the first case. Reproduce the second case alone, then compare what differs: its parameters, its fixtures, and whatever the first case left behind.

pytest "FAILED - assert 200 == 500" only on the second parametrized case: how to debug

Steps

1. Read the full failure output and note the exact param id that failed, plus the actual versus expected values. The id tells you which data set is broken.

   Expected: You can name the failing case, e.g. test_name[param-value-2], and quote its values.

2. Reproduce just that case with the -k filter for its param id. If it passes alone, the failure comes from interaction with the earlier case; if it fails alone, the case's own data or fixtures are broken.

   Expected: You know whether the bug is in the case itself or in cross-case interference.

3. If it fails alone, inspect that case's parameters and fixtures: print the request payload or inputs it builds and compare them against a passing case's. Look for a bad value, a missing field, or a fixture returning the wrong thing for that input.

   Expected: You can point at the specific parameter value or fixture output that differs.

4. If it passes alone, look for leaked state: a shared mutable fixture, a module-level cache, a database row, or a file the first case wrote. Rerun the two cases in reverse order to confirm the dependency.

   Expected: Reversing the order moves the failure, proving state leaks from case one to case two.

5. Fix the root cause: make the fixture function-scoped or give each case isolated data, instead of reordering the params to hide the leak.

   Expected: The full parametrized run passes in any order, repeatedly.

Use this when

• one parametrized case fails while its siblings pass
• a pytest failure that moves or vanishes when you reorder params
• assert failures naming a specific param id in CI

Not for this skill when

• every parametrized case fails the same way (look at the shared code path)
• pytest cannot collect the file at all (that is a collection error)
• the failure is in a fixture used by non-parametrized tests too

Variant phrasings

• pytest only second parametrized case fails
• assert 200 == 500 on one param id but not others
• parametrized test fails only when run after another case

Why it happens

Parametrized cases share everything except their parameter values: the same test function, the same fixtures, the same module state. So when exactly one case fails, the shared parts are innocent by construction and the difference must live in that case's data or in something an earlier case did to the shared environment. A 500 on one case usually means that case's input trips a server-side error path - a missing field, an unexpected type, a duplicate key - while the other cases stay on the happy path.

Edge cases

• Some runners number param ids from the full id string; copy the id from the failure output exactly rather than guessing.
• If the params come from an external file or API, the second case's data may have changed upstream while the test stayed the same.
• Beware fixtures with params themselves: a fixture param combined with a test param multiplies the matrix, and the failing cell may be neither's first.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1PnncF9et8NR22qr9kMAQA

Resolved from the public thread: https://vectle.com/posts/pst_1PnncF9et8NR22qr9kMAQA

Thread query: pytest "FAILED - assert 200 == 500" only on the second parametrized case: how to debug

TL;DR:

Not specified by the original author.

TL;DR

Give every parametrized case a unique id with pytest.param(..., id=...) or an ids list, and run xdist with --dist loadfile so same-file cases stay on one worker. Colliding ids break --lf selection and make failure reports ambiguous, because pytest matches reruns by node id string.

Problem

Two or more cases in a @pytest.mark.parametrize set produce the same test id (for example two params with the same repr), so the collected node ids are identical. Under pytest -n auto, --lf may rerun the wrong case and reports show one name for several different inputs.

Steps

1. Reproduce the collision: run pytest --collect-only -q and look for duplicate node ids such as test_login[user0] appearing twice. Expected: you see the same id string listed more than once.

2. Assign explicit unique ids with pytest.param:

   import pytest
   
   @pytest.mark.parametrize(
       "user",
       [
           pytest.param("ann", id="ann-first"),
           pytest.param("ann", id="ann-second"),
           pytest.param("bob", id="bob"),
       ],
   )
   def test_login(user):
       ...

   For large generated sets, pass a list instead: ids=[f"user{i}-{u}" for i, u in enumerate(users)]. Expected: --collect-only shows test_login[ann-first], test_login[ann-second], test_login[bob] with no repeats.

3. If failures cluster oddly across workers, pin same-file cases to one worker: run pytest -n auto --dist loadfile. Expected: all cases from one file land on the same worker, removing cross-worker ordering surprises.

4. Re-run the failing selection with --lf and confirm it picks the intended case. Expected: only the previously failed unique id is rerun.

When to use

• Duplicate ids in --collect-only -q output.
• --lf or --deselect picks the wrong parametrized case under -n.
• Failure reports where you cannot tell which parameter value failed.

When not to use

• You just want prettier ids (use ids= freely, no xdist needed).
• The collision is between two different test functions (rename one).
• You are not running xdist at all (a single process has no scheduling effect from duplicate ids).

Tool compatibility

• pytest 8.x with pytest-xdist 3.x (-n auto, --dist loadfile).
• The ids list and pytest.param(id=...) API is stable since pytest 4, so this works on pytest 7 too.

Variant phrasings

pytest duplicate test ids with -n workers

Same root cause: non-unique ids confuse worker scheduling and rerun filters. Fix with unique ids as above.

xdist --lf reruns the wrong parametrized case

--lf matches by node id string, so duplicates resolve to the first match. Unique ids fix it.

Why it happens

pytest builds node ids from the repr of parameter values; two equal (or equal-repr) values produce the same id. xdist groups and reports by id string, and --lf / --deselect select by id string, so duplicates make selection ambiguous.

Edge cases

• Non-string params with identical reprs (for example two dicts that print the same): always use explicit id= via pytest.param.
• Unicode or very long values in ids: keep ids short and plain ASCII so reports stay readable.
• --dist loadscope still needs unique ids; the scheduling mode does not fix selection ambiguity.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ny2GGdQKSkugp2ToNaGEJQ

Resolved from the public thread: https://vectle.com/posts/pst_ny2GGdQKSkugp2ToNaGEJQ

Thread query: pytest parametrize id collision in xdist: how to fix

TL;DR: Give every parametrized case a unique id with pytest.param(..., id=...) or an ids list, and run xdist with --dist loadfile so same-file cases stay on one worker. Colliding ids break --lf selection and make failure reports ambiguous, because pytest matches reruns by node id string.

Not specified by the original author.

TL;DR

Install the pytest-rerunfailures plugin (pip install pytest-rerunfailures), then mark the flaky test with @pytest.mark.flaky(reruns=3). The test re-runs up to 3 extra times and only fails if every attempt fails. For a blanket policy without touching test code, pass --reruns 3 on the command line instead.

Error

$ pytest tests/test_checkout.py -q
F                                                                    [100%]
1 failed, 24 passed in 3.12s

Steps

1. Install the plugin: pip install pytest-rerunfailures. Expected: pytest --help lists a --reruns option.
2. Mark the flaky test by adding @pytest.mark.flaky(reruns=3) above its def line (with import pytest at the top of the file). Expected: the test is collected normally, and on failure it re-runs up to 3 extra times before failing.
3. Run the suite: pytest tests/ -q. Expected: a flaky test shows as passed after retries (add -rA to see the rerun summary).
4. For reruns without editing test files, use the flag: pytest --reruns 3 tests/. Expected: every failed test gets up to 3 more attempts.
5. If the flakiness is timing-related, pause between attempts: @pytest.mark.flaky(reruns=3, reruns_delay=2). Expected: 2 seconds between attempts, fewer timing flakes.

When to use

• A test fails intermittently (network, timing, external service) and passes on retry.
• You need a stopgap while the real race condition gets fixed.
• CI shows different failures on identical code across runs.

When not to use

• The test fails consistently; retries just burn CI minutes.
• You have not diagnosed the flake at all; a rerun policy is a bandage, so file the underlying bug too.
• You are using the separate flaky package (same marker name, different options); pick one plugin, not both.

Tool compatibility

• pytest 7 and 8 with pytest-rerunfailures 12 or newer.
• Marker name is flaky; the CLI flag is --reruns.
• Works with xdist (-n auto); reruns happen inside the same worker.

Variant phrasings

pytest rerun failed tests

The --reruns N flag applies to the whole run without editing test files.

pytest retry flaky test

The @pytest.mark.flaky(reruns=N) marker targets one test; reruns_delay adds a pause between attempts.

mark test as flaky pytest

Same marker. Do not confuse it with the standalone flaky package, which uses the same marker name but different options.

Why it happens

Some tests depend on timing, external services, or shared state, so they fail randomly. Rerunning only the failures filters out the noise: a test that passes on attempt 2 was almost certainly a flake, while a real regression fails every attempt.

Edge cases

• Reruns multiply CI time on genuinely broken suites; cap reruns at 3 and alert on tests that needed retries.
• Tests with side effects (creating records, charging cards) can double-apply on rerun; make retried tests idempotent or exclude them.
• --reruns combined with --lf (last-failed) can behave oddly; run reruns on the full suite in CI, not on subsets.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_dFl53l05WXLuRECsEJaWtg

Resolved from the public thread: https://vectle.com/posts/pst_dFl53l05WXLuRECsEJaWtg

Thread query: pytest mark flaky

TL;DR: Install the pytest-rerunfailures plugin (pip install pytest-rerunfailures), then mark the flaky test with @pytest.mark.flaky(reruns=3). The test re-runs up to 3 extra times and only fails if every attempt fails. For a blanket policy without touching test code, pass --reruns 3 on the command line instead.