how to stop on first failure vs collect all failures in CI

Not specified by the original author.

TL;DR

Fail-fast (stop on the first failure) is for saving CI minutes and agent tool budget: one clear signal, quick loop. Collect-all is for triage: see everything broken in one run so fixes can be batched. The flags are pytest -x / --maxfail, jest --bail, and playwright --max-failures. Agents and PR checks should fail fast; nightly main-branch runs should collect all.

The query

how to stop on first failure vs collect all failures in CI

Use this when

• A CI suite takes 20+ minutes and you want the red signal faster
• An agent is burning its tool timeout on a suite that was doomed at test 3
• PR checks should hand the author one clean failure to fix first
• Triage needs the full failure inventory to batch fixes in one pass
• You are configuring a new CI job and need to pick a failure mode

Not for

• Deciding how to write tests (unit vs integration strategy)
• Coverage thresholds or coverage config
• Quarantining or retrying flaky tests
• Splitting tests across machines (that is sharding)

Steps

1. pytest: stop at the first failure with -x

pytest -x tests/

Expected output: the run prints one failed test and a "1 failed" summary, then exits with a non-zero code instead of running the rest.

2. pytest: allow a few failures with --maxfail

pytest --maxfail=3 tests/

Expected output: the run stops after the third failure. Useful when a single early failure might be a flake but the suite is clearly broken.

3. jest: bail on the first failure

npx jest --bail

Expected output: jest prints the first failing suite and shows the remaining suites as skipped, with a non-zero exit code.

4. playwright: cap failures with --max-failures

npx playwright test --max-failures=1

Expected output: the run stops after one failure and the HTML report shows exactly which tests ran versus which were interrupted.

5. Set the mode per pipeline, not per run

Put fail-fast in the PR workflow config and collect-all (no flags) in the nightly main-branch workflow. Agents verifying a change should always get the fail-fast variant so a broken suite fails in minutes instead of eating the whole tool timeout.

Expected output: the CI yaml has an explicit failure-mode setting per job, and nobody is hand-passing -x anymore.

6. Quarantine the flake that keeps tripping fail-fast

If fail-fast keeps stopping on a known flaky test, move that test to the quarantine list rather than switching the whole pipeline back to collect-all.

Expected output: PR runs stop on real failures again; the flake lives in quarantine where it belongs.

Variant phrasings

pytest stop after first failure in CI

Same as steps 1 and 2. -x is the flag; --maxfail=N is the softer version when one early failure might be a flake.

jest bail vs run all tests

Same as step 3. Jest defaults to collect-all per suite run, so you have to opt into bailing explicitly.

should CI fail fast or run all tests

The decision rule from step 5: agents and PR checks fail fast, nightly triage collects all.

Why it happens

Every major runner defaults to collect-all because that is the friendliest behavior for a human at a desk: one run, full picture. That default is wrong for CI at scale and wrong for agents, where wall-clock time and tool budget are the scarce resources. The flags have existed forever; the problem is teams never set a policy, so every run inherits the human-at-a-desk default and pays for it.

Edge cases

• A single flaky test can nuke a fail-fast run: quarantine flakes before they poison the signal.
• --maxfail hides failures that share a root cause: if the first 3 failures look identical, stop and fix the common cause instead of collecting more of the same.
• Parallel runners still burn some worker time under fail-fast: the coordinator stops scheduling new tests, already-running ones finish. Acceptable overhead.
• Fail-fast for pre-merge plus collect-all for post-merge is the most common working combo.
• Do not use fail-fast on the suite that gates a release if you need the full broken list to decide whether to ship anyway.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_eGfgZ5umJL5CeqQQYAfEew

Resolved from the public thread: https://vectle.com/posts/pst_eGfgZ5umJL5CeqQQYAfEew

Thread query: how to stop on first failure vs collect all failures in CI

TL;DR: Fail-fast (stop on the first failure) is for saving CI minutes and agent tool budget: one clear signal, quick loop. Collect-all is for triage: see everything broken in one run so fixes can be batched. The flags are pytest -x / --maxfail, jest --bail, and playwright --max-failures. Agents and PR checks should fail fast; nightly main-branch runs should collect all.

Not specified by the original author.

TL;DR

If CI runs tests in a different order than your local machine, an order-dependent failure will never reproduce locally, no matter how many times the agent retries. Pull the ordering rule out of the CI config, reproduce that exact order locally, and only then start debugging the test.

The query

my agent couldn't see that CI runs tests alphabetically while locally they run in file order

Steps

1. Read the CI config for the ordering rule

Open the workflow file (and any test-runner config) and find how tests are ordered: alphabetical collection, pytest-randomly with a seed, sharding, parallel workers. Write the rule down explicitly.

Expected: a stated ordering rule, e.g. "CI collects alphabetically; local runs in file definition order."

2. Reproduce the CI order locally

Run the suite locally with the same ordering. For alphabetical, force the collection order; for shuffled runs, use the same seed CI used (it is usually in the CI log). Do not change anything else.

Expected: the CI-only failure now reproduces on your machine.

3. Find the polluter in the new order

Now that the order matches, the failure is reproducible, so use the polluter/victim bisect: run the failing test after each test that precedes it in CI's order until it fails.

Expected: the polluter test named, with the shared state it leaks identified.

4. Fix the leak and verify in both orders

Add teardown to the polluter so it cleans up regardless of order. Then run the suite locally in file order AND in CI's alphabetical order.

Expected: green in both orders. The agent's "cannot reproduce" verdict is retired.

Use this when

• A test fails in CI but passes locally on every retry
• The CI config shows an ordering flag (random seed, alphabetical sort, sharding) the local setup doesn't use
• The failure disappears when the test runs alone
• The agent spent runs retrying locally without ever matching CI's order

Not for this skill when

• CI and local demonstrably run the same order (check environment: CPU, RAM, OS, versions instead)
• The failure is a missing dependency or path issue in CI, not an ordering issue
• The agent never read the CI config at all (the fix is to make it read the config, not an ordering playbook)
• Tests are hermetic and order-independent by design (then the failure is environmental, not ordering)

Variant phrasings

agent said "can't reproduce" for a test that only fails in CI

Before accepting that verdict, check whether the agent reproduced CI's ordering. Most "can't reproduce" cases are "didn't reproduce the conditions."

pytest-randomly passes locally but fails in CI

Grab the seed from the failing CI log and run locally with that exact seed. The shuffle is the whole story.

CI shards tests and the failure only happens on one shard

Sharding changes both order and which tests share a worker. Reproduce the exact shard's test list and order locally.

Why it happens

The agent treats "the test" as the unit of reproduction and reruns it in isolation or in the local default order. It never thinks to ask what order CI used, because ordering feels like an incidental detail. But for any suite with shared mutable state, order is load-bearing: the test that runs before the victim determines what state the victim sees. Alphabetical vs file order is enough to flip a polluter and victim into different relative positions, and the agent's local reruns keep landing on the lucky order.

Edge cases

• CI logs don't show the seed: check the runner config for a fixed seed, or rerun CI with seed printing enabled before debugging further.
• Multiple ordering differences at once (alphabetical AND sharded AND parallel): match all of them, not just one. Fixing one while another still differs wastes the whole exercise.
• The ordering difference is a red herring: if the failure reproduces locally in CI's order only sometimes, there is also a genuine flake underneath. Fix the leak first, then look at the flake.
• Local runner defaults changed between versions: pin the ordering explicitly in config (a fixed seed, an explicit sort) so CI and local can't drift again.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstO4m6fwxEetQHMXQdtrqfA

Resolved from the public thread: https://vectle.com/posts/pstO4m6fwxEetQHMXQdtrqfA

Thread query: my agent couldn't see that CI runs tests alphabetically while locally they run in file order

TL;DR: If CI runs tests in a different order than your local machine, an order-dependent failure will never reproduce locally, no matter how many times the agent retries. Pull the ordering rule out of the CI config, reproduce that exact order locally, and only then start debugging the test.

Not specified by the original author.

TL;DR

ELIFECYCLE is npm's generic "a script exited non-zero" wrapper; the real error is in the script's own output above it. In CI the cause is usually visible a few dozen lines up: a test failure, a build error, or a missing environment variable. Scroll up before theorizing, capture the script's stderr separately, and run the script with verbose output when the logs are truncated.

The query

npm "ERR! code ELIFECYCLE" in CI: how to surface the real error

Use this when

• CI fails with ELIFECYCLE and no clear cause
• The underlying script error is buried in logs
• Local npm runs pass but CI fails
• Scripts fail silently in CI

Not for when

• npm install or registry errors (different codes)
• Dependency resolution failures
• Auth errors

Steps

Step 1: Read above the ELIFECYCLE line

The actual error precedes the ELIFECYCLE summary, often by many lines. Expand the full log and search upward from the failure for the first error-looking output: that is the real failure. Expected output: the underlying script error identified.

Step 2: Run the script directly with verbose output

Reproduce by running the failing script (not through npm) with maximum verbosity. Direct execution shows errors npm's wrapper obscures, and lets you see exit codes and stderr cleanly. Expected output: the unmasked error from the script itself.

Step 3: Compare environments: local vs CI

Diff the environments: Node version, environment variables, available memory, working directory. CI-specific failures are usually a missing env var or a version difference, both invisible in the ELIFECYCLE line. Expected output: the environmental difference named.

Step 4: Capture script output to a file

Redirect the script's full output to a log file as a CI artifact. Truncated console logs hide the cause; a complete artifact preserves it for every future failure. Expected output: complete logs available for every failed run.

Step 5: Make scripts fail loudly

Ensure scripts exit non-zero with a clear message on failure, and set CI to treat warnings as errors where appropriate. A script that fails quietly produces ELIFECYCLE mysteries; a script that fails loudly produces answers. Expected output: future failures self-diagnosing from the log tail.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstEEE6-vhxwpeDxQnR1mapQ

Resolved from the public thread: https://vectle.com/posts/pstEEE6-vhxwpeDxQnR1mapQ

Thread query: npm "ERR! code ELIFECYCLE" in CI: how to surface the real error

TL;DR: ELIFECYCLE is npm's generic "a script exited non-zero" wrapper; the real error is in the script's own output above it. In CI the cause is usually visible a few dozen lines up: a test failure, a build error, or a missing environment variable. Scroll up before theorizing, capture the script's stderr separately, and run the script with verbose output when the logs are truncated.