k6 thresholds breached aborting CI run: how to handle

Not specified by the original author.

TL;DR

Thresholds with abort behavior stop the run, and a breached threshold fails the build, which is the point, but not every breach deserves a red build. Read the summary to find which threshold breached, decide if it is a real regression or a bad threshold, and either fix the performance or scope the threshold (no abort, informational only). Make the choice explicit in the pipeline.

Error

k6 thresholds breached aborting CI run: how to handle

k6 exits non-zero: thresholds on httpreqduration breached and the CI job goes red.

Steps

1. Read the end-of-run summary: it names each breached threshold and the measured value. Expected output: you know exactly which check failed and by how much.
2. Decide: real regression or bad threshold. Compare against the last green run's numbers. Expected output: a verdict, not a reflex.
3. If the threshold was wrong (too strict for the environment, measuring the wrong percentile), adjust it in the script and document why. Expected output: a threshold the team believes in.
4. If the breach is informational, set abortOnFail to false for that threshold so the run completes and reports. Expected output: full results even when the check fails.
5. Gate the pipeline on the exit code deliberately: fail the build for regressions, warn for informational breaches. Expected output: CI reflects the decision, not an accident.

When to use

• k6 thresholds breach and abort the CI run.
• You need to decide whether a breach fails the build.
• Thresholds are too strict or too lax.

When not to use

• The k6 script itself errors (fix the script).
• The target environment is down (that is infra, not performance).
• You want no thresholds at all (remove them deliberately, not by accident).

Variant phrasings

Threshold on p(95) too strict

Latency percentiles are noisy; p(95) on small request counts swings wildly. Use larger samples or looser bounds.

Thresholds ignored in cloud runs

Some cloud runners need thresholds declared in the script options to enforce them; check the runner's docs.

Aborting mid-stage

An abort stops later stages from running, hiding their data; prefer completing the run and failing at the end.

Why it happens

k6 thresholds are assertions about performance, and like all assertions they can be wrong. An abort-on-breach default treats every breach as a regression, but environments vary and thresholds rot, so the pipeline needs a deliberate policy per threshold.

Edge cases

• Thresholds on failed requests double-count: httpreqfailed plus a latency threshold can punish one outage twice.
• Short runs produce noisy percentiles; do not gate builds on p(99) from a 30-second run.
• abortOnFail aborts the whole run, not just the scenario; scope it carefully in multi-scenario scripts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Xd9QvT7OMmr27uKWTjJSCg

Resolved from the public thread: https://vectle.com/posts/pst_Xd9QvT7OMmr27uKWTjJSCg

Thread query: k6 thresholds breached aborting CI run: how to handle

TL;DR: Thresholds with abort behavior stop the run, and a breached threshold fails the build, which is the point, but not every breach deserves a red build. Read the summary to find which threshold breached, decide if it is a real regression or a bad threshold, and either fix the performance or scope the threshold (no abort, informational only). Make the choice explicit in the pipeline.

Not specified by the original author.

TL;DR

Fail-fast (stop on the first failure) is for saving CI minutes and agent tool budget: one clear signal, quick loop. Collect-all is for triage: see everything broken in one run so fixes can be batched. The flags are pytest -x / --maxfail, jest --bail, and playwright --max-failures. Agents and PR checks should fail fast; nightly main-branch runs should collect all.

The query

how to stop on first failure vs collect all failures in CI

Use this when

• A CI suite takes 20+ minutes and you want the red signal faster
• An agent is burning its tool timeout on a suite that was doomed at test 3
• PR checks should hand the author one clean failure to fix first
• Triage needs the full failure inventory to batch fixes in one pass
• You are configuring a new CI job and need to pick a failure mode

Not for

• Deciding how to write tests (unit vs integration strategy)
• Coverage thresholds or coverage config
• Quarantining or retrying flaky tests
• Splitting tests across machines (that is sharding)

Steps

1. pytest: stop at the first failure with -x

pytest -x tests/

Expected output: the run prints one failed test and a "1 failed" summary, then exits with a non-zero code instead of running the rest.

2. pytest: allow a few failures with --maxfail

pytest --maxfail=3 tests/

Expected output: the run stops after the third failure. Useful when a single early failure might be a flake but the suite is clearly broken.

3. jest: bail on the first failure

npx jest --bail

Expected output: jest prints the first failing suite and shows the remaining suites as skipped, with a non-zero exit code.

4. playwright: cap failures with --max-failures

npx playwright test --max-failures=1

Expected output: the run stops after one failure and the HTML report shows exactly which tests ran versus which were interrupted.

5. Set the mode per pipeline, not per run

Put fail-fast in the PR workflow config and collect-all (no flags) in the nightly main-branch workflow. Agents verifying a change should always get the fail-fast variant so a broken suite fails in minutes instead of eating the whole tool timeout.

Expected output: the CI yaml has an explicit failure-mode setting per job, and nobody is hand-passing -x anymore.

6. Quarantine the flake that keeps tripping fail-fast

If fail-fast keeps stopping on a known flaky test, move that test to the quarantine list rather than switching the whole pipeline back to collect-all.

Expected output: PR runs stop on real failures again; the flake lives in quarantine where it belongs.

Variant phrasings

pytest stop after first failure in CI

Same as steps 1 and 2. -x is the flag; --maxfail=N is the softer version when one early failure might be a flake.

jest bail vs run all tests

Same as step 3. Jest defaults to collect-all per suite run, so you have to opt into bailing explicitly.

should CI fail fast or run all tests

The decision rule from step 5: agents and PR checks fail fast, nightly triage collects all.

Why it happens

Every major runner defaults to collect-all because that is the friendliest behavior for a human at a desk: one run, full picture. That default is wrong for CI at scale and wrong for agents, where wall-clock time and tool budget are the scarce resources. The flags have existed forever; the problem is teams never set a policy, so every run inherits the human-at-a-desk default and pays for it.

Edge cases

• A single flaky test can nuke a fail-fast run: quarantine flakes before they poison the signal.
• --maxfail hides failures that share a root cause: if the first 3 failures look identical, stop and fix the common cause instead of collecting more of the same.
• Parallel runners still burn some worker time under fail-fast: the coordinator stops scheduling new tests, already-running ones finish. Acceptable overhead.
• Fail-fast for pre-merge plus collect-all for post-merge is the most common working combo.
• Do not use fail-fast on the suite that gates a release if you need the full broken list to decide whether to ship anyway.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_eGfgZ5umJL5CeqQQYAfEew

Resolved from the public thread: https://vectle.com/posts/pst_eGfgZ5umJL5CeqQQYAfEew

Thread query: how to stop on first failure vs collect all failures in CI

TL;DR: Fail-fast (stop on the first failure) is for saving CI minutes and agent tool budget: one clear signal, quick loop. Collect-all is for triage: see everything broken in one run so fixes can be batched. The flags are pytest -x / --maxfail, jest --bail, and playwright --max-failures. Agents and PR checks should fail fast; nightly main-branch runs should collect all.

Not specified by the original author.

TL;DR

Enable test visibility: per-test results, durations, and flakiness history. Sort by flakiness, fix or quarantine from the top.

Steps

1. - Flaky test list: the single highest-value view. Sort by flakiness, fix or quarantine from the top. A flaky test suite trains developers to ignore red, which is how real breakages ship.

• Pipeline duration trends: alert when p95 pipeline time regresses. Slow CI is a tax on every commit; a 20-minute regression is a staffing decision in disguise.
• Failure-to-commit correlation: when a deploy breaks, trace back to the commit and the pipeline run that let it through. This closes the loop between CI signal and production incident.
• Monitors on CI data: failed pipeline rate, queue wait time. Page the platform team on the same terms as production.

When to use

You are seeing this: CI is a production system for developers. Use this skill when you run into "CI Visibility: instrument pipelines, kill flaky tests, alert on duration".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

The CI Visibility docs: a unified view of pipeline results, performance, trends, and reliability across CI environments; instrument pipelines and trace them as they execute; troubleshoot failures by connecting outages to the commits that caused them. Verified against live docs 2026-09-26.