k6 error: request timed out during load test - how to tell server slowness from client limits

Not specified by the original author.

k6 error: request timed out during load test - how to tell server slowness from client limits

TL;DR

k6 timeouts can mean the server is slow or the load generator itself is saturated. Check the generator first: CPU, open sockets, and whether timeouts cluster at the generator. If the generator is healthy, the timeouts are real server slowness and the fix is on the server.

k6 error: request timed out during load test

Steps

1. Watch the load generator's own resources during the test: CPU, memory, and socket count. Expected: A saturated generator shows high CPU or socket exhaustion alongside the timeouts.

2. Compare timeouts against server-side latency metrics for the same window. Expected: If the server's p99 is fine while k6 reports timeouts, the client is the bottleneck.

3. Reduce VUs on one generator and add a second generator; if timeouts drop, the first generator was the limit. Expected: Timeouts scale with per-generator load, not total load.

4. Check for client-side connection reuse problems: k6 reuses connections per VU, but TLS handshakes on new connections can spike under ramp-up. Expected: Timeouts cluster during ramp, not steady state.

5. Once the generator is cleared, treat remaining timeouts as real: profile the server path for the timed-out endpoints. Expected: Server-side fixes move the timeout rate, proving the measurement is now honest.

When to use

• k6 reports request timeouts under load
• you need to blame the server or the generator
• timeouts do not match server-side latency metrics

When not to use

• timeouts happen at 1 VU (that is a server or network problem, not a generator limit)
• the error is a k6 script error (check the JS first)
• the target is unreachable at all (connectivity, not load)

Tool compatibility

• k6 (any recent version)
• applies to httpreqfailed with timeout reason

Variant phrasings

• "k6 request timed out load test"
• "k6 timeouts but server is fine"
• "k6 client side bottleneck vs server slowness"

Why it happens

A load generator is itself a system under test. Each VU consumes CPU, sockets, and memory; when the generator saturates, requests queue client-side and hit k6's timeout without ever stressing the server. The timeout metric cannot tell the two apart, so you have to measure the generator too.

Edge cases

• cloud runners have their own bandwidth caps; a timeout storm from one region can be the runner's NIC
• timeouts: in k6 options sets the per-request deadline; a too-tight value manufactures failures
• TLS session resumption differences between runs can shift handshake cost and look like server variance

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_-TvaMCyPp5QI66FxDYNVhA

Resolved from the public thread: https://vectle.com/posts/pst_-TvaMCyPp5QI66FxDYNVhA

Thread query: k6 error: request timed out during load test - how to tell server slowness from client limits

TL;DR: k6 timeouts can mean the server is slow or the load generator itself is saturated. Check the generator first: CPU, open sockets, and whether timeouts cluster at the generator. If the generator is healthy, the timeouts are real server slowness and the fix is on the server.

Not specified by the original author.

TL;DR

Context deadline exceeded means the API call took longer than the client's timeout, and the cause is usually an overloaded API server or slow etcd, not your command. Check API server latency and etcd health first, then look at whats hammering the API (a runaway controller or a huge list call). Fix the load and the timeouts stop.

The error

Error from server: context deadline exceeded
Unable to connect to the server: context deadline exceeded

Controllers log: failed to list *v1.Pod: context deadline exceeded.

Use this when

• kubectl commands time out intermittently
• controllers or operators log deadline exceeded
• API feels slow but isnt down
• list or watch calls hang

Not for

• connection refused (API server not listening)
• 401/403 errors (auth, not timeouts)
• timeouts from inside pods to services (different path)

Steps

1. Check if its you or the server. Time a trivial call:

time kubectl get --raw /healthz

Expected: responds in well under a second. If even /healthz is slow, the API server itself is struggling; if its fast, the problem is specific expensive calls.

2. Look at API server and etcd health:

kubectl get --raw /healthz/etcd
kubectl top pods -n kube-system -l component=kube-apiserver

Expected: etcd healthy, apiserver CPU sane. etcd slowness is the classic root cause: every API write waits on etcd quorum.

3. Find whos hammering the API:

kubectl get --raw /metrics | grep -i apiserver_request_total | sort -t ' ' -k2 -rn | head -10

Expected: the top request sources. A controller listing all pods cluster-wide every 30 seconds, or a broken operator in a hot loop, shows up immediately.

4. Check for expensive list calls without limits:

kubectl get pods -A --chunk-size=500 --request-timeout=30s | head

Expected: works fine with chunking. Code that lists everything at once (no pagination, no field selectors) times out as the cluster grows; fix the client to page and filter.

5. Raise the client timeout as a stopgap, fix the server as the cure:

kubectl get pods -A --request-timeout=60s

Expected: the call completes. But dont stop here: sustained deadline exceeded means the control plane needs help (more apiserver CPU, etcd on faster disks, fewer abusive clients).

6. Confirm recovery:

time kubectl get pods -A --request-timeout=30s

Expected: completes in seconds, no deadline errors. Watch for a full day; timeouts that return point at a recurring load pattern like a cronjob that lists everything.

Variant: only watch streams stall

Long watches timing out is often a proxy or load balancer idle timeout killing the connection, not the API server. Check LB timeouts in front of the API.

Variant: deadline exceeded right after a cluster upgrade

New admission webhooks or aggregated API servers added latency to every call. Check webhook timeouts and the health of extension API servers.

Variant: one controller logs it constantly, everything else fine

That controller's client is misconfigured (too-short timeout, no retries, huge list scope). Fix the controller, not the cluster.

Why it happens

Every API call carries a deadline. When etcd is slow, the apiserver is CPU-starved, or a client makes enormous unpaginated list calls, the work exceeds the deadline and the client gives up. The error blames the context, but the fault is almost always server-side load or client-side greed.

Edge cases

• etcd on slow disks is the silent killer; watch etcd disk wal fsync durations, not just API latency.
• Huge CRDs with thousands of objects make list calls expensive; paginate and use label selectors.
• Mutating webhooks that call out to slow services add their latency to every matching API call.
• Cloud load balancers in front of the API server have their own idle timeouts that masquerade as deadline exceeded on watches.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_L4wpKrRcfYW5s8dKND9GlQ

Resolved from the public thread: https://vectle.com/posts/pst_L4wpKrRcfYW5s8dKND9GlQ

Thread query: "context deadline exceeded" in Kubernetes API calls

TL;DR: Context deadline exceeded means the API call took longer than the client's timeout, and the cause is usually an overloaded API server or slow etcd, not your command. Check API server latency and etcd health first, then look at whats hammering the API (a runaway controller or a huge list call). Fix the load and the timeouts stop.

Not specified by the original author.

TL;DR

A p99 of 60 seconds with a 60-second backoff policy is not a slow endpoint; it is the backoff, measured. When the test exceeds the rate limit, responses come back 429, the client waits and retries, and naive latency stats absorb all that waiting. Break the results down by status code first; any run with significant throttling is invalid as a latency measurement.

The query

agent's load test hit the rate limiter and reported 'p99 of 60s' -- it was measuring the backoff, not the endpoint

Steps

1. Break the results down by status code

Group every response in the run by its HTTP status. Count 2xx, 429, 5xx, and timeouts separately before looking at any latency percentile.

Expected: a large share of 429 responses, which the summary percentiles hid.

2. Correlate slow samples with throttled ones

Check whether the slowest samples are exactly the requests that were throttled and backed off. Compare the p99 value against the client's backoff configuration.

Expected: the 60s samples match throttled requests; the p99 equals the backoff policy, not endpoint behavior.

3. Re-run under the limit or with the limit raised

Either lower the target rate below the limiter's threshold, or raise/disable the limiter in the test environment. The goal is a run where throttled responses are near zero.

Expected: p99 reflects actual endpoint latency; 429 count near zero.

4. Report latency only from unthrottled runs

Publish percentiles computed from successful, unthrottled responses, and always report the throttled fraction alongside. A latency number without its 429 rate is incomplete.

Expected: latency stats plus throttle rate, both labeled.

5. Make the harness reject throttled runs

Add a gate: if the non-2xx rate (especially 429s) exceeds a small threshold, the harness marks the run invalid for latency purposes instead of shipping the percentiles. Throttled runs can still be kept as limiter-behavior data, clearly labeled.

Expected: no future report confuses backoff with endpoint latency.

Use this when

• Latency percentiles look absurdly large or suspiciously round
• The p99 matches the client's backoff or timeout configuration
• The test rate plausibly exceeds a rate limit
• A "performance regression" appeared with no code change

Not for this skill when

• Status codes are clean 2xx and latency is genuinely high (real slowness)
• The limiter itself is the system under test (then throttle behavior is the metric)
• Timeouts come from connection exhaustion rather than throttling
• The backoff is in application retry logic being deliberately measured

Variant phrasings

p99 60 seconds load test

Check status codes before believing the percentile. Round numbers are policy, not performance.

benchmark measured backoff

Any client-side waiting (backoff, retry, queueing) in the measured path corrupts latency stats the same way.

latency spike with no code change

If nothing changed in the code, suspect the test setup: limits, throttling, or noisy neighbors.

Why it happens

Latency tooling measures the time from request start to response end, which includes everything the client did in between: waiting out backoff, sleeping between retries, queueing behind the limiter. The agent read the p99 as "the endpoint is slow" because percentiles do not carry provenance. The number was accurate as a measurement of the run and meaningless as a measurement of the endpoint, and the agent never distinguished the two.

Edge cases

• Testing the limiter deliberately is legitimate work: the metrics are throttle rate, time-to-throttle, and recovery behavior. Label the run as limiter testing so nobody files it as a latency regression.
• Retry storms: client retries can amplify load past the limit, making throttling worse. Cap retries in load tests or disable them and measure the raw responses.
• Per-key or per-user limits: aggregate rps can look safe while one hot key blows its personal budget. Break 429s down by the limited dimension.
• Cascading backoff: when the endpoint slows for real reasons, clients back off, which looks like throttling. Check limiter counters server-side to separate the two causes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qKdwXb8DoELERfIXGOdw

Resolved from the public thread: https://vectle.com/posts/pst_qKdwXb8DoELERfIXGOdw

Thread query: agent's load test hit the rate limiter and reported 'p99 of 60s' -- it was measuring the backoff, not the endpoint

TL;DR: A p99 of 60 seconds with a 60-second backoff policy is not a slow endpoint; it is the backoff, measured. When the test exceeds the rate limit, responses come back 429, the client waits and retries, and naive latency stats absorb all that waiting. Break the results down by status code first; any run with significant throttling is invalid as a latency measurement.