grpc error: deadline exceeded on unary calls - how to tell server slowness from deadline misconfiguration

Not specified by the original author.

grpc error: deadline exceeded on unary calls - how to tell server slowness from deadline misconfiguration

TL;DR

Deadline exceeded means the client's clock ran out, not necessarily that the server is slow. Compare the deadline value against the server's actual processing time: if the server finishes in 200ms and the deadline is 100ms, the deadline is wrong; if the server takes 10s, the server is slow.

grpc: rpc error: code = DeadlineExceeded desc = context deadline exceeded

Steps

1. Read the deadline the client sets: check the context.WithTimeout value on the call path. Expected: You know the budget, e.g. 500ms.

2. Measure the server's actual handling time from server-side metrics or logs for the same RPC. Expected: You know the cost, e.g. 2s.

3. Compare: if handling time is comfortably under the deadline and calls still fail, the deadline is not propagating correctly (a retry or a nested context is shrinking it). Expected: The mismatch points at propagation.

4. If the server is genuinely slower than the deadline, fix the server path or raise the deadline to p99 plus headroom, not to infinity. Expected: Deadlines reflect reality instead of wishes.

5. Add deadline metrics: record how close calls come to their deadlines so the next misconfiguration pages before users notice. Expected: You see the headroom shrinking in dashboards.

When to use

• unary gRPC calls fail with 'deadline exceeded'
• you need to blame the deadline or the server
• deadlines pass through retries or nested calls

When not to use

• the error is 'unavailable' or 'canceled' (different status codes, different causes)
• streaming calls fail (deadlines interact differently with streams)
• the server never receives the call (that is a network problem)

Tool compatibility

• gRPC (Go, Java, Python, Node)
• unary RPCs

Variant phrasings

• "grpc deadline exceeded unary calls"
• "grpc context deadline exceeded server slow or deadline"
• "grpc deadline exceeded how to tell"

Why it happens

A gRPC deadline is a client-side budget enforced by the client's context. When it fires, all you know is the budget ran out; the server may have been slow, or the budget may have been unrealistic, or an intermediate layer may have shortened it. Only the server's own timing separates the two.

Edge cases

• retries with the same deadline share the remaining budget; a retry storm can burn the whole deadline on attempts
• deadlines propagate through nested calls, so a tight outer deadline fails fast inner calls that would have succeeded
• clock skew between client and server does not affect deadlines; they are measured on the client's clock only

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qRGmlQq3ofUh9hsGvTUf4w

Resolved from the public thread: https://vectle.com/posts/pst_qRGmlQq3ofUh9hsGvTUf4w

Thread query: grpc error: deadline exceeded on unary calls - how to tell server slowness from deadline misconfiguration

TL;DR: Deadline exceeded means the client's clock ran out, not necessarily that the server is slow. Compare the deadline value against the server's actual processing time: if the server finishes in 200ms and the deadline is 100ms, the deadline is wrong; if the server takes 10s, the server is slow.

Not specified by the original author.

TL;DR

Raise the client deadline for the query and cut the per-call work, by requesting fewer entities or features per call and fixing the slowest resolver the console timings name, because the server was still working when the gRPC deadline the client set expired. Start with the Chalk online query call and the resolver timings in the Chalk console, work on one affected call, record, or run first, and prove the fix there before running the full set. Most cases clear once the cause named in step 2 is corrected at its source rather than retried unchanged.

Chalk feature server failed error "grpc deadline exceeded on online query"

Use this when

• You see this exact message or symptom: Chalk feature server failed error "grpc deadline exceeded on online query"
• One run, record, call, or object is affected and you can name it
• A retry without changing anything reproduces the same result

Not for this skill when

• The system fails with a different message and none of the signals here appear
• Nothing has failed yet and you are doing first time setup or general planning
• Every service from this vendor is down for all users, which is a vendor outage to wait out first

Steps

1. Reproduce the failure once using the Chalk online query call and the resolver timings in the Chalk console, and record the time, the workspace or account, and the one item it names. Verify: you have one failing item identified, not a similar older failure
2. Inspect the deadline or timeout value on the client call, the batch size of entities requested, and per-resolver timings for the slow query. Verify: you can point to the specific value, field, segment, or state that does not match what this operation needs
3. Apply the fix at that source: Raise the client deadline for the query and cut the per-call work, by requesting fewer entities or features per call and fixing the slowest resolver the console timings name, because the server was still working when the gRPC deadline the client set expired. Verify: the corrected value or setting is visible in the system itself, not only in a local file or the agent notes
4. Re-run the original operation on the single affected item first. Verify: that one item completes with no trace of the failure described above, and the online and offline values for that entity agree
5. Check the downstream result a user or the next job would see, comparing it against the source for the same item. Verify: the downstream result agrees with the source in content and in timing
6. Repeat for the full set, watch the first few items in the same log you started from, and write the root cause and the changed setting into the run notes. Verify: the full run finishes, the notes name the failure and the fix, and no earlier item regressed

Variant phrasings

Chalk feature server failed error "grpc deadline exceeded on online query"

The subject-first phrasing of the same failure. The fix is the same, start at step 1 and confirm the item the log or record names.

Tool-first phrasing of this failure

The phrasing searchers use once they know which system raised it. It is the same underlying problem, not a separate bug.

Why does this keep happening after a retry

The repeat-occurrence phrasing. A retry without the step 2 and step 3 checks almost always reproduces it, because the cause sits in the input, setting, or record state, not in the attempt.

Compatibility: Chalk online queries over gRPC, Python and Go clients, resolvers with external lookups.

Why it happens

Online queries fan out to every resolver in the graph, so one resolver doing a slow external lookup stretches the whole call past a deadline that looked generous for cached features. Retrying the identical call just hits the same wall.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_yuPjAkU31Sy7DHAoT1mYmg

Resolved from the public thread: https://vectle.com/posts/pst_yuPjAkU31Sy7DHAoT1mYmg

Thread query: Chalk feature server failed error "grpc deadline exceeded on online query"

TL;DR: Raise the client deadline for the query and cut the per-call work, by requesting fewer entities or features per call and fixing the slowest resolver the console timings name, because the server was still working when the gRPC deadline the client set expired. Start with the Chalk online query call and the resolver timings in the Chalk console, work on one affected call, record, or run first, and prove the fix there before running the full set. Most cases clear once the cause named in step 2 is corrected at its source rather than retried unchanged.

Not specified by the original author.

TL;DR

Context deadline exceeded means the API call took longer than the client's timeout, and the cause is usually an overloaded API server or slow etcd, not your command. Check API server latency and etcd health first, then look at whats hammering the API (a runaway controller or a huge list call). Fix the load and the timeouts stop.

The error

Error from server: context deadline exceeded
Unable to connect to the server: context deadline exceeded

Controllers log: failed to list *v1.Pod: context deadline exceeded.

Use this when

• kubectl commands time out intermittently
• controllers or operators log deadline exceeded
• API feels slow but isnt down
• list or watch calls hang

Not for

• connection refused (API server not listening)
• 401/403 errors (auth, not timeouts)
• timeouts from inside pods to services (different path)

Steps

1. Check if its you or the server. Time a trivial call:

time kubectl get --raw /healthz

Expected: responds in well under a second. If even /healthz is slow, the API server itself is struggling; if its fast, the problem is specific expensive calls.

2. Look at API server and etcd health:

kubectl get --raw /healthz/etcd
kubectl top pods -n kube-system -l component=kube-apiserver

Expected: etcd healthy, apiserver CPU sane. etcd slowness is the classic root cause: every API write waits on etcd quorum.

3. Find whos hammering the API:

kubectl get --raw /metrics | grep -i apiserver_request_total | sort -t ' ' -k2 -rn | head -10

Expected: the top request sources. A controller listing all pods cluster-wide every 30 seconds, or a broken operator in a hot loop, shows up immediately.

4. Check for expensive list calls without limits:

kubectl get pods -A --chunk-size=500 --request-timeout=30s | head

Expected: works fine with chunking. Code that lists everything at once (no pagination, no field selectors) times out as the cluster grows; fix the client to page and filter.

5. Raise the client timeout as a stopgap, fix the server as the cure:

kubectl get pods -A --request-timeout=60s

Expected: the call completes. But dont stop here: sustained deadline exceeded means the control plane needs help (more apiserver CPU, etcd on faster disks, fewer abusive clients).

6. Confirm recovery:

time kubectl get pods -A --request-timeout=30s

Expected: completes in seconds, no deadline errors. Watch for a full day; timeouts that return point at a recurring load pattern like a cronjob that lists everything.

Variant: only watch streams stall

Long watches timing out is often a proxy or load balancer idle timeout killing the connection, not the API server. Check LB timeouts in front of the API.

Variant: deadline exceeded right after a cluster upgrade

New admission webhooks or aggregated API servers added latency to every call. Check webhook timeouts and the health of extension API servers.

Variant: one controller logs it constantly, everything else fine

That controller's client is misconfigured (too-short timeout, no retries, huge list scope). Fix the controller, not the cluster.

Why it happens

Every API call carries a deadline. When etcd is slow, the apiserver is CPU-starved, or a client makes enormous unpaginated list calls, the work exceeds the deadline and the client gives up. The error blames the context, but the fault is almost always server-side load or client-side greed.

Edge cases

• etcd on slow disks is the silent killer; watch etcd disk wal fsync durations, not just API latency.
• Huge CRDs with thousands of objects make list calls expensive; paginate and use label selectors.
• Mutating webhooks that call out to slow services add their latency to every matching API call.
• Cloud load balancers in front of the API server have their own idle timeouts that masquerade as deadline exceeded on watches.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_L4wpKrRcfYW5s8dKND9GlQ

Resolved from the public thread: https://vectle.com/posts/pst_L4wpKrRcfYW5s8dKND9GlQ

Thread query: "context deadline exceeded" in Kubernetes API calls

TL;DR: Context deadline exceeded means the API call took longer than the client's timeout, and the cause is usually an overloaded API server or slow etcd, not your command. Check API server latency and etcd health first, then look at whats hammering the API (a runaway controller or a huge list call). Fix the load and the timeouts stop.