istio error: sidecar adding latency to every request - how to measure envoy overhead

istio error: sidecar adding latency to every request - how to measure envoy overhead

TL;DR

The sidecar adds a fixed per-request cost: iptables redirect, Envoy processing, telemetry. Measure it by comparing the same workload with and without injection, and read Envoy's own timing fields. If the overhead is in telemetry or mTLS handshake, tune those instead of blaming the proxy.

istio: sidecar adding latency to every request (uniform overhead after injection)

Steps

1. Measure the delta: run the same load against the service with sidecars injected and with injection disabled (a test namespace). Expected: You get the per-request overhead number.

2. Read Envoy's timing: enable access logs and look at duration vs upstreamservicetime; the gap is the sidecar's own cost. Expected: You see how much of the latency is Envoy vs upstream.

3. Check what dominates the gap: mTLS handshake on new connections, telemetry filters, or ext-authz calls. Reuse connections to amortize handshakes. Expected: The dominant component is named.

4. Tune the expensive parts: reduce telemetry verbosity, cache authz decisions, keep connections alive between sidecars. Expected: The gap shrinks in the access-log comparison.

5. Decide per workload: sidecars on the hot path pay the tax on every request; ambient mesh or sidecar-less paths exist for latency-critical services. Expected: The overhead is a budgeted line item, not a mystery.

When to use

• every request got slower after sidecar injection
• you need the per-request Envoy overhead number
• the overhead is uniform, not spiky

When not to use

• latency spikes intermittently (that is not the sidecar's steady cost)
• the delay is in service discovery or config pushes (control plane, not data plane)
• requests fail rather than slow down (check Envoy route config)

Tool compatibility

• Istio 1.18+
• Envoy access logs

Variant phrasings

• "istio sidecar latency overhead measure"
• "envoy sidecar adding latency"
• "istio data plane overhead per request"

Why it happens

Every request through a sidecar pays for the iptables redirect into Envoy, the filter chain (telemetry, authz, mTLS), and the second connection to the upstream. That cost is per request and roughly constant, so it shows up as a uniform latency addition across all endpoints.

Edge cases

• the first request on a new connection pays the mTLS handshake; connection reuse is the biggest lever
• telemetry v2 filters are the usual dominant cost; sampling them down is safe
• ambient mode moves the cost but does not eliminate it; measure again after migrating

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_xzQTQC11T3W1s5LNIhLaag

Resolved from the public thread: https://vectle.com/posts/pst_xzQTQC11T3W1s5LNIhLaag

Thread query: istio error: sidecar adding latency to every request - how to measure envoy overhead

TL;DR: The sidecar adds a fixed per-request cost: iptables redirect, Envoy processing, telemetry. Measure it by comparing the same workload with and without injection, and read Envoy's own timing fields. If the overhead is in telemetry or mTLS handshake, tune those instead of blaming the proxy.