agent profiled the endpoint in isolation but the latency only appears when 4 other services saturate the same node

Latency only appears when other services saturate the same node

TL;DR: Profile the node, not just the endpoint. When four services share a box, CPU steal, memory pressure, and disk I/O from neighbors show up as your service's latency. Correlate the endpoint's p99 with node-level saturation metrics, then isolate with resource limits or move the noisy neighbor.

agent profiled the endpoint in isolation but the latency only appears when 4 other services saturate the same node

Steps

1. Confirm the correlation: overlay the endpoint's p99 latency with node CPU, memory pressure, and disk I/O over the incident window (from your APM plus node exporter, CloudWatch, or equivalent).

Expected: p99 spikes line up with node saturation (CPU above 85%, iowait high, or memory near limit) rather than with your service's own traffic.

1. Identify the noisy neighbor: rank processes or containers on the node by CPU and I/O during the spike:

   # on the node during (or from a captured snapshot of) the incident:
   ps -eo pid,comm,%cpu,%mem --sort=-%cpu | head -15

Expected: a sibling service (batch job, indexer, backup agent) tops the list while your service sits far below.

1. Check for the classic shared-resource signatures: CPU steal on virtualized hosts, OOM-killer-adjacent memory pressure, and iowait from a neighbor's bulk writes. Container CPU throttling shows as container_cpu_cfs_throttled_seconds_total climbing in your metrics.

Expected: at least one saturation signal coincides with the latency spike.

1. Reproduce honestly: run your endpoint benchmark while simultaneously loading the node the way the neighbors do (a CPU burner plus a disk writer at the observed levels), not on an idle box.

Expected: the isolated-fast endpoint degrades to near-production latency - you now have a reproduction.

1. Fix it with isolation, in order of cheapness: set CPU/memory requests and limits so neighbors cannot starve your service; pin the batch workloads to specific cores or nodes; move the heaviest neighbor to its own node or schedule it outside peak hours.

Expected: p99 drops and decouples from the neighbor's schedule.

1. Add node-saturation metrics to the agent's profiling checklist so isolated-endpoint benchmarks always ship with a "shared node state" section.

Expected: future profiles note contention instead of declaring the endpoint clean.

Use this when

• An endpoint is fast in isolation but slow in production on shared infrastructure.
• Latency spikes correlate with other services' activity, not your own traffic.
• You run multiple services per node, VM, or Kubernetes worker.
• You need to prove contention before asking for isolation or new capacity.

Not for this skill when

• The service has dedicated nodes and still slows down - look at its own code, queries, or downstream dependencies.
• Latency is constant rather than spiky - steady slowness is a code or query problem, not contention.
• You cannot get node-level metrics - without them this stays a hypothesis; fix observability first.

Variant phrasings

• "service slow only when other services busy same node"
• "noisy neighbor latency kubernetes"
• "endpoint fast alone slow in production shared host"
• "how to prove cpu contention from sibling service"

Why it happens

Profilers attribute time to code, but wall-clock latency includes time the process spends waiting: descheduled while a neighbor burns CPU, stalled on disk behind a neighbor's writes, or throttled by the container runtime. An isolated benchmark removes all of that and measures a fantasy. The agent did the profiling right and the scoping wrong - it profiled the service instead of the system the service runs in.

Edge cases

• CPU limits in Kubernetes can throttle your own service before the node saturates - check throttling metrics before blaming neighbors.
• Memory pressure causes page-cache eviction that looks like slow disk; the fix may be more RAM, not less neighbor.
• Network contention on shared hosts shows up as latency with clean CPU - check retransmits and interface saturation too.
• Moving the neighbor helps until traffic grows into the new headroom; set alerts on node saturation so the next collision gets caught early.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_CBNU3tXTw3B8kuHUwhMwyw