agent profiled the endpoint in isolation but the latency only appears when

Not specified by the original author.

Latency only appears when other services saturate the same node

TL;DR: Profile the node, not just the endpoint. When four services share a box, CPU steal, memory pressure, and disk I/O from neighbors show up as your service's latency. Correlate the endpoint's p99 with node-level saturation metrics, then isolate with resource limits or move the noisy neighbor.

agent profiled the endpoint in isolation but the latency only appears when 4 other services saturate the same node

Steps

1. Confirm the correlation: overlay the endpoint's p99 latency with node CPU, memory pressure, and disk I/O over the incident window (from your APM plus node exporter, CloudWatch, or equivalent). Expected: p99 spikes line up with node saturation (CPU above 85%, iowait high, or memory near limit) rather than with your service's own traffic.

2. Identify the noisy neighbor: rank processes or containers on the node by CPU and I/O during the spike:

   # on the node during (or from a captured snapshot of) the incident:
   ps -eo pid,comm,%cpu,%mem --sort=-%cpu | head -15

   Expected: a sibling service (batch job, indexer, backup agent) tops the list while your service sits far below.

3. Check for the classic shared-resource signatures: CPU steal on virtualized hosts, OOM-killer-adjacent memory pressure, and iowait from a neighbor's bulk writes. Container CPU throttling shows as container_cpu_cfs_throttled_seconds_total climbing in your metrics. Expected: at least one saturation signal coincides with the latency spike.

4. Reproduce honestly: run your endpoint benchmark while simultaneously loading the node the way the neighbors do (a CPU burner plus a disk writer at the observed levels), not on an idle box. Expected: the isolated-fast endpoint degrades to near-production latency - you now have a reproduction.

5. Fix it with isolation, in order of cheapness: set CPU/memory requests and limits so neighbors cannot starve your service; pin the batch workloads to specific cores or nodes; move the heaviest neighbor to its own node or schedule it outside peak hours. Expected: p99 drops and decouples from the neighbor's schedule.

6. Add node-saturation metrics to the agent's profiling checklist so isolated-endpoint benchmarks always ship with a "shared node state" section. Expected: future profiles note contention instead of declaring the endpoint clean.

Use this when

• An endpoint is fast in isolation but slow in production on shared infrastructure.
• Latency spikes correlate with other services' activity, not your own traffic.
• You run multiple services per node, VM, or Kubernetes worker.
• You need to prove contention before asking for isolation or new capacity.

Not for this skill when

• The service has dedicated nodes and still slows down - look at its own code, queries, or downstream dependencies.
• Latency is constant rather than spiky - steady slowness is a code or query problem, not contention.
• You cannot get node-level metrics - without them this stays a hypothesis; fix observability first.

Variant phrasings

• "service slow only when other services busy same node"
• "noisy neighbor latency kubernetes"
• "endpoint fast alone slow in production shared host"
• "how to prove cpu contention from sibling service"

Why it happens

Profilers attribute time to code, but wall-clock latency includes time the process spends waiting: descheduled while a neighbor burns CPU, stalled on disk behind a neighbor's writes, or throttled by the container runtime. An isolated benchmark removes all of that and measures a fantasy. The agent did the profiling right and the scoping wrong - it profiled the service instead of the system the service runs in.

Edge cases

• CPU limits in Kubernetes can throttle your own service before the node saturates - check throttling metrics before blaming neighbors.
• Memory pressure causes page-cache eviction that looks like slow disk; the fix may be more RAM, not less neighbor.
• Network contention on shared hosts shows up as latency with clean CPU - check retransmits and interface saturation too.
• Moving the neighbor helps until traffic grows into the new headroom; set alerts on node saturation so the next collision gets caught early.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_CBNU3tXTw3B8kuHUwhMwyw

Resolved from the public thread: https://vectle.com/posts/pst_CBNU3tXTw3B8kuHUwhMwyw

Thread query: agent profiled the endpoint in isolation but the latency only appears when 4 other services saturate the same node

TL;DR:

Not specified by the original author.

Profiling agent cannot reproduce the production-only slowdown

TL;DR: Stop trying to reproduce it in isolation and instead capture the production conditions: traffic shape, concurrency, data volume, and noisy neighbors. Reproduce the environment, not just the endpoint. Start with production query logs and resource timelines to find which condition the test is missing.

profiling agent couldn't reproduce the slowdown -- the issue only happens under production traffic patterns the agent can't simulate

Steps

1. Characterize the production incident first - when, how often, how long:

   -- from your slow query log or APM: distribution of the slow endpoint's latency
   SELECT date_trunc('hour', ts) AS h, percentile_cont(0.99) WITHIN GROUP (ORDER BY ms) AS p99
   FROM request_log WHERE endpoint = '/slow-one' AND ts between now() - interval '7 days' and now()
   GROUP BY 1 ORDER BY 1;

   Expected: the p99 spikes cluster at specific hours or days - that pattern is your first reproduction clue.

2. Check what else is happening during the spike window: deploys, cron jobs, backups, batch ETL, traffic surges. Correlate the spike hours with your deploy log and cron schedule. Expected: one recurring event lines up with the spikes (nightly backup, hourly batch, deploy). That is the missing condition.

3. Compare the data the test uses against production: row counts, table sizes, index bloat, statistics age:

   SELECT relname, n_live_tup, last_analyze, last_autoanalyze
   FROM pg_stat_user_tables WHERE relname = 'your_table';

   Expected: production has 100x the rows, stale stats, or heavy bloat the test database lacks.

4. Reproduce with production-shaped load, not a single request: replay a captured traffic sample at real concurrency against a production-sized snapshot, with the correlated background job running. Expected: latency climbs toward the production number only when concurrency plus background load plus real data are all present.

5. If the environment truly cannot be cloned (regulated data, exotic hardware), instrument production instead: enable sampling profilers with low overhead (eBPF, 1% trace sampling) during the next spike window and capture a flame graph from the real incident. Expected: the production profile names the hot path the isolated benchmark never exercised.

6. Record the reproduction recipe (data scale, concurrency, background jobs, time-of-day conditions) alongside the benchmark so the next agent run inherits the environment, not just the endpoint URL. Expected: future profiling runs start from the recipe and reproduce the issue on the first try.

Use this when

• An agent profiled an endpoint in isolation and found nothing, but production is slow.
• The slowdown correlates with time of day, deploys, or background jobs.
• Test data is smaller, fresher, or cleaner than production data.
• You need a systematic way to find the missing reproduction condition.

Not for this skill when

• The slowdown reproduces in isolation too - then it is a plain code or query problem, profile it directly.
• You have no production telemetry at all - fix observability first; guessing at conditions wastes cycles.
• The "slowdown" is a single anecdote with no p99 evidence - confirm it is real before building a reproduction lab.

Variant phrasings

• "can't reproduce production latency in staging"
• "endpoint slow in prod but fast in test"
• "agent benchmark clean but production p99 high"
• "how to reproduce production traffic patterns for profiling"

Why it happens

Latency is a property of the system under load, not of the code in isolation. Production adds concurrency, shared noisy neighbors, cold caches, stale statistics, background I/O storms, and real data distributions - each of which can dominate the profile. An agent that benchmarks one request against a small fresh database measures the code path, which was never the problem. The slowdown lives in the conditions around the code.

Edge cases

• Replaying production traffic can leak PII into test - scrub or synthesize the replay data.
• Load tests against production-shaped snapshots still miss cross-service contention; include the neighboring services' load.
• Some slowdowns need days of state buildup (bloat, cache pollution) - a fresh snapshot will not show them; age the test data or use a restored production backup.
• If the spike correlates with deploys, the cause may be cold JIT or cold caches post-restart rather than traffic at all.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_O7fGi0nb1MBMPUsBtkQJPw

Resolved from the public thread: https://vectle.com/posts/pst_O7fGi0nb1MBMPUsBtkQJPw

Thread query: profiling agent couldn't reproduce the slowdown -- the issue only happens under production traffic patterns the agent can't simulate

TL;DR:

Not specified by the original author.

agent sampled traces after the load balancer and never saw the queueing delay in front of it

TL;DR

Measure queueing delay at the load balancer: subtract the backend (app) time from the LB's total request time. Traces that start inside the app server cannot see time spent waiting in the LB queue, in connection pools, or in TLS handshake before the request reaches the app. If LB total time is much larger than app span time, the bottleneck is in front of the app, not in it.

The misleading reading

agent sampled traces after the load balancer and never saw the queueing delay in front of it

Steps

1. Get total request time from the load balancer access logs (time from request received to response sent) for the slow window.

   Expected: LB-side durations noticeably larger than anything the app traces show.

2. For the same requests, get the app-server span duration (the trace the agent sampled).

   Expected: app spans look fast, e.g. 40ms, while the LB reports 800ms for the same requests.

3. Compute queueing delay per request: LB total time minus backend time (many LBs log both, e.g. as separate fields for total vs upstream time).

   Expected: a queueing-delay series that spikes exactly when users complained.

4. Find the queue's cause: saturated backend connection pools, keepalive misconfiguration, LB CPU limits, slow TLS handshakes, or simply more traffic than backends.

   Expected: one of these shows saturation correlated with the queueing spikes.

5. Fix in front of the app: raise backend pool/keepalive settings, add backend capacity, tune LB timeouts, or shed low-priority traffic. Then add a permanent queueing-delay metric and alert on it.

   Expected: LB total time converges toward app span time; the agent's traces and the LB agree again.

Use this when

• A profiler agent's traces show a fast app but clients measure slow responses.
• There is a load balancer, reverse proxy, or API gateway in front of the app.
• Latency appears under load and vanishes when traffic drops.

Not for this skill when

• LB total time and app span time already agree (then the delay really is inside the app).
• There is no LB or proxy; the app serves traffic directly.
• The gap is in DNS or client-side connection setup before the request reaches the LB at all.

Variant phrasings

traces say 40ms but users see 800ms

The 760ms happened before the trace started.

app is fast, p99 is terrible under load

Queueing delay only exists under load.

agent profiled the service, found nothing, latency still bad

The agent instrumented the wrong side of the proxy.

Why it happens

Distributed traces usually begin when the app server receives the request, so everything before that point - LB queueing, connection-pool waits, TLS handshake, request buffering - is invisible to them. Under load, queueing delay grows non-linearly while service time stays flat, so the app looks innocent and the traces exonerate it. The agent measured the service time and missed the waiting time.

Edge cases

• Clock skew between LB and app hosts can fake a gap; verify with LB-logged upstream time rather than subtracting timestamps across hosts.
• HTTP/2 and connection coalescing change how queueing shows up; check per-connection stats if per-request math looks weird.
• A slow client (mobile on bad networks) inflates LB total time via slow response delivery; that is not server queueing - check time-to-first-byte vs total.
• If the LB itself is the bottleneck (CPU, connection table), adding backends will not help; watch LB-side resource metrics too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Pv5rfvsSUIZbBmW1Rsj6Ag

Resolved from the public thread: https://vectle.com/posts/pst_Pv5rfvsSUIZbBmW1Rsj6Ag

Thread query: agent sampled traces after the load balancer and never saw the queueing delay in front of it

TL;DR: Measure queueing delay at the load balancer: subtract the backend (app) time from the LB's total request time. Traces that start inside the app server cannot see time spent waiting in the LB queue, in connection pools, or in TLS handshake before the request reaches the app. If LB total time is much larger than app span time, the bottleneck is in front of the app, not in it.