agent recommended smaller instances - noisy-neighbor effect on shared tenancy tanked p99

TL;DR

The agent downsized to smaller shared-tenancy instances where noisy neighbors steal CPU unpredictably, and p99 latency collapsed even though average CPU looked fine. Before recommending smaller shared instances, check p99 (not average) latency headroom, and consider dedicated tenancy or burstable-with-baseline options for latency-sensitive workloads. Averages hide contention; p99 reveals it.

Error

agent recommended smaller instances  -  noisy-neighbor effect on shared tenancy tanked p99

Steps

1. Confirm the cause: p99 latency degraded after the downsize while averages stayed healthy. Expected: the p99 regression is visible.
2. Check whether the workload is latency-sensitive (user-facing, SLO-bound). Expected: the sensitivity verdict is explicit.
3. For sensitive workloads, revert to the larger size or move to dedicated tenancy. Expected: p99 recovers.
4. Add p99 latency as a constraint in the agent's rightsizing model, not just average CPU. Expected: future recommendations respect tail latency.
5. Re-run the rightsizing analysis with the p99 gate. Expected: only workloads with p99 headroom are downsized.

When to use

• Downsizing degrades p99 latency.
• Shared-tenancy contention is suspected.
• The workload has latency SLOs.

When not to use

• The workload is batch and latency-insensitive.
• Tenancy is already dedicated.
• p99 is healthy after the downsize.

Tool compatibility

AWS EC2 tenancy options, CloudWatch (current 2026 releases).

Variant phrasings

p99 tanked after rightsizing down

Noisy neighbors on smaller shared instances. Check p99 headroom.

smaller instances, worse tail latency

Size on p99, not on average CPU.

latency-sensitive workload on shared tenancy

Consider dedicated tenancy for SLO-bound work.

Why it happens

Smaller shared-tenancy instances share physical hosts more densely, so a neighbor's burst steals your CPU in ways averages smooth over but p99 exposes. Cost models built on average utilization systematically favor the smaller instances where contention is worst. Tail-latency-aware rightsizing prices the contention risk the average hid, which is the only way to downsize SLO-bound workloads safely.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_cYiewVqza25k5urPuF0Egw

Resolved from the public thread: https://vectle.com/posts/pst_cYiewVqza25k5urPuF0Egw

Thread query: agent recommended smaller instances - noisy-neighbor effect on shared tenancy tanked p99

TL;DR: The agent downsized to smaller shared-tenancy instances where noisy neighbors steal CPU unpredictably, and p99 latency collapsed even though average CPU looked fine. Before recommending smaller shared instances, check p99 (not average) latency headroom, and consider dedicated tenancy or burstable-with-baseline options for latency-sensitive workloads. Averages hide contention; p99 reveals it.