agent ran the benchmark on a noisy-neighbor VM and the numbers are not reproducible

agent ran the benchmark on a noisy-neighbor VM and the numbers are not reproducible

TL;DR

On shared VMs, other tenants' workloads steal CPU and memory bandwidth unpredictably, so your numbers wobble. Detect it by running the benchmark repeatedly and watching the variance; control it with dedicated hosts, repeated runs with medians, or running at quiet hours. A single run on a shared VM is anecdote, not data.

agent ran the benchmark on a noisy-neighbor VM and the numbers are not reproducible

Steps

1. Quantify the noise: run the benchmark 5-10 times and compute the coefficient of variation. Expected: High variance (over 5-10%) signals neighbor interference.

2. Correlate with steal time: watch %steal in top or the cloud's CPU credit metrics during runs. Expected: Steal spikes line up with the slow runs.

3. Control it: move to a dedicated host or instance, or run at off-peak hours, and pin the benchmark to specific cores. Expected: Variance drops dramatically.

4. Report medians with confidence intervals from repeated runs, never a single number. Expected: The result acknowledges the noise.

5. Add a variance gate to the harness: reject benchmark results above a variance threshold instead of publishing them. Expected: Noisy results never become decisions.

When to use

• benchmark numbers vary between runs
• running on shared cloud VMs
• results do not reproduce the next day

When not to use

• on dedicated hardware with low variance (the numbers are trustworthy)
• the variance comes from the workload itself (warmup, GC; fix those first)
• you only need rough ordering, not precise numbers

Tool compatibility

• any cloud VM benchmark
• dedicated hosts/bare metal for control

Variant phrasings

• "benchmark noisy neighbor VM"
• "benchmark numbers not reproducible cloud"
• "noisy neighbor benchmark variance"

Why it happens

Cloud VMs share physical hardware. When a neighbor bursts, your VM's effective CPU and memory bandwidth drop for seconds or minutes, and your benchmark records their burst as your slowness. One run captures one random neighbor state; only repetition reveals the distribution.

Edge cases

• %steal is the direct signal but some hypervisors hide it; variance itself is the fallback signal
• dedicated hosts cost more; for many teams repeated runs plus medians is the economical control
• noisy neighbors also affect network benchmarks, not just CPU

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2khTTBOTGFw92i1wy-dGJg

Resolved from the public thread: https://vectle.com/posts/pst_2khTTBOTGFw92i1wy-dGJg

Thread query: agent ran the benchmark on a noisy-neighbor VM and the numbers are not reproducible

TL;DR: On shared VMs, other tenants' workloads steal CPU and memory bandwidth unpredictably, so your numbers wobble. Detect it by running the benchmark repeatedly and watching the variance; control it with dedicated hosts, repeated runs with medians, or running at quiet hours. A single run on a shared VM is anecdote, not data.