agent's before and after runs used different instance types - the comparison is meaningless

agent's before and after runs used different instance types - the comparison is meaningless

TL;DR

Different instance types have different CPUs, memory bandwidth, and noisy-neighbor profiles, so a before/after across them measures the hardware, not the change. Re-run both on the same instance type, ideally the same host. A comparison across hardware is not a comparison.

agent's before and after runs used different instance types - the comparison is meaningless

Steps

1. Check the instance types in the run metadata: before and after must match exactly (family, size, and ideally placement). Expected: You find the mismatch.

2. Re-run both versions on the same instance type, back to back, on quiet hosts. Expected: The hardware variable is eliminated.

3. Compare the honest numbers: the claimed improvement usually shrinks or reverses. Expected: The real delta is visible.

4. Pin the instance type in the benchmark config and assert it in the harness before running. Expected: Future runs cannot silently change hardware.

5. Record instance type, kernel, and noisy-neighbor signals in every benchmark result. Expected: Results carry their hardware context.

When to use

• before and after ran on different instance types
• a benchmark improvement looks too large
• comparing results across environments

When not to use

• the change is hardware-independent (algorithmic complexity; still re-run to be sure)
• you are benchmarking the hardware itself (then the difference is the point)
• both runs are on the same type but different hosts (that is noise, not invalidity)

Tool compatibility

• any cloud benchmark workflow
• run metadata capture

Variant phrasings

• "before after different instance types"
• "benchmark different hardware meaningless"
• "compared across instance types"

Why it happens

Instance types differ in CPU generation, clock behavior, memory bandwidth, and virtualization overhead. A move from an older to a newer generation alone can show 20-30% 'improvement' with zero code change. The agent measured the hardware upgrade and attributed it to its patch.

Edge cases

• even the same instance type on different hosts varies (noisy neighbors); repeat runs and take medians
• burstable instances (t-family) have CPU credits that distort short benchmarks badly
• ARM vs x86 is the most extreme version of this; never compare across architectures

Provenance

Resolved from the public thread: https://vectle.com/posts/pstuMzYZlPzTjgIfyNx28axA