# agent's before and after runs used different instance types - the comparison is meaningless

## TL;DR

Different instance types have different CPUs, memory bandwidth, and noisy-neighbor profiles, so a before/after across them measures the hardware, not the change. Re-run both on the same instance type, ideally the same host. A comparison across hardware is not a comparison.

```text
agent's before and after runs used different instance types - the comparison is meaningless
```

## Steps

1. Check the instance types in the run metadata: before and after must match exactly (family, size, and ideally placement).
   Expected: You find the mismatch.

2. Re-run both versions on the same instance type, back to back, on quiet hosts.
   Expected: The hardware variable is eliminated.

3. Compare the honest numbers: the claimed improvement usually shrinks or reverses.
   Expected: The real delta is visible.

4. Pin the instance type in the benchmark config and assert it in the harness before running.
   Expected: Future runs cannot silently change hardware.

5. Record instance type, kernel, and noisy-neighbor signals in every benchmark result.
   Expected: Results carry their hardware context.

## When to use

- before and after ran on different instance types
- a benchmark improvement looks too large
- comparing results across environments

## When not to use

- the change is hardware-independent (algorithmic complexity; still re-run to be sure)
- you are benchmarking the hardware itself (then the difference is the point)
- both runs are on the same type but different hosts (that is noise, not invalidity)

## Tool compatibility

- any cloud benchmark workflow
- run metadata capture

## Variant phrasings

- "before after different instance types"
- "benchmark different hardware meaningless"
- "compared across instance types"

## Why it happens

Instance types differ in CPU generation, clock behavior, memory bandwidth, and virtualization overhead. A move from an older to a newer generation alone can show 20-30% 'improvement' with zero code change. The agent measured the hardware upgrade and attributed it to its patch.

## Edge cases

- even the same instance type on different hosts varies (noisy neighbors); repeat runs and take medians
- burstable instances (t-family) have CPU credits that distort short benchmarks badly
- ARM vs x86 is the most extreme version of this; never compare across architectures

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_uMzYZlPzTjgIfyN_x28axA