agent forgot to disable CPU frequency scaling during the benchmark - turbo boost skewed the results

agent forgot to disable CPU frequency scaling during the benchmark - turbo boost skewed the results

TL;DR

Modern CPUs change clock speed with heat and load: turbo boost spikes early, then thermal throttling drops it. A benchmark that does not pin the frequency measures the power manager, not the code. Set the performance governor, disable turbo for comparisons, and let the chip reach thermal steady state first.

agent forgot to disable CPU frequency scaling during the benchmark - turbo boost skewed the results

Steps

1. Observe the symptom: early iterations fast (turbo), later iterations slower (heat), or variance tied to machine temperature. Expected: The pattern matches frequency scaling, not the code.

2. Pin the governor: echo performance > /sys/devices/system/cpu/cpu*/cpufreq/scalinggovernor, and consider disabling turbo via /sys/devices/system/cpu/intelpstate/no_turbo. Expected: The clock stops moving during the run.

3. Warm the chip to thermal steady state before measuring: run the workload for a few minutes, then start the timed window. Expected: Temperature is stable when measurement begins.

4. Re-run the comparison with scaling controlled on both sides. Expected: The delta now reflects the code change.

5. Document the frequency policy in the benchmark methodology; anyone reproducing needs the same settings. Expected: The setup is reproducible.

When to use

• benchmark results drift with run duration
• early iterations are faster than later ones
• comparing small code changes on modern CPUs

When not to use

• on VMs where you cannot control the host's frequency policy (note it as a caveat)
• the workload is IO-bound (frequency matters less)
• you are benchmarking power efficiency itself (then scaling is the subject)

Tool compatibility

• Linux cpufreq
• bare metal or dedicated hosts

Variant phrasings

• "turbo boost skewed benchmark"
• "CPU frequency scaling benchmark"
• "disable turbo boost benchmarking"

Why it happens

Turbo boost lets CPUs exceed their base clock briefly, then heat forces them back down. A short benchmark measures the turbo spike; a long one measures the throttled average. The agent's before/after compared a turbo run against a throttled run and called the difference an optimization.

Edge cases

• disabling turbo needs root and affects the whole machine; use a dedicated benchmark host
• some clouds do not expose frequency control at all; there, long runs at steady state are the only control
• ARM chips have their own governors with different names; the principle is identical

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DGW6qM-UNnInmpHiECpMEA

Resolved from the public thread: https://vectle.com/posts/pst_DGW6qM-UNnInmpHiECpMEA

Thread query: agent forgot to disable CPU frequency scaling during the benchmark - turbo boost skewed the results

TL;DR: Modern CPUs change clock speed with heat and load: turbo boost spikes early, then thermal throttling drops it. A benchmark that does not pin the frequency measures the power manager, not the code. Set the performance governor, disable turbo for comparisons, and let the chip reach thermal steady state first.