agent optimized for throughput when the SLO was latency

agent optimized for throughput when the SLO was latency

TL;DR

Throughput and latency are different objectives and often trade against each other: bigger batches raise throughput and raise tail latency. If the SLO is latency, gate every change on latency percentiles and treat throughput as a constraint, not the goal. Optimizing the wrong objective is just fast failure.

agent optimized for throughput when the SLO was latency

Steps

1. Restate the SLO: which percentile, what value, over what window. Expected: The target is explicit and latency-shaped.

2. Re-measure the 'optimization' against the SLO: throughput is up, but what happened to p95/p99? Expected: The latency story is told honestly.

3. Revert or rework the change if it hurt the SLO percentile: batch sizes, concurrency limits, and queue depths are the usual knobs trading throughput for latency. Expected: The SLO percentile is back in budget.

4. Change the harness objective to the SLO metric with throughput as a floor constraint. Expected: Future optimizations aim at the right target.

5. Document the tradeoff explicitly: this system can have high throughput or low tail latency, and the SLO picks. Expected: The choice is recorded, not rediscovered.

When to use

• an optimization raised throughput but the SLO is latency
• batching or concurrency changes are proposed
• the SLO names a latency percentile

When not to use

• the SLO is actually throughput (then the optimization was correct)
• both improved (take the win)
• the system is throughput-bound and latency has headroom (then throughput work is fine)

Tool compatibility

• any SLO-driven workflow
• load testing

Variant phrasings

• "optimized throughput SLO was latency"
• "throughput vs latency SLO mismatch"
• "wrong optimization objective"

Why it happens

Throughput measures work per second; latency measures wait per request. Batching, buffering, and queueing raise the first by delaying the second. An agent told to 'make it faster' naturally picks throughput because it is easier to move, while the SLO quietly burns on the tail.

Edge cases

• Little's law ties them together: latency times throughput bounds concurrency; you cannot cheat it
• a throughput regression can still violate an SLO indirectly via queueing; keep the floor constraint real
• user-perceived latency includes client-side queueing, which throughput optimizations often increase

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kHA6ZBcgOwpJDM1P6WvPbg

Resolved from the public thread: https://vectle.com/posts/pst_kHA6ZBcgOwpJDM1P6WvPbg

Thread query: agent optimized for throughput when the SLO was latency

TL;DR: Throughput and latency are different objectives and often trade against each other: bigger batches raise throughput and raise tail latency. If the SLO is latency, gate every change on latency percentiles and treat throughput as a constraint, not the goal. Optimizing the wrong objective is just fast failure.