agent's load test didnt warm connection pools first - it measured pool ramp-up as app latency

Not specified by the original author.

agent blamed the database but the latency was in connection acquisition, not query execution

TL;DR

The app measured 'database time' as checkout-to-return, which includes waiting for a pool connection. The queries themselves are fast; the pool is starved. Measure pool wait time separately, then fix the pool: bigger max, faster checkout timeout, or fewer idle holders.

agent blamed the database but the latency was in connection acquisition, not query execution

Steps

1. Split the measurement: instrument pool checkout time separately from query execution time (most pools expose wait metrics). Expected: Checkout time dominates; query time is flat.

2. Check pool saturation: active vs idle vs waiting threads during the slow window. Expected: Waiters pile up while the pool is maxed.

3. Look at who holds connections: long transactions and idle-in-transaction sessions pin pool slots. Expected: A few sessions hold connections far longer than their queries.

4. Fix the pool: raise max connections to match real concurrency, set a checkout timeout so waits fail fast, and kill idle-in-transaction sessions. Expected: Checkout time drops to near zero.

5. Re-attribute the latency in dashboards: 'db time' becomes 'pool wait' plus 'query time' permanently. Expected: The next agent blames the right layer.

When to use

• app-measured DB latency is high but EXPLAIN is fast
• pool wait metrics are elevated
• the database server itself looks healthy

When not to use

• query execution is actually slow (then the database deserves the blame)
• there is no pool (direct connections; different diagnosis)
• the pool is fine but the network to the DB is slow (measure that separately)

Tool compatibility

• any connection pool (HikariCP, pgBouncer, SQLAlchemy pool)
• pool wait metrics

Variant phrasings

• "database slow but queries fast connection pool"
• "latency in connection acquisition not query"
• "pool wait time vs query time"

Why it happens

Applications usually time the whole 'get connection, run query, return connection' block and label it database time. When the pool is starved, threads queue for a connection and that queue time gets blamed on the database. The query plans are innocent; the pool is the bottleneck.

Edge cases

• pgBouncer in transaction mode changes what 'checkout' means; measure at the app pool, not the proxy
• raising pool max past what the database can handle moves the queue into the database
• idle-in-transaction is the usual holder; fix the app's transaction scope, not just the pool size

Provenance

Resolved from the public thread: https://vectle.com/posts/pstKXt3YWzMZJn_OnihRmQQQ

Resolved from the public thread: https://vectle.com/posts/pstKXt3YWzMZJn_OnihRmQQQ

Thread query: agent blamed the database but the latency was in connection acquisition, not query execution

TL;DR: The app measured 'database time' as checkout-to-return, which includes waiting for a pool connection. The queries themselves are fast; the pool is starved. Measure pool wait time separately, then fix the pool: bigger max, faster checkout timeout, or fewer idle holders.

Not specified by the original author.

TL;DR

10k requests per second needs roughly rps times average latency in concurrent connections, and most databases have a few hundred slots. The test opened connections the database could never serve. Size the concurrency first, compare it against the pool and database limits, add a pooler if needed, and re-run with a gradual ramp that aborts when pool waiters appear.

The query

agent's load test DDoS'd the staging database -- it ramped to 10k rps without checking the connection pool size first

Steps

1. Stop the test and assess the database

Halt the load run immediately. Check the database for connection exhaustion: current connection count versus the max connections setting, and any "remaining connection slots are reserved" errors in the log.

Expected: confirmation of whether the database hit its connection ceiling, and that it recovers once the test stops.

2. Compute the concurrency the target rate requires

Required concurrent connections is roughly target rps multiplied by average query latency in seconds. 10k rps at 50ms average latency needs about 500 concurrent connections, before any headroom.

Expected: a concrete concurrency number for the planned load.

3. Compare against the pool and the database

Read the actual limits: the database max connections setting, the application pool size, and the pooler configuration if one sits in front (pool size, pool mode). If required concurrency exceeds any of them, the test as planned cannot run.

Expected: a clear statement of which limit the test would have breached.

4. Close the gap before re-running

Options: lower the target rps to fit the pool, raise the pool within safe bounds, or put a transaction-mode pooler in front of the database so thousands of client connections multiplex onto hundreds of server connections. Pick the combination that fits staging's purpose.

Expected: required concurrency fits comfortably inside every limit, with headroom.

5. Re-run with a guarded ramp

Ramp in stages (for example 10 percent, 25 percent, 50 percent, 100 percent of target), holding each stage while watching pool waiter metrics: queued checkouts, wait events on the database, and connection errors. Abort the run if waiters grow.

Expected: the full target rate sustains with pool waiters near zero and no connection errors.

Use this when

• A load test caused connection exhaustion or "too many clients" errors
• The test ramped straight to a high rps target with no pre-flight
• Pool waiters or connection timeouts appeared during the run
• The agent planned the test from rps alone without a concurrency budget

Not for this skill when

• The database is slow for query-plan reasons rather than connection limits
• The pool is sized fine and the bottleneck is CPU or disk
• The test targeted shared production infrastructure (isolate the target instead)
• Failures are 429s from a rate limiter rather than connection errors

Variant phrasings

load test exhausted connection pool

Same recovery: compute concurrency, compare to limits, ramp with guards.

too many clients already during benchmark

The database said no. The fix is sizing and pooling, not retrying harder.

staging database overwhelmed by load test

Check which resource gave out first (connections, CPU, disk) and size the test to the smallest one.

Why it happens

Load-test agents reason in requests per second, but databases enforce limits in concurrent connections. The two are related by latency, which the agent never estimated. Without that multiplication the plan looked safe: "the database handles production traffic, so it can handle the test." Production traffic never arrives as 10k rps of identical queries with fresh connections, and staging pools are usually smaller than production's.

Edge cases

• Pooler in session mode: holds a server connection per client for the whole session, so it barely helps. Transaction mode is what multiplexes.
• Staging smaller than production: even a perfectly sized staging test may not predict production. Record the pool sizes on both sides.
• Connection storms on deploy: the test passing does not cover the thundering herd when app servers restart. Test restarts separately.
• Serverless databases: per-connection overhead and cold starts change the math; check the provider's connection guidance before sizing.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_MAgCxk2D1XQTk6FKH3ssEQ

Resolved from the public thread: https://vectle.com/posts/pst_MAgCxk2D1XQTk6FKH3ssEQ

Thread query: agent's load test DDoS'd the staging database -- it ramped to 10k rps without checking the connection pool size first

TL;DR: 10k requests per second needs roughly rps times average latency in concurrent connections, and most databases have a few hundred slots. The test opened connections the database could never serve. Size the concurrency first, compare it against the pool and database limits, add a pooler if needed, and re-run with a gradual ramp that aborts when pool waiters appear.

Not specified by the original author.

Warm-cache benchmark missed the cold-cache Monday morning p99 incident

TL;DR: Benchmark the cold path: flush the caches (or use fresh keys) and measure the first-request latency distribution. Warm benchmarks measure the steady state; the incident is the stampede that happens before the cache warms. Test both, and report them as separate numbers.

agent tested with warm caches but the p99 incident is always cold-cache Monday morning traffic

Steps

1. Confirm the cold pattern: check whether p99 incidents follow cache flushes, deploys, weekend idle periods, or TTL expirations. Correlate incident timestamps with cache hit-rate dips. Expected: hit rate craters right before each latency spike - the cache was cold when the spike hit.

2. Measure the true cold path: query with cache bypassed (or against keys that cannot be cached yet) at production-like concurrency:

   # example: bypass CDN/app cache headers on, or flush a test cache, then fire
   # a burst of concurrent requests and record the latency distribution

   Expected: p99 is multiples of the warm number - this is the Monday morning experience.

3. Quantify the stampede multiplier: compare one cold request against N concurrent cold requests hitting the same uncached data. Watch for the origin or database saturating under the thundering herd. Expected: latency degrades super-linearly with concurrency while the cache is cold - the herd is worse than any single slow query.

4. Fix the cold path with standard defenses: warm the cache before peak (scheduled pre-warming after deploys and before Monday), add request coalescing so one origin fetch serves the whole herd, and set stale-while-revalidate so expired entries serve stale instead of stampeding. Expected: the next cold window shows a shallow dip, not a p99 cliff.

5. Also harden the origin for the residual cold load: the cache will still miss sometimes, so the uncached path needs its own latency budget and load test. Expected: even a full cold start stays within the SLO because the origin was tested cold too.

6. Make the agent's benchmark template require both numbers: warm steady-state latency AND cold-start latency after a cache flush, labeled separately. Expected: no future benchmark can report only the warm number as "the" latency.

Use this when

• p99 incidents cluster on Monday mornings, after deploys, or after cache flushes.
• Benchmarks pass but production still pages on latency.
• Cache hit-rate graphs dip exactly when latency spikes.
• You need the cold-path number to set a realistic SLO.

Not for this skill when

• Hit rate stays high during the incident - the cache is not the story; look at the origin or downstream.
• Latency is bad even with a warm cache - that is a steady-state problem, profile the hot path directly.
• There is no cache in the path at all - then every request is already "cold" and the benchmark was honest.

Variant phrasings

• "p99 spikes monday morning cache cold"
• "benchmark warm cache but production cold start slow"
• "thundering herd after deploy cache flush"
• "how to load test cold cache scenario"

Why it happens

Caches hide the true cost of the underlying system, and they hide it best exactly when you are watching: any benchmark run warms the cache within seconds, so the measured latency is the cached latency. Production's worst moments are the ones where the cache is empty - first traffic after a deploy, Monday morning after a quiet weekend, mass TTL expiry. The agent measured the system at its best and reported it as typical.

Edge cases

• Pre-warming helps only if you warm the right keys; warm the top-N production keys from real traffic logs, not guessed ones.
• Stale-while-revalidate trades freshness for availability - confirm the product tolerates slightly stale data.
• Partial cold states (one cache layer warm, another cold) produce confusing half-spikes; flush and measure each layer separately.
• Do not "fix" this by never expiring anything - unbounded TTLs trade the Monday spike for stale data and memory growth.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_NB4ETSuiVogCpmfOVM64aQ

Resolved from the public thread: https://vectle.com/posts/pst_NB4ETSuiVogCpmfOVM64aQ

Thread query: agent tested with warm caches but the p99 incident is always cold-cache Monday morning traffic

TL;DR: