posthog capture queue flushed on every event and agent hit request limits in minutes

TL;DR

Accumulate events in memory and flush on a size threshold or a time interval, whichever comes first, from a background timer so capture never blocks. On 429 or 5xx, back off and keep buffering up to a cap, then drop the oldest with a counter. Request volume drops by orders of magnitude and rate limits stop being hit.

posthog capture queue flushed on every event and agent hit request limits in minutes

Use this when

• analytics capture fires one request per event
• request limits are hit within minutes of a busy period
• event capture slows down the hot path
• an outage in the analytics endpoint threatens to crash the app

Not for this skill when

• events are malformed or missing properties
• you need real-time per-event delivery guarantees
• the problem is dashboard queries rather than ingestion

Steps

1. Switch the capture path to batching: accumulate events in memory and flush on size (for example 50 events) or interval (for example 30 seconds), whichever comes first. Expected: request volume drops by orders of magnitude.
2. Run the flusher on a background timer so event capture never blocks the hot path. Expected: capture calls return in microseconds.
3. On 429 or 5xx, back off exponentially and keep buffering up to a cap, then drop the oldest events with a counter. Expected: an outage degrades to sampling, not a crash.
4. Treat rate-limit responses as a signal to slow down, never as a cue to retry harder or work around the limit. Expected: limits stop being hit within minutes.
5. Emit the buffer size and dropped-event count as metrics. Expected: you can see pressure building before data is lost.

Variant phrasings

analytics flushed on every event and hit rate limits

Short version: batch on size or interval from a background timer.

capture queue overwhelmed the endpoint in minutes

Short version: accumulate in memory, flush in batches, back off on 429.

event capture slowed the hot path

Short version: move flushing off the hot path onto a background timer.

Compatibility: applies to PostHog and any batched ingestion API; the pattern is sender-side and needs no SDK version. Read the project key with os.environ.get("POSTHOG_KEY") and never log it.

Why it happens

The capture call flushed the queue on every event, turning each user action into its own HTTP request. A busy minute meant thousands of requests, and the endpoint's rate limit did exactly what rate limits do. The queue existed but was never allowed to do its job.

Edge cases / pitfalls

• A pure size trigger stalls on low traffic; the interval trigger guarantees freshness.
• Dropping the oldest events loses data silently unless the dropped counter is monitored; alert on it.
• Background flushers must flush on shutdown or the last partial batch is lost; hook process exit.
• Batching adds seconds of delay; if any event truly needs immediacy, send that one type outside the batch.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ToBNptul9K44VXzgb4TIxA