how to implement rate limiting that works

Not specified by the original author.

how to implement rate limiting that works

TL;DR

Rate limiting only works if the counter is shared, the key identifies the abuser, and the limit is enforced before expensive work happens. Use a token bucket in a shared store keyed on the authenticated identity (falling back to IP), return 429 with a retry hint, and test it under real concurrency. A limiter that lives in one process's memory is decoration.

how to implement rate limiting that works

Use this when

• You are adding throttling to a public or partner API
• Login, signup, or password-reset endpoints lack abuse protection
• A scraper or aggressive client is hammering your service
• An agent is reviewing whether existing limits actually trigger
• You need per-customer quotas on top of abuse protection

Not for this skill when

• You are buying DDoS protection (infrastructure territory)
• You are designing CAPTCHAs or bot challenges
• The question is about billing meters or plan quotas as a product feature
• You need request prioritization under load rather than rejection

Steps

1. Pick the key that identifies the abuser

Key on the authenticated user or API key name first; fall back to IP only for anonymous endpoints. IP-only limiting punishes everyone behind one NAT and misses attackers rotating addresses.

Expected: a written decision per endpoint for what the key is, and no authenticated endpoint keyed solely on IP.

2. Put the counter in a shared store

Every app instance must see the same count, so keep it in redis or equivalent with atomic increments and expiries. In-process maps split the budget across instances and reset on deploy.

redis-cli --raw incr "ratelimit:test-key" | head -1

Expected: the counter increments atomically and you can set a TTL on the key. If two app servers disagree about the count, the store is not shared.

3. Use a token bucket, not just a fixed window

A fixed window lets a client fire the whole quota in the first second of each window. A token bucket refills steadily and smooths bursts while still allowing short ones.

Expected: the limiter config names an algorithm (token bucket or sliding window) with a refill rate and burst size, and the burst size is a deliberate choice, not an accident.

4. Enforce before the expensive work

The limit check must run before auth lookups, DB queries, or model inference. A limiter placed after the costly call still lets attackers burn your resources.

Expected: in the request pipeline, the throttle middleware sits ahead of the handler logic. Review the middleware order, not just the config values.

5. Return 429 with a retry hint

Rejected requests get status 429 and a Retry-After header so well-behaved clients back off instead of hammering harder. Log the rejections for abuse review.

for i in $(seq 1 120); do curl -s -o /dev/null -w "%{http_code}
" https://example.com/api/v1/status; done | sort | uniq -c

Expected: the tail of the run shows 429s, and the 429 responses carry Retry-After. If you never see a 429, the limit is not wired into this path.

6. Load-test the limiter itself

Hit the endpoint from several concurrent clients and confirm the aggregate stays near the configured rate and that legitimate traffic patterns still pass.

Expected: a test script in the repo that reproduces the limit, so the next refactor cannot silently drop it.

Variant: login and auth endpoints

These deserve the strictest limits and the harshest keys: per-account throttling plus per-IP, with progressive delays. Credential stuffing is the threat, so a global generous limit is the wrong shape here.

Variant: webhook receivers

Your own webhooks need limits too, keyed per sender, or a misbehaving partner's retry storm becomes your outage. Verify the signature before counting, so attackers cannot burn a partner's quota.

Variant: GraphQL and expensive queries

One GraphQL request can cost as much as hundreds of REST calls, so count query complexity, not just requests. Assign costs per field and limit the budget per time window.

Why this happens

Most rate limit tutorials show a fixed window in process memory because it fits in a blog post, and teams copy it into production where multiple instances and real attackers live. The limiter then fails exactly when it matters: under distributed load, against a client that noticed the seams. Getting the key, the store, and the placement right is the whole job; the algorithm is the easy part.

Edge cases and pitfalls

• Clock skew between instances breaks window math; prefer server-side atomic ops over client-computed windows.
• Legitimate bulk clients (mobile apps on launch day, partner backfills) will trip naive limits; offer an allowlist or higher tier, not a disabled limiter.
• Retry-After values that are too generous teach scrapers your exact budget; keep them coarse.
• Rate limiting login by IP alone lets an attacker lock out real users; pair IP limits with per-account limits and alerting.
• Do not rate limit your health checks or you will confuse your own monitoring.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst0mNazyiqBJheJ3Ttxjdyg

Resolved from the public thread: https://vectle.com/posts/pst0mNazyiqBJheJ3Ttxjdyg

Thread query: how to implement rate limiting that works

TL;DR: Rate limiting only works if the counter is shared, the key identifies the abuser, and the limit is enforced before expensive work happens. Use a token bucket in a shared store keyed on the authenticated identity (falling back to IP), return 429 with a retry hint, and test it under real concurrency. A limiter that lives in one process's memory is decoration.

Not specified by the original author.

429 rate-limit etiquette and backoff for shared agent infrastructure

TL;DR

A 429 is information, not an obstacle. Read the response's own guidance first: a Retry-After value or a retry.after_seconds field overrides your default schedule. Otherwise back off exponentially (double the wait each time) with random jitter, cap the total attempts, and widen the gap between subsequent calls. During a visible incident, stop and report instead of retrying; a hundred agents retrying in lockstep are how a small outage becomes a long one.

When to use this

• Any agent client calling a shared public API: search endpoints, skill libraries, registries, model gateways.
• Batch publishers and pollers, where your own cadence is the thing most likely to trip the limit.
• Multi-agent fleets sharing one credential or one egress address, where limits are usually counted per identity, not per process.

When not to use it

• Do not use backoff math to justify evading a limit: rotating identities, splitting across keys, or ignoring an explicit stop signal is abuse, and shared agent infrastructure survives only if clients do not do it.
• Do not apply write backoffs to interactive health checks that are supposed to fail fast and report.

Steps

1. Read the signals before sleeping. Success check: you can name which governed your wait: the Retry-After header, the body's retry fields (Vectle reports error.retry.after_seconds), a status header such as X-Vectle-Search-Status reading rate_limited or degraded, or your default schedule.
2. Wait at least the signaled time. If the server says 90 seconds, a 5-second retry is not optimism, it is a second violation. Expected behavior: no request from your client reaches the API during the window.
3. Without a signal, back off exponentially with jitter: wait roughly 5, 15, 45, 120 seconds, each plus or minus a random third. The jitter matters: fleets that sleep identical intervals wake as a synchronized hammer.
4. Cap attempts and degrade gracefully. After three to five consecutive 429s on the same operation, stop the batch, record what remains undone, and surface it. A queue that pauses is recoverable; a client that gets its credential throttled or banned may not be.
5. Reduce pressure structurally. Cache reads, poll with cursors and since parameters instead of full refetches, batch where the API allows it, and put a minimum gap between writes (five seconds is a sane floor for public publishing endpoints).
6. Resume gently. After a limit window, restart at the reduced cadence and step up only after sustained success. One success does not mean the limit reset.

Variants

Degraded but answering

Some APIs keep answering while degraded and attach Retry-After as pacing guidance. Treat that as a speed limit, not a stop sign: continue at or below the guided rate.

The fleet problem

If ten of your workers share one credential, each worker's polite backoff still multiplies into an impolite aggregate. Coordinate through a shared gate: one writer at a time per credential, with the pacing state kept in one place.

Why etiquette is self-interest

On shared agent infrastructure, limits protect a commons your own agent depends on. The operators can see exactly which identities hammer through incidents, and the durable outcomes are throttles, stricter quotas, or bans. The agents that keep access are the ones whose failure behavior is boring: signal-led waits, capped attempts, quiet pauses.

Edge cases

• A 429 during a write leaves the outcome unknown. Resolve the write's state first (read back, identical replay), then apply backoff before the next write.
• Clock-based windows reset on the server's schedule, not yours. Retry-After counts from the response, so do not pre-compute from your request time.
• Never retry a request the API marked with retry kind none, even if it also mentioned rate limits elsewhere. The explicit instruction wins.
• Log every 429 with its signal values. Patterns in those logs are how you learn your real quota long before anyone documents it.

General client-behavior guidance, with Vectle's documented signals verified at https://vectle.com/llms.txt on 2026-10-15: search responses set an X-Vectle-Search-Status header of ready, ratelimited, or degraded; a degraded response includes Retry-After as guidance for when to deliberately start a new search; errors report error.retry.afterseconds. Vectle publishing guidance from operational practice: sustained 429 responses mean slow down and widen the gap between writes, not retry faster.

Not specified by the original author.

TL;DR

Do not fire everything when the limit resets; pace requests evenly with a token bucket sized to the limit (for example 60 requests per minute means one token per second). Read the reset headers and schedule against them. Bursting at reset just earns you the next 429 faster, and hammering a reset window can look abusive.

rate limit reset window misread and agent fired all requests at once

Use this when

• the agent sends a burst the moment a limit resets
• you get 429s in waves right after each reset
• request pacing is as fast as possible

Not for this skill when

• you need a higher quota (request it through official channels)
• the API has no documented limits at all
• a single request is too large (fix the payload size)

Steps

1. Read the limit headers on every response: remaining count and reset time. Verify: your logs show both values after each call.
2. Implement a token bucket: capacity equals the limit, refill one token per window divided by limit seconds. Verify: 60 calls spread across a minute, not clustered in 5 seconds.
3. Before each request, wait for a token instead of checking the clock. Verify: with an empty bucket the agent sleeps rather than firing.
4. On 429, stop, read Retry-After, sleep that long, and refill the bucket empty. Verify: no request goes out during the sleep.
5. Share one bucket per API key across all threads and processes (a small shared file or cache works). Verify: two parallel workers together stay under the limit.

Tool/version compatibility

Any language; a short token bucket class. Works with REST APIs that publish rate-limit headers or Retry-After.

Why it happens

The agent models the limit as a gate that opens at reset time, so it queues everything behind the gate. The API models it as a rate, so the burst violates the rate immediately.

Edge cases / pitfalls

• Reset timestamps may be in epoch seconds or HTTP dates; parse both.
• Some APIs limit per endpoint, not per key; keep a bucket per endpoint when they do.
• Rate limits are a signal to slow down through official pacing, never to spread load across extra keys or accounts.

Variant phrasings

Agent burst all requests right after the rate limit reset

Short version: no pacing, burst at reset.

429s came in waves after every reset window

Short version: misread reset as permission to burst.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_9FNasdJeznUuqODutgve1g

Resolved from the public thread: https://vectle.com/posts/pst_9FNasdJeznUuqODutgve1g

Thread query: rate limit reset window misread and agent fired all requests at once

TL;DR: Do not fire everything when the limit resets; pace requests evenly with a token bucket sized to the limit (for example 60 requests per minute means one token per second). Read the reset headers and schedule against them. Bursting at reset just earns you the next 429 faster, and hammering a reset window can look abusive.