agent's OAuth token refresh looped until the identity provider rate-limited it

agent's OAuth token refresh looped until the identity provider rate-limited it

TL;DR

The agent retries refresh on failure without backoff, or every concurrent request refreshes independently, hammering the identity provider. Cache the token, make refresh single-flight, and back off with jitter when the provider says slow down.

agent's OAuth token refresh looped until the identity provider rate-limited it

Use this when

• An agent's logs show a tight loop of token refresh attempts
• The identity provider starts returning 429s to the agent
• Refresh storms happen on every deploy or scale-up event
• Multiple agent replicas share one client credential

Not for this skill when

• A single refresh fails once (that's normal error handling)
• The refresh credential is dead and needs re-authentication (backoff won't save you)
• The loop is in a user-facing login flow rather than machine-to-machine refresh

Steps

1. Stop the bleeding: pause the agent's loop and check the provider's rate-limit headers on recent responses (Retry-After and any quota headers). Honor Retry-After before any retry.

Expected: you know the actual limit instead of guessing.

2. Cache the access value in memory with its expiry; never refresh when the cached one is still valid:

def get_token():
    if cached and still_valid(cached, skew_seconds=60):
        return cached.value
    return refresh_single_flight()

Expected: steady state makes zero refresh calls.

3. Make refresh single-flight so ten concurrent requests trigger one refresh, not ten. A simple lock around the refresh call does it; the waiters share the result.

Expected: under load, the provider sees one refresh per expiry, not N.

4. Add exponential backoff with jitter on refresh failures, and stop retrying on 4xx: a bad refresh grant never succeeds on retry, it needs re-authentication.

Expected: transient 5xx recovers quietly; bad grants surface as one clear error instead of a ban.

5. Alert on refresh failure rate so the next loop pages someone before the provider does.

Expected: a dashboard or alert fires long before rate limits bite.

Variant: multiple agent replicas each refreshing

Share the token via a small shared cache, or designate one replica as the refresher. N replicas with independent refresh logic is N times the load.

Variant: invalid_grant loops

The refresh value is dead. No amount of backoff fixes it; re-authenticate and get a new pair, then find out why it died.

Variant: device-code flows polling too fast

Respect the polling interval the provider returns. Polling faster doesn't get you the token sooner; it gets you rate-limited.

Why this happens

Refresh logic usually gets written for the happy path: token expired, get a new one. Under failure, expired token, refresh fails, retry immediately, the same code becomes a tight loop, and concurrency multiplies it. The provider's rate limiter is the last line of defense, and by the time it fires the agent is already banned for a while.

Edge cases and pitfalls

• Jitter matters; without it, N agents retry in lockstep and stay synchronized.
• Don't cache the token past its expiry to reduce calls; that's how you get mass 401s.
• If the provider bans by client id, every agent sharing that id is down; use separate credentials per agent fleet.
• Log the refresh attempt count per minute; a rising count is the early warning.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstMv9fnLpuKE7MAwbeuNVhA