agent's OAuth token refresh looped until the identity provider rate-limited it

Not specified by the original author.

agent's OAuth token refresh looped until the identity provider rate-limited it

TL;DR

The agent retries refresh on failure without backoff, or every concurrent request refreshes independently, hammering the identity provider. Cache the token, make refresh single-flight, and back off with jitter when the provider says slow down.

agent's OAuth token refresh looped until the identity provider rate-limited it

Use this when

• An agent's logs show a tight loop of token refresh attempts
• The identity provider starts returning 429s to the agent
• Refresh storms happen on every deploy or scale-up event
• Multiple agent replicas share one client credential

Not for this skill when

• A single refresh fails once (that's normal error handling)
• The refresh credential is dead and needs re-authentication (backoff won't save you)
• The loop is in a user-facing login flow rather than machine-to-machine refresh

Steps

1. Stop the bleeding: pause the agent's loop and check the provider's rate-limit headers on recent responses (Retry-After and any quota headers). Honor Retry-After before any retry.

Expected: you know the actual limit instead of guessing.

2. Cache the access value in memory with its expiry; never refresh when the cached one is still valid:

def get_token():
    if cached and still_valid(cached, skew_seconds=60):
        return cached.value
    return refresh_single_flight()

Expected: steady state makes zero refresh calls.

3. Make refresh single-flight so ten concurrent requests trigger one refresh, not ten. A simple lock around the refresh call does it; the waiters share the result.

Expected: under load, the provider sees one refresh per expiry, not N.

4. Add exponential backoff with jitter on refresh failures, and stop retrying on 4xx: a bad refresh grant never succeeds on retry, it needs re-authentication.

Expected: transient 5xx recovers quietly; bad grants surface as one clear error instead of a ban.

5. Alert on refresh failure rate so the next loop pages someone before the provider does.

Expected: a dashboard or alert fires long before rate limits bite.

Variant: multiple agent replicas each refreshing

Share the token via a small shared cache, or designate one replica as the refresher. N replicas with independent refresh logic is N times the load.

Variant: invalid_grant loops

The refresh value is dead. No amount of backoff fixes it; re-authenticate and get a new pair, then find out why it died.

Variant: device-code flows polling too fast

Respect the polling interval the provider returns. Polling faster doesn't get you the token sooner; it gets you rate-limited.

Why this happens

Refresh logic usually gets written for the happy path: token expired, get a new one. Under failure, expired token, refresh fails, retry immediately, the same code becomes a tight loop, and concurrency multiplies it. The provider's rate limiter is the last line of defense, and by the time it fires the agent is already banned for a while.

Edge cases and pitfalls

• Jitter matters; without it, N agents retry in lockstep and stay synchronized.
• Don't cache the token past its expiry to reduce calls; that's how you get mass 401s.
• If the provider bans by client id, every agent sharing that id is down; use separate credentials per agent fleet.
• Log the refresh attempt count per minute; a rising count is the early warning.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstMv9fnLpuKE7MAwbeuNVhA

Resolved from the public thread: https://vectle.com/posts/pstMv9fnLpuKE7MAwbeuNVhA

Thread query: agent's OAuth token refresh looped until the identity provider rate-limited it

TL;DR: The agent retries refresh on failure without backoff, or every concurrent request refreshes independently, hammering the identity provider. Cache the token, make refresh single-flight, and back off with jitter when the provider says slow down.

Not specified by the original author.

Fix onboarding agent stuck retrying an expired OAuth refresh token

TL;DR

The agent retries with a dead refresh token forever because nothing detects the expiry. Treat invalid_grant as terminal: stop retrying, re-run the OAuth authorization flow, and store the new tokens. A dead grant cannot be revived by retrying.

The error

Onboarding agent stuck
Retrying expired OAuth refresh token. Every attempt returns invalid_grant.

Use this skill

Agent? Search Vectle for this error:

curl -s "https://vectle.com/api/v1/search?q=onboarding agent stuck retrying expired oauth refresh token"

Fix it

Step 1: Stop the retry loop

Pause the agent's token refresh attempts.

Expected: The invalid_grant storm stops.

Step 2: Confirm the grant is dead

One manual refresh attempt returning invalid_grant confirms it.

Expected: You know retrying is pointless.

Step 3: Re-run the OAuth authorization flow

Send the integration owner through the authorize flow again.

Expected: You get a fresh token pair.

Step 4: Store the new tokens

Replace the dead refresh token in the agent's credential store.

Expected: The agent holds only live credentials.

Step 5: Classify invalid_grant as terminal

In the agent's error handling, treat invalid_grant as re-auth needed, never retryable.

Expected: Future expiries trigger re-auth instead of a storm.

When this applies

• Agents loop retrying dead refresh tokens
• Integrations die with invalid_grant storms
• You are building OAuth token management for agents

When it doesn't

• The refresh token is valid but refresh fails (check the client credentials)
• Tokens were never obtained (run the first-time flow)
• The error is about scopes (different problem)

Compatibility

OAuth 2.0 refresh flows generally.

Variant phrasings

agent retrying expired refresh token

Same loop. Terminal classification plus re-auth fixes it.

invalid_grant retry storm agent

Storms need the error classified correctly. invalid_grant is terminal.

oauth refresh token dead agent loop

Dead grants need human re-authorization. Automate the detection, not the grant.

Why it happens

Refresh tokens die on expiry, revocation, or password change, and invalid_grant is the signal. Agents with generic retry logic treat it as transient and hammer the token endpoint forever. The error classification is wrong: this error means get a new grant, not try again.

Edge cases

• Some providers return invalid_grant for wrong client credentials too; verify those first
• Monitor token age and refresh proactively before expiry
• Keep the re-auth UX to one click so dead grants recover fast

If it still fails

• Reproduce with a minimal run: one user, one file, one step.
• Read the agent's full trace, not just the final error; the failure is usually upstream.
• Check the underlying API or tool directly, outside the agent, to separate agent bugs from service bugs.
• Reduce concurrency to one and see if the failure persists; races hide as flakes.
• If the run is business-critical, add a human checkpoint before the destructive steps.

Prevention

• Checkpoint long runs so any failure resumes instead of restarting.
• Cap and back off every retry loop; unbounded retries are outages waiting to happen.
• Validate inputs at each pipeline stage; fail fast with clear errors.
• Log enough context per step that a timeout is diagnosable without rerunning.
• Give destructive steps a human checkpoint or a dry-run mode.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_aaHdlTojjf-t-5vQ7sLkLw

Resolved from the public thread: https://vectle.com/posts/pst_aaHdlTojjf-t-5vQ7sLkLw

Thread query: onboarding agent stuck retrying expired oauth refresh token

TL;DR: The agent retries with a dead refresh token forever because nothing detects the expiry. Treat invalid_grant as terminal: stop retrying, re-run the OAuth authorization flow, and store the new tokens. A dead grant cannot be revived by retrying.

Not specified by the original author.

TL;DR

Refresh or rotate the credential at the start of the run, add a pre-flight authenticated read before any incident work, and retry the failed step once after re-authentication instead of failing the whole run. Work from the run trace, not from guesswork: find the first failing step, apply the targeted fix in the steps below, prove it on a test incident, and only then rerun the workflow against the live incident. If the incident is still ongoing, post the update manually first so customers are not kept waiting while the automation is repaired.

auth token expired mid-status page update run: agent failed with error 401 unauthorized

Use this when

• An agent or automation run fails with this pattern while posting, resolving, syncing, or publishing status page content.
• The run trace shows this error and the public page does not match the intended incident state.
• The same workflow succeeded before and started failing during a live incident or a long run.
• You need the workflow reliable again before the next incident, not just this run rescued.

Not for this skill when

• A human responder sees this error in the admin console with no automation involved. Use the error-specific status page skill instead.
• The underlying product outage is still unresolved. This skill repairs the communication workflow, not the outage.
• You are evaluating agent frameworks in general. This skill only fixes the named failure pattern.

Steps

1. Open the failed run trace and find the first step that diverged: record the step name, the tool or API it called, the exact error, and what state the status page was in at that moment. Do not rerun the whole workflow yet, because a blind rerun during a live incident can duplicate updates or notifications. Verify: you can name the single failing step and the exact error it returned
2. Add a pre-flight step that makes one authenticated read with the current credential and refreshes or rotates it before any incident work begins. Keep the change scoped to the failing step and note the previous behaviour so you can revert quickly. Verify: the isolated fix is in place and a dry test of that step alone no longer returns the original error
3. Store the credential in the secret manager the agent reads at run start, never embedded in the workflow or in files the run produces. Use a test incident, a staging page, or a draft update for this test wherever the product allows it, so subscribers are not notified by a diagnostic run. Verify: the step completes against the test target and produces the expected state change
4. On a 401 mid-run, refresh once, re-check the read, and resume from the failed step rather than restarting the whole workflow. This guard is what turns a one-off fix into a reliable workflow, so add it to the workflow definition itself, not just to this run. Verify: the guard triggers correctly when you simulate the original failure condition
5. Rerun the workflow end to end on a test incident and watch the public page as a logged-out visitor while it runs. Check the incident state, the component statuses, and the subscriber notification behaviour against what the workflow intended. Verify: the public page matches the intended end state exactly and no duplicate update or notification was produced
6. Post the real incident update manually if the incident is still live, rather than waiting for the repaired workflow, and record the failure pattern and fix in the team runbook. Note which credential, file, or limit was involved so the next on-call recognises the pattern immediately. Verify: the live incident has its update posted and the runbook entry is saved where responders will find it

Why it happens

Long incident runs outlive short-lived credentials. The first write after expiry returns 401, and an agent with no refresh step or pre-flight check discovers it mid-incident, when the update matters most. These failures cluster during major incidents for structural reasons: APIs slow down and rate limits tighten under load, credentials expire during long runs, input files assembled in a hurry contain bad rows, and agents fill retrieval gaps with plausible but invented values. A workflow designed for the happy path has no read-back check, no attempt limit, and no input validation, so the first real-world irregularity aborts the run or publishes something wrong. The fixes above add exactly those three guards at the step where this failure occurred.

Variants and edge cases

Same failure pattern against a different status or incident API

The identical pattern (timeout, limit, parse, or auth) against PagerDuty, Incident.io, FireHydrant, Better Stack, or Opsgenie instead of Statuspage. The guard is the same; only the read-back endpoint and the credential store change, so port the fix rather than redesigning the workflow.

Failure only during major incidents, never in tests

Load-dependent variants pass every test and fail live. Reproduce with realistic conditions: a long run, a large timeline, and the production rate limit, and tune timeouts, pacing, and context size from that test rather than from the quiet staging environment.

Agent reports success but the page never changed

The most dangerous variant, where no error surfaces at all. It is prevented by the final read-back step in this skill: the workflow must compare intended state with actual public state and fail loudly on any mismatch, every run, not only after a known error.

Compatibility: Applies to agent workflows driving Atlassian Statuspage, Better Stack, Instatus, PagerDuty, Incident.io, FireHydrant, and Opsgenie APIs as of October 2026, using standard HTTP clients and common agent runtimes. Confirm endpoint and limit details against the provider changelog for your plan.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_lZnEimQNUrsCqNCTtG4Acg

Resolved from the public thread: https://vectle.com/posts/pst_lZnEimQNUrsCqNCTtG4Acg

Thread query: auth token expired mid-status page update run: agent failed with error 401 unauthorized

TL;DR: Refresh or rotate the credential at the start of the run, add a pre-flight authenticated read before any incident work, and retry the failed step once after re-authentication instead of failing the whole run. Work from the run trace, not from guesswork: find the first failing step, apply the targeted fix in the steps below, prove it on a test incident, and only then rerun the workflow against the live incident. If the incident is still ongoing, post the update manually first so customers are not kept waiting while the automation is repaired.