support agent oauth expired mid-run: refresh failed at 2am

TL;DR

An OAuth token expiring at 2am kills the unattended run, and a failed refresh means nobody is awake to fix it. Design for the night: refresh proactively well before expiry, treat refresh failure as a page-worthy alert (not a log line), and checkpoint so the run resumes after re-auth instead of restarting. Overnight runs must survive the dark hours without a human.

The query

support agent oauth expired mid-run: refresh failed at 2am

Use this when

• Overnight runs die on expired tokens
• Token refresh fails silently
• You are building unattended agents
• Debugging 2am run failures

Not for

• Initial OAuth app setup
• Choosing authorization grant types
• Interactive user login flows
• SSO configuration

Steps

1. Refresh proactively, not on 401

Schedule token refresh at 70 to 80 percent of token lifetime, in the background, before any call needs it. Refresh-on-401 means the first failure already happened; at 2am that failure is fatal. Proactive refresh moves the risk into waking hours.

Expected output: background refresh firing before expiry.

2. Alert loudly on refresh failure

A failed refresh is a paging event, not a log line. If the refresh endpoint errors or the refresh token is invalid, page the on-call immediately: every minute of delay is an hour of dead runs. Silent refresh failure is how 2am becomes 9am with nothing done.

Expected output: refresh-failure paging with urgency.

3. Checkpoint the run so it resumes after re-auth

Persist progress continuously so a re-authenticated rerun picks up where the dead run stopped. Without checkpoints, the morning rerun restarts from zero and the night's work is lost twice over.

Expected output: durable checkpoints enabling resume.

4. Separate the token health check from the run

A lightweight scheduled check validates the token and refreshes it outside any run's context. The check keeps tokens warm even when no run is active, so runs start with a fresh token instead of discovering expiry mid-work.

Expected output: an independent token health check on a schedule.

5. Design the run to degrade without the token

When auth is truly unavailable, the run should park gracefully: stop making calls, preserve state, and report "waiting on auth" instead of crashing. A parked run resumes; a crashed run loses context.

Expected output: graceful parking on auth unavailability.

Template: the checklist

Overnight OAuth survival:
[ ] Proactive refresh at 70-80 percent of lifetime
[ ] Refresh failure paging, not just logging
[ ] Durable checkpoints for resume after re-auth
[ ] Independent token health check on schedule
[ ] Graceful parking when auth is unavailable

Variant phrasings

oauth expired during overnight job

Steps 1 and 3. Refresh early, checkpoint always.

token refresh failed unattended run

Steps 2 and 4. Page on failure, keep tokens warm.

agent died 2am expired token

Steps 1 and 5. Proactive refresh plus graceful parking.

Why it happens

OAuth tokens are issued with lifetimes measured in hours, and overnight runs are measured in hours: the two clocks collide in the dark. Refresh is usually built and tested during the day, where a failure gets a human in minutes. At night the same failure gets nobody, and the run dies alone. The fix is to move every auth risk into the daylight: refresh early, alert loud, checkpoint always.

Edge cases

• Refresh token expiry: refresh tokens expire too, on longer clocks. Track both lifetimes; the long one still ends.
• Provider outages at night: the refresh endpoint itself can be down. The parking behavior covers this.
• Multiple agents sharing a token - one agent's refresh can invalidate another's. Coordinate or use separate tokens.
• Clock skew on the runner: a skewed clock misjudges expiry. Sync time on the runner.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_cP8ddigM6tYGSD1NkkaOgA