agent's oauth token expired during an overnight backfill watch

TL;DR

Access credentials from OAuth flows are short lived by design, often an hour, so any watch, poll, or backfill babysitter that runs overnight will outlive the credential it started with. When it expires, calls start failing with unauthorised errors and an agent that does not distinguish auth failure from job failure may report the backfill as broken or just go quiet. Build the watch on a refreshable credential, renew it before expiry inside the loop, and treat an auth failure as a renewal event, then resume watching the same backfill from its recorded state.

agent's oauth token expired during an overnight backfill watch

Use this when

• An overnight watch stops reporting and the logs show unauthorised errors starting at a consistent time after start.
• The backfill itself continued fine in the orchestrator while the watcher went blind.
• Short watches by the same agent work, and only long ones die.

Not for this skill when

• The credential was revoked or the account's access was removed. Renewal will fail and a human must restore access deliberately.
• The watcher died because its host or session ended. That is a process lifetime problem, not a credential lifetime one.

Steps

1. Separate the two patients: check the backfill's own state in the orchestrator and note whether it completed, failed, or is still running, independent of the watcher. Note the backfill's true state. Verify: the backfill is judged from its own system, not from the blind watcher
2. Read the watcher logs and find the first unauthorised error and its time relative to the watch start. Note whether the gap matches the credential's known lifetime. Verify: expiry is confirmed by the timing pattern
3. Identify the credential type the watch uses and whether the flow gave it a refresh path. If the current setup holds only a short lived access value with no renewal, switch the watch to the refreshable flow your platform supports, stored in the credential store. Note the flow in use. Verify: the watch can renew itself instead of borrowing a human session
4. Add renewal to the watch loop: check the credential's expiry before each polling cycle, renew when it is near, and on an unexpected unauthorised response, renew once and retry the read before reporting any job state. Note the renewal logic and its one retry limit. Verify: expiry becomes a handled event inside the loop
5. Resume the watch against the same backfill from its recorded progress, confirm current state is reported correctly, and let it run past one full credential lifetime as the test. Note it survives a renewal boundary. Verify: the watch is proven across the boundary that killed it

Variant phrasings

oauth access credential expired long running polling agent refresh

The refresh pattern searched directly.

overnight job watch stopped 401 unauthorized after one hour

The timing symptom phrasing, with the common one hour lifetime.

Compatibility: Any agent or script holding an OAuth based session against a data platform API, with refresh supported by the provider. Exact lifetimes vary by provider and by the scopes and flow used.

Why it happens

Short lived access credentials are a security feature working as intended, and they collide with an operational pattern built for static keys: start a watcher, let it poll for hours. The watcher's failure mode is treacherous because it fails looking outward: its reports stop or turn negative while the thing it watches is healthy, and an overnight incident review can burn hours on a healthy backfill. The one hour signature is the tell, since genuine job failures do not schedule themselves by the credential clock. Any agent workflow expected to outlive an hour needs renewal as a first class step.

Edge cases / pitfalls

• A refresh credential can itself be revoked or expire under some providers' policies. The watch must surface a renewal failure as a loud auth incident, not retry it forever.
• Never write the credential values into the watch log while debugging expiry. Log expiry times and renewal outcomes only.
• Concurrent watchers sharing one refreshable credential can invalidate each other's renewals on some providers. Give a long watch its own credential where the provider requires it.
• When resuming a watch, re-read the job state fresh. Decisions queued from the blind period may be based on a state that has moved on.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_3I1HpXU8bxRUzsvkDdNzew