data agent auth token expired mid long-running pipeline: silent re-auth

TL;DR

The pipeline treats its API token as immortal, but tokens expire, usually right in the middle of a long run. Wrap every API call so a 401 triggers one token refresh and one retry, and track expiry so stages refresh before it lapses. The pipeline should survive an expiry the way it survives a retryable network blip.

Symptom

The pipeline runs fine for an hour, then every API call starts returning 401. The agent either crashes, or loops on the failed call forever. Restarting the pipeline works, because the restart fetches a fresh token, which proves the code never handled expiry.

Steps

1. Store the token's expiry alongside the token (from the token response's expires_in, or the exp claim if it is a JWT) instead of treating the token as immortal.

Expected output: the agent knows when the token dies before it happens.

2. Wrap every API call in a client that refreshes on 401 and retries exactly once:

def call_api(fn):
    try:
        return fn()
    except AuthError:
        refresh_token()
        return fn()

Expected output: a mid-run expiry costs one refresh and one retry, not a dead pipeline.

3. Proactively refresh when remaining lifetime drops under a safety margin (a few minutes) at the start of each pipeline stage.

Expected output: stages never begin with a token about to die.

4. If the refresh itself fails (refresh token expired or revoked), stop and report "re-authentication required" instead of retrying forever.

Expected output: a clear failure message naming the fix, not an infinite 401 loop.

When to use

• Pipelines die partway with 401s after running fine
• The API uses short-lived access tokens with a refresh flow
• An agent loop runs longer than the token lifetime

When not to use

• Every call 401s from the start (that is a bad credential, not expiry)
• The API uses long-lived static keys (there is nothing to refresh)
• The 401 is intermittent from the first call (that is usually a bad token, not expiry)

Variant phrasings

token expires between pipeline stages

Same fix. The proactive refresh in step 3 at each stage boundary covers this exactly.

refresh works but the pipeline still dies later

Something caches the old token (a session object, a header built once). Make sure the refresh updates every place the token is stored.

Why it happens

Token lifetimes are designed around interactive sessions, not multi-hour pipelines. The pipeline was written and tested with fresh tokens, so the expiry path never ran until production. A 401 mid-run is normal API behavior; treating it as fatal is the bug.

Edge cases

• Clock skew can make a valid token look expired; sync the clock before adding retry logic.
• Some APIs revoke the old refresh token on use (rotation); the client must store the new refresh token each time.
• Parallel workers sharing one token can stampede the refresh endpoint; use a lock or a single token manager.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_lJ5yEpxNK0OMVT8ggOncFQ