auth token expired mid-run: secret rotation used a stale token, half the secrets now missing

TL;DR

The agent rotated secrets one by one; the token died halfway, so some secrets updated and the rest are gone or stale. The worker is now in a mixed state. Inventory which secrets are set, re-rotate ONLY the missing ones with a fresh token, and verify the full set before declaring recovery.

The query

auth token expired mid-run: secret rotation used a stale token, half the secrets now missing

Use this when

• A secret rotation run failed partway on authentication.
• Some secrets work, others are missing or stale.
• The worker behaves inconsistently across requests.

Not for

• Planned rotations that completed (no incident).
• A single secret failing to set (isolated problem).
• Secrets missing that were never rotated (config problem).

Steps

Step 1: Inventory the current secret state

List which secrets the worker can actually read (via a debug endpoint or by checking behavior), and compare against the expected set.

Expected result: a definitive list of present vs missing secrets. Do not guess from the run log alone; the log lies about what the API accepted.

Step 2: Get a fresh token and verify it

Re-authenticate and confirm the new session with whoami before any mutating command.

Expected result: a working session. Every subsequent secret operation uses this session only.

Step 3: Re-set ONLY the missing secrets

Set each missing secret individually, verifying each one. Do not re-run the full rotation blindly; re-setting already-correct secrets risks new inconsistency.

Expected result: the missing secrets exist again. The previously-correct ones are untouched.

Step 4: Verify the full set end to end

Exercise every code path that reads secrets - auth flows, API calls, webhook signatures.

Expected result: all secret-dependent behavior works. The mixed state is resolved.

Step 5: Make rotation atomic and resumable

Rewrite the rotation as: snapshot the expected set, rotate with per-secret verification, and record progress so a rerun skips completed secrets.

Expected result: the next rotation either completes fully or resumes cleanly. Half-applied states become impossible.

Why it happens

Secret rotation is a multi-step mutation with no transaction. The agent assumed the session would survive the whole run, and token expiry turned a routine rotation into a partial one. Partial secret state is uniquely nasty because the worker half-works, which looks like an application bug rather than a config incident.

Edge cases

• Some 'missing' secrets may actually be present but with old values. Verify values, not just presence, where possible.
• Preview and production secrets are separate. Confirm which environment the rotation targeted before repairing the other.
• Never log secret values during inventory. Log names and presence only.
• If the rotation was for a compromised credential, treat the window between rotation start and recovery as exposure time and rotate again cleanly.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstsHAiLMkE9i99chEbk3Zig