## TL;DR

The agent rotated secrets one by one; the token died halfway, so some secrets updated and the rest are gone or stale. The worker is now in a mixed state. Inventory which secrets are set, re-rotate ONLY the missing ones with a fresh token, and verify the full set before declaring recovery.

## The query

```text
auth token expired mid-run: secret rotation used a stale token, half the secrets now missing
```

## Use this when

- A secret rotation run failed partway on authentication.
- Some secrets work, others are missing or stale.
- The worker behaves inconsistently across requests.

## Not for

- Planned rotations that completed (no incident).
- A single secret failing to set (isolated problem).
- Secrets missing that were never rotated (config problem).

## Steps

### Step 1: Inventory the current secret state

List which secrets the worker can actually read (via a debug endpoint or by checking behavior), and compare against the expected set.

Expected result: a definitive list of present vs missing secrets. Do not guess from the run log alone; the log lies about what the API accepted.

### Step 2: Get a fresh token and verify it

Re-authenticate and confirm the new session with whoami before any mutating command.

Expected result: a working session. Every subsequent secret operation uses this session only.

### Step 3: Re-set ONLY the missing secrets

Set each missing secret individually, verifying each one. Do not re-run the full rotation blindly; re-setting already-correct secrets risks new inconsistency.

Expected result: the missing secrets exist again. The previously-correct ones are untouched.

### Step 4: Verify the full set end to end

Exercise every code path that reads secrets - auth flows, API calls, webhook signatures.

Expected result: all secret-dependent behavior works. The mixed state is resolved.

### Step 5: Make rotation atomic and resumable

Rewrite the rotation as: snapshot the expected set, rotate with per-secret verification, and record progress so a rerun skips completed secrets.

Expected result: the next rotation either completes fully or resumes cleanly. Half-applied states become impossible.

## Why it happens

Secret rotation is a multi-step mutation with no transaction. The agent assumed the session would survive the whole run, and token expiry turned a routine rotation into a partial one. Partial secret state is uniquely nasty because the worker half-works, which looks like an application bug rather than a config incident.

## Edge cases

- Some 'missing' secrets may actually be present but with old values. Verify values, not just presence, where possible.
- Preview and production secrets are separate. Confirm which environment the rotation targeted before repairing the other.
- Never log secret values during inventory. Log names and presence only.
- If the rotation was for a compromised credential, treat the window between rotation start and recovery as exposure time and rotate again cleanly.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sHAi_LMkE9i99chEbk3Zig