agent's service-account password rotated at close and every reconciliation API call failed with 401
Fixes an agent's reconciliation API calls all failing with 401 after the service account's secret rotated at close: update the stored secret in the approved vault, restart or refresh the agent's client, and add rotation-aware handling. Use when 401s start right after a rotation event. Not for 401s on a single endpoint with a bad scope.
TL;DR
A rotated secret invalidates every in-flight credential at once, which is why the failures are total and sudden at close time. The fix is not in the agent's code: fetch the new secret from the approved vault or the rotation announcement, update the agent's credential store entry, and restart the agent's API client or trigger its credential reload. Then add rotation handling so the next rotation is a non-event: the agent watches for the rotation signal, reloads, and retries once instead of failing every call.
agent's service-account password rotated at close and every reconciliation API call failed with 401Use this when
- All reconciliation API calls return 401 starting at a rotation time, here during close.
- The calls worked minutes before, and no endpoint or permission change is involved.
- The secret store or admin announced a rotation at the matching time.
Not for this skill when
- A single call or endpoint 401s while others succeed. That is a scope or role problem on that call, not a rotated secret.
- 401s began after a code deploy that changed auth handling. Review the deploy diff before blaming rotation.
Steps
- Confirm the rotation: check the vault or secret store for the new version and its activation time against the 401 start time. Verify: the timing correlation is established from logs
- Update the agent's credential configuration to the new secret through the approved store, not by pasting values into code or config files. Verify: the stored credential is the new version
- Restart the agent's API client or invoke its credential reload so in-flight calls use the new secret. Verify: a test API call succeeds with the refreshed credential
- Retry or replay the failed reconciliation calls from their checkpoints rather than from scratch. Verify: the close work resumes without duplicated postings
- Add rotation readiness: the agent subscribes to rotation events or checks credential version at job start, reloads on 401 patterns, and retries once after reload. Verify: a simulated rotation in a test environment is handled without total failure
Variant phrasings
service account secret rotated 401 all API calls agent
The rotation-401 phrasing.
agent credential rotation reload secret vault retry
When adding rotation handling.
Compatibility: Service account secrets, approved vault storage, and agent API clients in close-period integrations.
Why it happens
Secret rotation is a security control doing its job on a schedule the agent was never told about. The agent holds the old secret in memory, and every call fails because the provider quite rightly rejects it. The pain is timing: rotations scheduled during close maximise damage. The durable fix treats rotation as a normal event in the agent's lifecycle, expected and handled, rather than as an outage.
Edge cases / pitfalls
- Rotations during close are a planning failure as much as a technical one; coordinate rotation windows with the close calendar.
- Old secrets often have a grace period. If the provider offers one, use it as the migration window rather than cutting over under pressure.
- Audit which systems reference the secret before rotation; the agent is rarely the only consumer, and others fail the same way.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_KaomdiWTF0CbezstmapO8w