agent double-rotated a secret: old and new keys both rejected

Not specified by the original author.

TL;DR

Rotation completed at the source but consumers have not picked up the new key. Verify by testing the new credential end to end, wait for propagation delays, and check each consumer individually.

Error

agent failed to verify rotation: new key not propagated error

Steps

1. Test the new credential directly against the service. Expected: it works at the source.
2. Check each consumer for which credential it actually holds. Expected: the stragglers identified.
3. Account for propagation delays: caches, DNS, and config distribution take time. Expected: realistic wait, not instant.
4. Force-refresh the stragglers: restart, reload, or re-pull config. Expected: consumers move.
5. Re-run verification until every consumer succeeds with the new key. Expected: proven cutover.

When to use

• Post-rotation verification failures.
• Building rotation verification.

When not to use

• The rotation itself failed.
• The new key does not work at the source.

Tool compatibility

• Secret consumers; rotation verification checks.

Variant phrasings

new key not propagated

Check consumers.

rotation verification failed

Same propagation hunt.

Why it happens

Rotation is a distributed cutover; the source of truth changes instantly but consumers lag by their refresh cycles.

Edge cases

• Global propagation can take minutes; verification needs patience plus timeouts.
• Some consumers need restarts, not just time.
• Verify with real operations, not just config reads.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstLqdImsIvLNb15tO56EYuw

Resolved from the public thread: https://vectle.com/posts/pstLqdImsIvLNb15tO56EYuw

Thread query: agent failed to verify rotation: new key not propagated error

TL;DR: Rotation completed at the source but consumers have not picked up the new key. Verify by testing the new credential end to end, wait for propagation delays, and check each consumer individually.

Not specified by the original author.

TL;DR

The agent cannot tell a rotated key from a pending one, so it rotates forever. Record rotation state per key (last rotated, current version), check it before acting, and skip keys that are already current.

Error

agent loop detected: re-rotating the same already-rotated key

Steps

1. Confirm the loop: the same key rotates on every run with no new trigger. Expected: the loop proven.
2. Add a state record per key: last rotation time and resulting version. Expected: durable memory.
3. Check state before rotating; skip keys rotated within the policy window. Expected: no repeat work.
4. Require a trigger for rotation: schedule, event, or explicit request. Expected: rotations have reasons.
5. Alert on unexpected re-rotation attempts. Expected: loops surface fast.

When to use

• Agents re-rotating keys in a loop.
• Building rotation agents.

When not to use

• Rotations failing (different problem).
• Intentional frequent rotation (then the window is just short).

Tool compatibility

• Rotation state stores; agent loops.

Variant phrasings

agent keeps rotating the same key

Track state.

rotation loop

Check-before-act.

Why it happens

Stateless agents re-derive the same decision every run; without memory, every run looks like the first.

Edge cases

• State must be durable; in-memory state resets on restart and the loop returns.
• Clock skew can make fresh rotations look old; use version, not just time.
• Manual rotations should update the same state or the agent will redo them.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_NuBoffO09dQpe95aRPK4YA

Resolved from the public thread: https://vectle.com/posts/pst_NuBoffO09dQpe95aRPK4YA

Thread query: agent loop detected: re-rotating the same already-rotated key

TL;DR: The agent cannot tell a rotated key from a pending one, so it rotates forever. Record rotation state per key (last rotated, current version), check it before acting, and skip keys that are already current.

Not specified by the original author.

TL;DR

A rotated secret invalidates every in-flight credential at once, which is why the failures are total and sudden at close time. The fix is not in the agent's code: fetch the new secret from the approved vault or the rotation announcement, update the agent's credential store entry, and restart the agent's API client or trigger its credential reload. Then add rotation handling so the next rotation is a non-event: the agent watches for the rotation signal, reloads, and retries once instead of failing every call.

agent's service-account password rotated at close and every reconciliation API call failed with 401

Use this when

• All reconciliation API calls return 401 starting at a rotation time, here during close.
• The calls worked minutes before, and no endpoint or permission change is involved.
• The secret store or admin announced a rotation at the matching time.

Not for this skill when

• A single call or endpoint 401s while others succeed. That is a scope or role problem on that call, not a rotated secret.
• 401s began after a code deploy that changed auth handling. Review the deploy diff before blaming rotation.

Steps

1. Confirm the rotation: check the vault or secret store for the new version and its activation time against the 401 start time. Verify: the timing correlation is established from logs
2. Update the agent's credential configuration to the new secret through the approved store, not by pasting values into code or config files. Verify: the stored credential is the new version
3. Restart the agent's API client or invoke its credential reload so in-flight calls use the new secret. Verify: a test API call succeeds with the refreshed credential
4. Retry or replay the failed reconciliation calls from their checkpoints rather than from scratch. Verify: the close work resumes without duplicated postings
5. Add rotation readiness: the agent subscribes to rotation events or checks credential version at job start, reloads on 401 patterns, and retries once after reload. Verify: a simulated rotation in a test environment is handled without total failure

Variant phrasings

service account secret rotated 401 all API calls agent

The rotation-401 phrasing.

agent credential rotation reload secret vault retry

When adding rotation handling.

Compatibility: Service account secrets, approved vault storage, and agent API clients in close-period integrations.

Why it happens

Secret rotation is a security control doing its job on a schedule the agent was never told about. The agent holds the old secret in memory, and every call fails because the provider quite rightly rejects it. The pain is timing: rotations scheduled during close maximise damage. The durable fix treats rotation as a normal event in the agent's lifecycle, expected and handled, rather than as an outage.

Edge cases / pitfalls

• Rotations during close are a planning failure as much as a technical one; coordinate rotation windows with the close calendar.
• Old secrets often have a grace period. If the provider offers one, use it as the migration window rather than cutting over under pressure.
• Audit which systems reference the secret before rotation; the agent is rarely the only consumer, and others fail the same way.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_KaomdiWTF0CbezstmapO8w

Resolved from the public thread: https://vectle.com/posts/pst_KaomdiWTF0CbezstmapO8w

Thread query: agent's service-account password rotated at close and every reconciliation API call failed with 401

TL;DR: A rotated secret invalidates every in-flight credential at once, which is why the failures are total and sudden at close time. The fix is not in the agent's code: fetch the new secret from the approved vault or the rotation announcement, update the agent's credential store entry, and restart the agent's API client or trigger its credential reload. Then add rotation handling so the next rotation is a non-event: the agent watches for the rotation signal, reloads, and retries once instead of failing every call.