## TL;DR

A 500 mid-rotation leaves unknown state: the key may or may not have rotated. Make every step idempotent, record what completed, and on retry, reconcile actual state instead of blindly repeating.

## Error

```text
agent failed to rotate the key: cloud api returned 500 mid-run
```

## Steps

1. On failure, query the current state: which keys exist and which are active. Expected: ground truth, not assumptions.
2. Design each rotation step to be safe to repeat (create-if-missing, idempotent updates). Expected: retries never double-apply.
3. Record a marker after each completed step. Expected: resume starts after the last good step.
4. Retry with backoff; 500s are often transient. Expected: the retry completes.
5. Verify the final state matches the intended rotation end state. Expected: one active new key, old key deactivated.

## When to use

- Agent rotations failing on transient API errors.
- Building resilient rotation playbooks.

## When not to use

- 4xx errors (fix the request, not the retry).
- The API is down for everyone (wait, then resume).

## Tool compatibility

- Cloud IAM APIs; agent rotation orchestration.

## Variant phrasings

### rotation failed with 500

Reconcile and resume.

### key rotation died halfway

Idempotent steps.

## Why it happens

Distributed APIs fail mid-operation; without idempotency and markers, retries either duplicate or skip work.

## Edge cases

- Two agents rotating concurrently need a lock; idempotency alone is not enough.
- Markers must be stored durably, not in agent memory.
- Verify against the provider, not the markers, before declaring done.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_LBIaU-4zc4YAGLMJc_RxNQ