## TL;DR
Rotate between runs, never under a running job. Before a backup or restore starts, pin the credential version the run will use and record it in the manifest, refuse to rotate while any run holding the old version is active, and keep the old credential valid through an overlap window until those runs finish. The rotation job must check the active run registry first, stage the new credential, let new runs pick it up, and revoke the old one only when the registry shows no run still pinned to it.

```text
agent's credential rotation invalidated the backup service account mid-run
```

## Use this when
- Backups or restores start failing with authentication or authorisation errors immediately after a scheduled credential rotation.
- A long restore begins on one credential and loses access partway through.
- Rotation succeeds in the vault while the backup service still holds, or rejects, a different version.

## Not for this skill when
- The service account never had permission for a repository or vault it is asked to use.
- A credential was revoked for a security incident and must stay revoked regardless of running jobs.

## Steps
1. Pin the version at run start. When a run begins, resolve the credential, record its version identifier and issue time in the run manifest without recording any credential value, and use that pinned reference for the whole run. If the platform returns a new version mid-run, the running job ignores it. Verify: a run manifest shows one version from start to finish even when a rotation lands during the run
2. Build a run registry the rotation can consult. Register every active backup and restore run with its pinned version and expected end time, and deregister on completion or confirmed failure. The registry is the single source the rotation job checks, so it must be updated by the harness, not inferred from logs. Verify: starting a test run makes it appear in the registry with its pinned version
3. Stage the new credential without revoking the old. Rotation creates or activates the new version for new runs, leaves the old version valid, and marks it draining. New runs pin the new version; in-flight runs keep the old one. Verify: after staging, a new short backup succeeds on the new version while a running restore on the old version continues
4. Gate revocation on an empty registry. Revoke or disable the old version only when no active run is pinned to it and a grace period past the longest expected run has passed. If runs are still pinned, the rotation job waits, reports draining status, and retries later rather than forcing completion. Verify: attempting rotation during a long restore leaves the old version valid and the rotation in draining state
5. Align the backup service copy. Where the backup product stores its own copy of the account credential, update that copy in the same staged sequence and run a short connectivity check from the product itself, not only from the agent, because a vault-side success can hide a stale product-side copy. Verify: the product connection test passes on the new version before the old one is revoked
6. Handle the failure case explicitly. If a run does fail after a forced or accidental revocation, classify it as a rotation collision, re-run it on a valid version, and record the incident against the rotation event so the pattern is visible. Do not report such runs as backup product failures. Verify: the incident record links the failed run id to the rotation event id

## Why it happens
Rotation systems and backup systems run on independent schedules, and both are usually automated, so a collision is a matter of time. The backup run holds a credential resolved at start, the rotator replaces or revokes it on its own clock, and the next call in a long job fails. Because the error surfaces inside the backup product, teams debug repositories and permissions while the actual cause sits in the rotation log. Pinning plus an overlap window removes the race without weakening rotation.

## Variants and edge cases
Edge cases: some services allow only one active key per account, which makes a true overlap impossible; there the answer is a maintenance style rotation inside a gap between runs, with the run registry used to find a real gap, plus a fast rollback to the previous version. Emergency revocation for a suspected leak overrides all of this, and affected runs should be failed deliberately and re-run. Clock based expiry can mimic a rotation collision, so check version change events before concluding.

### backups broke right after key rotation
Use when the timing lines up with a rotation event. Check the run registry and version pins before touching permissions.

### restore lost access halfway through
Use when a long job fails on authorisation mid-run. Pin the version at start and drain the old one.

### rotation succeeded but the backup product still fails
Use when the product keeps its own credential copy. Update and test that copy in the staged sequence.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_-YODe_iUP5XndgJ-DZAh1Q