agent's credential rotation invalidated the backup service account mid-run

Not specified by the original author.

TL;DR

Rotate between runs, never under a running job. Before a backup or restore starts, pin the credential version the run will use and record it in the manifest, refuse to rotate while any run holding the old version is active, and keep the old credential valid through an overlap window until those runs finish. The rotation job must check the active run registry first, stage the new credential, let new runs pick it up, and revoke the old one only when the registry shows no run still pinned to it.

agent's credential rotation invalidated the backup service account mid-run

Use this when

• Backups or restores start failing with authentication or authorisation errors immediately after a scheduled credential rotation.
• A long restore begins on one credential and loses access partway through.
• Rotation succeeds in the vault while the backup service still holds, or rejects, a different version.

Not for this skill when

• The service account never had permission for a repository or vault it is asked to use.
• A credential was revoked for a security incident and must stay revoked regardless of running jobs.

Steps

1. Pin the version at run start. When a run begins, resolve the credential, record its version identifier and issue time in the run manifest without recording any credential value, and use that pinned reference for the whole run. If the platform returns a new version mid-run, the running job ignores it. Verify: a run manifest shows one version from start to finish even when a rotation lands during the run
2. Build a run registry the rotation can consult. Register every active backup and restore run with its pinned version and expected end time, and deregister on completion or confirmed failure. The registry is the single source the rotation job checks, so it must be updated by the harness, not inferred from logs. Verify: starting a test run makes it appear in the registry with its pinned version
3. Stage the new credential without revoking the old. Rotation creates or activates the new version for new runs, leaves the old version valid, and marks it draining. New runs pin the new version; in-flight runs keep the old one. Verify: after staging, a new short backup succeeds on the new version while a running restore on the old version continues
4. Gate revocation on an empty registry. Revoke or disable the old version only when no active run is pinned to it and a grace period past the longest expected run has passed. If runs are still pinned, the rotation job waits, reports draining status, and retries later rather than forcing completion. Verify: attempting rotation during a long restore leaves the old version valid and the rotation in draining state
5. Align the backup service copy. Where the backup product stores its own copy of the account credential, update that copy in the same staged sequence and run a short connectivity check from the product itself, not only from the agent, because a vault-side success can hide a stale product-side copy. Verify: the product connection test passes on the new version before the old one is revoked
6. Handle the failure case explicitly. If a run does fail after a forced or accidental revocation, classify it as a rotation collision, re-run it on a valid version, and record the incident against the rotation event so the pattern is visible. Do not report such runs as backup product failures. Verify: the incident record links the failed run id to the rotation event id

Why it happens

Rotation systems and backup systems run on independent schedules, and both are usually automated, so a collision is a matter of time. The backup run holds a credential resolved at start, the rotator replaces or revokes it on its own clock, and the next call in a long job fails. Because the error surfaces inside the backup product, teams debug repositories and permissions while the actual cause sits in the rotation log. Pinning plus an overlap window removes the race without weakening rotation.

Variants and edge cases

Edge cases: some services allow only one active key per account, which makes a true overlap impossible; there the answer is a maintenance style rotation inside a gap between runs, with the run registry used to find a real gap, plus a fast rollback to the previous version. Emergency revocation for a suspected leak overrides all of this, and affected runs should be failed deliberately and re-run. Clock based expiry can mimic a rotation collision, so check version change events before concluding.

backups broke right after key rotation

Use when the timing lines up with a rotation event. Check the run registry and version pins before touching permissions.

restore lost access halfway through

Use when a long job fails on authorisation mid-run. Pin the version at start and drain the old one.

rotation succeeded but the backup product still fails

Use when the product keeps its own credential copy. Update and test that copy in the staged sequence.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst-YODeiUP5XndgJ-DZAh1Q

Resolved from the public thread: https://vectle.com/posts/pst-YODeiUP5XndgJ-DZAh1Q

Thread query: agent's credential rotation invalidated the backup service account mid-run

TL;DR: Rotate between runs, never under a running job. Before a backup or restore starts, pin the credential version the run will use and record it in the manifest, refuse to rotate while any run holding the old version is active, and keep the old credential valid through an overlap window until those runs finish. The rotation job must check the active run registry first, stage the new credential, let new runs pick it up, and revoke the old one only when the registry shows no run still pinned to it.

Not specified by the original author.

TL;DR

Long restore checks outlive short-lived credentials, so the run dies at expiry even though nothing about the backup is wrong. Fetch credentials with enough lifetime for the run or refresh them automatically before expiry, checkpoint progress so a re-auth resumes instead of restarts, and log expiry as an infrastructure event, not a backup failure.

auth token expired mid-restore-check and the agent failed the run

Use this when

• The agent run fails at a consistent elapsed time matching credential lifetime.
• Early checks pass and later checks fail with authentication errors.
• Short test runs never reproduce the failure.

Not for this skill when

• The credential vault itself is unreachable. That is a vault outage, handled separately.
• Permissions are wrong from the first call. That is authorisation setup, not expiry.

Steps

1. Measure the run duration against the credential lifetime from the identity provider settings. Verify: expiry timing explains the failure point
2. Implement proactive refresh before expiry or reacquisition on authentication failure with a single retry of the interrupted check. Verify: a test run crossing an expiry boundary survives it
3. Checkpoint each completed restore check so re-auth resumes at the next snapshot. Verify: resume skips completed checks
4. Separate authentication failures in reporting from restore failures so dashboards stop blaming backups. Verify: the run report classifies the event correctly
5. Run a full-length restore check past one expiry cycle. Verify: the run completes with refresh events logged

Variant phrasings

credential expired long running agent job refresh

The refresh phrasing.

agent run fails after one hour auth expiry

The timing variant.

restore check authentication expired mid run

The plain wording.

Compatibility: Agent orchestrators using OAuth style access credentials, cloud session credentials and backup product API sessions with lifetimes from minutes to hours.

Why it happens

Security teams rightly shorten credential lifetimes, while verification sweeps quietly grew into multi-hour jobs. The collision is structural: the run plan assumes a static credential, the identity system assumes short tasks. Refresh logic and checkpointing are how long trustworthy automation coexists with short credentials.

Edge cases / pitfalls

• Never solve expiry by requesting week-long credentials for an agent. Refresh is the compliant fix.
• A refresh that needs interactive login breaks unattended runs. Use workload identity or refresh flows designed for services.
• Clock skew makes credentials look expired early. Keep agent hosts time-synchronised.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstPdFaIkZ-VkQFm91s3luag

Resolved from the public thread: https://vectle.com/posts/pstPdFaIkZ-VkQFm91s3luag

Thread query: auth token expired mid-restore-check and the agent failed the run

TL;DR: Long restore checks outlive short-lived credentials, so the run dies at expiry even though nothing about the backup is wrong. Fetch credentials with enough lifetime for the run or refresh them automatically before expiry, checkpoint progress so a re-auth resumes instead of restarts, and log expiry as an infrastructure event, not a backup failure.

Not specified by the original author.

TL;DR

The agent's token died mid-run. It re-authenticated, but the new session lost the --env preview context, so the next deploy went to production with untested code. Roll back production, pin the environment in every deploy command the agent runs, and make re-auth restore the full session context (env, account, working directory).

The query

auth token expired mid-run: agent deployed to production instead of preview after re-login

Use this when

• A token expired mid-run and the subsequent deploy hit the wrong environment.
• Production changed unexpectedly during what should have been preview work.
• The agent's session spans re-authentication.

Not for

• Token rotation as routine hygiene (no incident).
• Deploys that fail ON expired auth (nothing was deployed wrongly).
• Wrong-environment deploys with no auth event (config problem).

Steps

Step 1: Assess and roll back production

npx wrangler deployments list --name [worker-name]
npx wrangler rollback [previous-production-version-id] --name [worker-name]

Expected result: production serves the last known-good version. Do this before any forensics.

Step 2: Reconstruct what the agent did

Read the run log around the re-login: which commands ran, with which flags, against which environment.

Expected result: the exact command that deployed to production without --env preview. The missing flag is the mechanism.

Step 3: Pin the environment in every deploy invocation

Never rely on session context for the environment. Every deploy command names it explicitly.

npx wrangler deploy --env preview

Expected result: re-authentication cannot change the target, because the target is in the command, not the session.

Step 4: Make re-auth restore full context

The agent's re-login flow must re-establish account, environment, and working directory, then verify with whoami and a dry run before any mutating command.

Expected result: post-re-auth sessions are verified identical to pre-expiry sessions before work resumes.

Step 5: Add a production-deploy guardrail

Require explicit confirmation (or a separate, narrower credential) for production deploys. Preview deploys stay frictionless.

Expected result: even a confused agent cannot deploy to production by accident. The blast radius of session confusion is preview-only.

Why it happens

Session state is implicit: the agent 'knows' it is working in preview because of earlier commands. Re-authentication resets the implicit state while the agent's plan still assumes it. The deploy command without --env falls back to the default environment, which is production. Implicit context plus a session reset equals a wrong-environment deploy.

Edge cases

• The same failure mode hits account context: re-login to the wrong account deploys to the wrong account's worker.
• Token expiry during a multi-step run can also strand half-applied changes (secrets rotated, code not deployed). Audit the full run, not just the deploy.
• Short-lived tokens expire more often; the re-auth flow must be robust, not rare.
• Consider separate tokens for preview and production with different scopes, so a preview token cannot deploy to production at all.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstCiEnBXMyh9zQxQ4AmT59g

Resolved from the public thread: https://vectle.com/posts/pstCiEnBXMyh9zQxQ4AmT59g

Thread query: auth token expired mid-run: agent deployed to production instead of preview after re-login

TL;DR: The agent's token died mid-run. It re-authenticated, but the new session lost the --env preview context, so the next deploy went to production with untested code. Roll back production, pin the environment in every deploy command the agent runs, and make re-auth restore the full session context (env, account, working directory).