cmdb auth token expired mid-run and agent failed to refresh

TL;DR

Long runs outlive short-lived credentials. If the CMDB issues tokens or sessions with a lifetime shorter than a wave, the agent must implement the refresh flow (or re-authenticate) proactively: track expiry, refresh at a threshold (for example 80 percent of lifetime), and retry the failed call once after a successful refresh. Where the platform supports it, use a service account with a refresh token or a certificate instead of an interactive login session.

cmdb auth token expired mid-run and agent failed to refresh

Use this when

• The run fails with CMDB 401 errors at roughly the same number of minutes after start every time.
• Short test runs succeed; only full waves die.
• The credential was obtained by an interactive login at run start.

Not for this skill when

• The credential was revoked or the account disabled. Refresh cannot help; reissue through the owner.
• Auth fails from the first call. That is a credentials or permissions problem, not expiry.

Steps

1. Measure the credential lifetime: note issue and expiry from the auth response or platform docs, and correlate with the failure time in logs. Verify: lifetime and failure time match, confirming expiry as the cause
2. Implement proactive refresh: before each CMDB call (or on a timer), check remaining lifetime and refresh when under the threshold. Verify: a forced short-lifetime test shows refresh occurring before expiry
3. Add reactive recovery as the second layer: on a 401 that indicates expiry, refresh once and retry the call once. Verify: an artificially expired credential mid-run recovers without operator help
4. Move the agent to a non-interactive credential type the platform supports for service accounts (refresh token flow, client certificate, or API key in the vault) so refresh does not depend on a human session. Verify: the run starts and refreshes with no interactive login present
5. Log refresh events and alert on refresh failures as their own signal. Verify: the wave report shows refreshes succeeded, and a failed refresh would page before the wave dies

Variant phrasings

cmdb token expired mid run refresh before expiry agent

The fix phrasing. Note the wording avoids presenting the credential value; only its lifecycle is handled.

401 at same elapsed time long run service account session

The diagnostic phrasing.

proactive token refresh 80 percent lifetime pattern

The pattern phrasing.

Compatibility: Compatibility: OAuth-style token APIs, session-based CMDB APIs (for example ServiceNow), and certificate auth; refresh mechanics are platform-specific.

Why it happens

Credential lifetimes are set for human session patterns, and a wave is not a human session: it runs for hours with bursts of API calls. Any agent that authenticates once at startup has a built-in self-destruct timer. The refresh logic is standard in every mature API client, but hand-rolled agent integrations routinely omit it because short tests never reach the expiry, so the defect ships and detonates on the first full run.

Edge cases / pitfalls

• Store refresh material in the vault with the same care as the primary credential; a refresh token is a credential.
• Clock skew between the agent host and the platform makes expiry math wrong; sync time before debugging refresh logic.
• Some platforms invalidate the old token at refresh; concurrent workers sharing one credential can then invalidate each other, which argues for one credential per worker or a shared refreshing client.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstQ42uqh6GNoQLrufDvY6WQ