vault token expired mid-run while agent failed to pull credentials

TL;DR

Vault leases are deliberately short: dynamic secrets and tokens expire by design, and a wave that fetches its vault session once at start has a built-in expiry. The agent needs a renewal loop for its own vault session (renew before expiry where the auth method allows), per-target secret fetches close to use time rather than bulk fetching at start, and retry-with-renewal on lease errors. Store nothing longer than needed: fetched host credentials should live in memory only, per host, and be dropped after that host completes.

vault token expired mid-run while agent failed to pull credentials

Use this when

• Vault calls begin failing mid-run with lease expired or permission denied on the agent own session.
• Secrets fetched at run start work for early hosts and fail for later ones.
• The failure timing matches the vault auth method TTL in the vault config.

Not for this skill when

• The vault is sealed or down. That is a platform incident; the agent should pause, not work around it.
• The agent role lacks permission for a specific secret path from the start. That is a policy authoring issue, visible immediately.

Steps

1. Map the timing: vault session TTL and dynamic secret lease times versus the run failure points. Verify: the expiring layer (session versus per-secret lease) is identified
2. Implement session renewal in the vault client with a background renewer and a pre-call validity check. Verify: a test with a short TTL shows renewals keeping the session alive across the run
3. Change secret fetching to just-in-time per target: fetch a host credential when that host turn starts, use it, and drop it. Verify: no host credential is fetched hours before its use in the run log
4. Add lease-error recovery: on a lease-expired response, renew or re-authenticate once and retry the fetch once. Verify: an artificially expired lease mid-run recovers transparently
5. Audit the run artifacts for secret hygiene after the change. Verify: state files, logs, and reports contain no credential values, only identities and timestamps

Variant phrasings

vault lease expired mid run agent renew session secrets

The main phrasing.

dynamic secrets just in time per host fetch pattern

The design phrasing.

vault token ttl shorter than patch wave renewal loop

The timing phrasing. As with all credential topics here, only the lifecycle is handled, never the secret values.

Compatibility: Compatibility: HashiCorp Vault and similar secret managers with lease-based dynamic secrets; auth methods include AppRole and cloud IAM.

Why it happens

Vault is designed so that stolen material expires quickly, which means correct clients are renewal clients. Agents that treat vault like a config file they read once collide with the core design. Just-in-time fetching has a second benefit beyond expiry: it shrinks the window in which any given host credential exists in agent memory, so the pattern is both a reliability and a security improvement.

Edge cases / pitfalls

• Renewal cannot extend past the auth method max TTL; runs longer than that need full re-authentication, which the client should schedule, not discover.
• If the agent caches secrets to disk to survive restarts, that cache needs encryption and the same TTL discipline, or it becomes the weakest copy of the vault.
• Clock skew between agent and vault servers distorts lease math; keep the agent host time-synced.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_JqLBMqooq0jAmoiBoZyJnQ