kerberos ticket expired mid-run and agent's patch window failed

TL;DR

Kerberos tickets default to about 10 hours, and a long patch window plus pre-work can exceed that; when the ticket-granting ticket expires, every new GSSAPI authentication fails while already-open sessions keep working, which produces a confusing partial failure. Run the agent under a keytab-driven renewal (kinit with a keytab, or a systemd service that renews), or use kstart or a credential cache renewal loop, and check ticket validity in preflight with klist.

kerberos ticket expired mid-run and agent's patch window failed

Use this when

• Linux patch agents authenticating over SSH with GSSAPI fail partway with Permission denied or ticket expired messages.
• klist on the agent host shows an expired ticket after the failure.
• Short waves succeed; weekend-scale waves hit the cliff.

Not for this skill when

• The host clocks are skewed beyond Kerberos tolerance (5 minutes default). That produces immediate, not mid-run, failures; fix NTP.
• The service principal or keytab is wrong. That fails from the first host.

Steps

1. Confirm the diagnosis: check the ticket cache times with klist against the failure timestamps in the wave log. Verify: expiry and failure onset align
2. Set up keytab-based acquisition: store the service account keytab with strict permissions in the vault-managed location and have the agent kinit from it at start. Verify: a fresh ticket is obtained non-interactively in a test
3. Add renewal: a background renewer (kstart, or a systemd timer running kinit from the keytab) that keeps a valid ticket for the run duration, plus a preflight klist check before each wave segment. Verify: during a long test, ticket validity never lapses
4. Alternatively delegate appropriately: where the design uses SSH GSSAPI delegation or a Windows service running as the account, confirm the platform renews tickets for the process. Verify: the chosen mechanism is documented and tested past one ticket lifetime
5. Add ticket-expiry monitoring with an alert window before expiry. Verify: an expiring ticket with a broken renewer alerts a human in time

Variant phrasings

kerberos ticket expired mid run kinit keytab renewal agent

The fix phrasing.

gssapi ssh authentication fails hours into patch window

The symptom phrasing.

klist ticket lifetime service account long running job

The diagnostic phrasing.

Compatibility: Compatibility: MIT and Heimdal Kerberos, Active Directory domains, SSSD-joined Linux hosts; Windows services use the machine account differently and rarely hit this exact pattern.

Why it happens

Kerberos is explicit about lifetimes: a ticket is a lease, and leases end. Interactive users renew by logging in again; a service needs a keytab and a renewal mechanism, which is the standard pattern for every long-running Kerberos client (batch systems have done this for decades). The partial-failure signature, existing sessions fine and new authentications dead, is the tell that separates expiry from every other auth fault.

Edge cases / pitfalls

• Keytabs are credentials: file permissions 600, owned by the service account, rotated on the account schedule, and never in the agent prompt or logs.
• Renewable ticket lifetimes have a hard cap set by domain policy (often 7 days); runs longer than that need full re-acquisition from the keytab, which the renewer should handle.
• Clock sync is a Kerberos prerequisite; an agent host with drifting time will fail in ways that look like expiry but are skew.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_lKfE17J74XSHcwTDZT36fQ