agent's mTLS client certificate expired mid-pipeline run

agent's mTLS client certificate expired mid-pipeline run

TL;DR

The pipeline's client cert expired and the mTLS handshake started failing. Renew it now, then automate renewal and add expiry alerting so nobody discovers this from a failed deploy again.

agent's mTLS client certificate expired mid-pipeline run

Use this when

• A pipeline that worked yesterday fails today on the TLS handshake
• The error mentions certificate expiry on the client side
• The failure lands mid-run, at the first mTLS connection
• Static cert files are mounted into runners or pods

Not for this skill when

• The server certificate expired (same commands, different file)
• The chain is untrusted (that's a CA problem, not expiry)
• mTLS never worked (that's configuration, not renewal)

Steps

1. Confirm expiry on the cert the pipeline uses:

openssl x509 -in /path/to/client.crt -noout -dates

Expected: the notAfter date is in the past. That date is the whole story.

2. Renew the certificate from your CA or cert-manager, keeping the same subject and key usage the server expects.

Expected: a new client.crt with a future notAfter.

3. Install it where the pipeline reads it and re-run the failing step.

Expected: the mTLS handshake succeeds; the pipeline goes green.

4. Automate renewal: short-lived certs (hours to days) issued at pipeline start, or cert-manager with automatic rotation for long-lived ones. The goal is that no static cert file ages out silently.

Expected: a documented renewal path, no manual calendar reminder.

5. Add expiry monitoring: alert at 30 days and 7 days before notAfter for every client cert in the fleet:

openssl x509 -in /path/to/client.crt -noout -checkend 2592000

Expected: exit code 0 means more than 30 days left; non-zero means alert. Wire that into your monitoring.

Variant: cert expired in a Kubernetes pod

Use cert-manager with a Certificate resource instead of a mounted static file. Static files in pods are expiry incidents waiting to happen.

Variant: SPIFFE or SPIRE workloads

Workload attestation gives you fresh SVIDs automatically. Use that instead of managing cert files at all.

Variant: certificate verify failed vs expired

Verify-failed is a trust problem, expired is a time problem. Check the dates first; it takes ten seconds and settles it.

Why this happens

mTLS certs expire like any other cert, but client certs get less attention than server certs: no browser warning, no user complaints, just a pipeline that worked yesterday and fails today. The failure lands mid-run because that's when the handshake happens.

Edge cases and pitfalls

• Renewing the cert isn't enough if the pipeline caches the old one in a secret or image; bust the cache.
• If the CA that signed it also expired, you're renewing the wrong thing; check the chain.
• Keep the private key permissions tight (0600) when you write the new files.
• After renewal, verify the server actually accepts the new cert before closing the incident; a wrong key usage fails differently.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst31-yc7HYvY8J1wzzIlJUw