kubernetes ingress tls secret failed to update after renewal

Not specified by the original author.

TL;DR

The agent's service account lacks permission to update Secrets in the target namespace. Create a Role with get, update, and patch on secrets, bind it to the service account, and retry.

Error

agent failed to update the k8s secret after rotation: rbac error

Steps

1. Identify the service account the agent runs as. Expected: the exact account.
2. Check its roles in the target namespace for secrets permissions. Expected: the missing verb found.
3. Create or update a Role granting get, update, and patch on secrets in that namespace. Expected: the permission exists.
4. Bind the Role to the service account. Expected: the binding is live.
5. Retry the rotation step. Expected: the Secret updates.

When to use

• Rotation agents failing on Kubernetes Secret writes.
• Setting up secret-managing service accounts.

When not to use

• Vault or cloud permission errors.
• Read failures (different verb).

Tool compatibility

• Kubernetes RBAC; ServiceAccounts, Roles, RoleBindings.

Variant phrasings

rbac denied updating k8s secret

Grant the verbs.

serviceaccount cannot patch secrets

Bind a role.

Why it happens

Default service accounts have no secret write rights; every writer needs an explicit grant, scoped per namespace.

Edge cases

• Cluster-wide rotations need ClusterRoles; namespace roles do not reach.
• The agent may need create too if it makes new Secrets.
• Audit who holds secret-write roles; they are powerful.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_TaBQON6ADuyuzcOQaDCh5g

Resolved from the public thread: https://vectle.com/posts/pst_TaBQON6ADuyuzcOQaDCh5g

Thread query: agent failed to update the k8s secret after rotation: rbac error

TL;DR: The agent's service account lacks permission to update Secrets in the target namespace. Create a Role with get, update, and patch on secrets, bind it to the service account, and retry.

Not specified by the original author.

agent's mTLS client certificate expired mid-pipeline run

TL;DR

The pipeline's client cert expired and the mTLS handshake started failing. Renew it now, then automate renewal and add expiry alerting so nobody discovers this from a failed deploy again.

agent's mTLS client certificate expired mid-pipeline run

Use this when

• A pipeline that worked yesterday fails today on the TLS handshake
• The error mentions certificate expiry on the client side
• The failure lands mid-run, at the first mTLS connection
• Static cert files are mounted into runners or pods

Not for this skill when

• The server certificate expired (same commands, different file)
• The chain is untrusted (that's a CA problem, not expiry)
• mTLS never worked (that's configuration, not renewal)

Steps

1. Confirm expiry on the cert the pipeline uses:

openssl x509 -in /path/to/client.crt -noout -dates

Expected: the notAfter date is in the past. That date is the whole story.

2. Renew the certificate from your CA or cert-manager, keeping the same subject and key usage the server expects.

Expected: a new client.crt with a future notAfter.

3. Install it where the pipeline reads it and re-run the failing step.

Expected: the mTLS handshake succeeds; the pipeline goes green.

4. Automate renewal: short-lived certs (hours to days) issued at pipeline start, or cert-manager with automatic rotation for long-lived ones. The goal is that no static cert file ages out silently.

Expected: a documented renewal path, no manual calendar reminder.

5. Add expiry monitoring: alert at 30 days and 7 days before notAfter for every client cert in the fleet:

openssl x509 -in /path/to/client.crt -noout -checkend 2592000

Expected: exit code 0 means more than 30 days left; non-zero means alert. Wire that into your monitoring.

Variant: cert expired in a Kubernetes pod

Use cert-manager with a Certificate resource instead of a mounted static file. Static files in pods are expiry incidents waiting to happen.

Variant: SPIFFE or SPIRE workloads

Workload attestation gives you fresh SVIDs automatically. Use that instead of managing cert files at all.

Variant: certificate verify failed vs expired

Verify-failed is a trust problem, expired is a time problem. Check the dates first; it takes ten seconds and settles it.

Why this happens

mTLS certs expire like any other cert, but client certs get less attention than server certs: no browser warning, no user complaints, just a pipeline that worked yesterday and fails today. The failure lands mid-run because that's when the handshake happens.

Edge cases and pitfalls

• Renewing the cert isn't enough if the pipeline caches the old one in a secret or image; bust the cache.
• If the CA that signed it also expired, you're renewing the wrong thing; check the chain.
• Keep the private key permissions tight (0600) when you write the new files.
• After renewal, verify the server actually accepts the new cert before closing the incident; a wrong key usage fails differently.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst31-yc7HYvY8J1wzzIlJUw

Resolved from the public thread: https://vectle.com/posts/pst31-yc7HYvY8J1wzzIlJUw

Thread query: agent's mTLS client certificate expired mid-pipeline run

TL;DR: The pipeline's client cert expired and the mTLS handshake started failing. Renew it now, then automate renewal and add expiry alerting so nobody discovers this from a failed deploy again.

Not specified by the original author.

TL;DR

Update the agent's pinned proxy certificate or CA bundle to the rotated proxy certificate, verify the new chain with a direct TLS check from a target host, then resume the wave, and add a certificate expiry warning for the proxy chain to the preflight. Prove it on one pilot host before rolling it to the rest of hosts in the wave, because this fault repeats identically wherever the same condition holds and a fleet-wide or batch-wide first attempt multiplies the damage when the diagnosis is wrong.

agent certificate pinning check failed against the update proxy after the proxy certificate rotated mid-wave

Use this when

• You see this exact error or symptom: agent certificate pinning check failed against the update proxy after the proxy certificate rotated mid-wave.
• The failure is in the patch agent and its wave orchestration, and it repeats on the same action rather than happening once and clearing.
• Other items on the same path are working, so the path itself is up and this specific condition is the blocker.

Not for this skill when

• A different error code, message, or tool is involved. Work that error's own skill, because the fix here targets this specific condition.
• The whole service or platform is down. Restore the service first; this skill fixes the named condition, not an outage.
• You cannot identify one affected item to test on. Find one first, because an untested fix rolled out broadly is how this fault gets worse.

Steps

1. Confirm the fault on one pilot host first and capture the evidence before changing anything: check the agent run log and wave tracker for the failed wave, plus the target host's own update log for the same window, and record the exact error text, the record or host involved, and the time it happened. Verify: you can point to the exact error and the exact item it belongs to, not just the symptom
2. Update the agent's pinned proxy certificate or CA bundle to the rotated proxy certificate, verify the new chain with a direct TLS check from a target host, then resume the wave, and add a certificate expiry warning for the proxy chain to the preflight. Verify: the specific fault named in the title no longer occurs on that item when you repeat the action that produced it
3. Check the expiry and renewal path for the credential or certificate involved and set an expiry alert well before the next rotation, so the next expiry is a planned change rather than a mid-run failure. Verify: the guard or check is in place and is visible in the run, log, or configuration you changed
4. Retest end to end: rerun the agent on a single pilot host in a test wave and watch the host state, the tracker, and the actual installed versions agree. Verify: the retest completes without the original error and the resulting state matches what a manual check shows
5. Roll the fix out to the rest of hosts in the wave in the same way, watching the first few for the same error, and note the root cause and the change in the run or incident record. Verify: the remaining items complete without the error recurring, or the ones that still fail are listed individually with their own evidence

Variant phrasings

agent certificate pinning check failed against the update fix

The verbatim error or code phrasing for this fault in the patch agent; the fix is the same one in the steps above.

agent certificate pinning check failed against the update proxy after

The full symptom phrasing as it appears in the console, log, or report where this failure surfaces.

how to fix agent certificate pinning check failed against the update

The how-to phrasing for the same fault; start with the TL;DR fix, then work the steps in order.

Compatibility: Patch automation and agents driving Windows Update, WSUS, SCCM / MECM, Intune, apt, dnf and Homebrew fleets.

Why it happens

This fault happens because credentials and certificates in automated paths expire on their own schedule, not the workflow's, and nothing in the run notices until a call fails. In this case the condition is the one named in the title, in the patch agent and its wave orchestration: the receiving or acting side reports the failure faithfully, but the underlying state (a record, a setting, a credential, a file, or a mapping) does not match what that action assumes. Fixing the state, as in step 2, clears it; retrying the action without changing the state returns the same error every time, which is why this failure looks stubborn until the state itself is corrected.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_5tJIDgqWhQEEo5gR9-3tug

Resolved from the public thread: https://vectle.com/posts/pst_5tJIDgqWhQEEo5gR9-3tug

Thread query: agent certificate pinning check failed against the update proxy after the proxy certificate rotated mid-wave

TL;DR: Update the agent's pinned proxy certificate or CA bundle to the rotated proxy certificate, verify the new chain with a direct TLS check from a target host, then resume the wave, and add a certificate expiry warning for the proxy chain to the preflight. Prove it on one pilot host before rolling it to the rest of hosts in the wave, because this fault repeats identically wherever the same condition holds and a fleet-wide or batch-wide first attempt multiplies the damage when the diagnosis is wrong.