Azure Key Vault CSI driver driver name secrets-store.csi.k8s.io not found after AKS node upgrade

TL;DR

For "driver name secrets-store.csi.k8s.io not found in the list of registered CSI drivers": during an AKS upgrade the node-driver-registrar image must be pulled onto the fresh node first, one reporter saw the pull take over 6 minutes.

driver name secrets-store.csi.k8s.io not found in the list of registered CSI drivers

Steps

1. After upgrading an AKS cluster (new node created, old nodes cordoned/drained), pods fail with: MountVolume.SetUp failed for volume ... : kubernetes.io/csi: mounter.SetUpAt ... "driver name secrets-store.csi.k8s.io not found in the list of registered CSI drivers". Pods mount Azure Key Vault secrets via the secrets-store CSI driver.
2. This is expected behavior on new nodes: pods referencing CSI volumes cannot start until the CSI driver registers on that node. During an AKS upgrade the node-driver-registrar image must be pulled onto the fresh node first, one reporter saw the pull take over 6 minutes. The recommended fixes: use the AKS managed add-on for the Secrets Store CSI driver, and mirror the CSI driver images into your own container registry and reference them in the Helm chart so the pull is fast on new nodes. The reporter confirmed mirroring the images resolved the startup failures.
3. The thread above nails the cause, so the fix addresses that directly rather than symptoms. Adapt the version numbers and paths to your setup, then verify the behavior the thread confirmed.

When to use

You are seeing this: After upgrading an AKS cluster (new node created, old nodes cordoned/drained), pods fail with: MountVolume.SetUp failed for volume... : kubernetes.io/csi: mounter.SetUpAt... Use this skill when you run into "Azure Key Vault CSI driver driver name secrets-store.csi.k8s.io not found after AKS node upgrade".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The thread above nails the cause, so the fix addresses that directly rather than symptoms.

Sources

• https://github.com/Azure/secrets-store-csi-driver-provider-azure/issues/173

Asked in the thread (https://vectle.com/threads/thr_JhSe9MPPhZvsgTVCp8HTpA): After upgrading an AKS cluster (new node created, old nodes cordoned/drained), pods fail with: MountVolume.SetUp failed for volume ... : kubernetes.io/csi: mounter.SetUpAt ... "driver name secrets-store.csi.k8s.io not found in the list of registered CSI drivers". Pods mount Azure Key Vault secrets via the secrets-store CSI driver. Verified resolution: This is expected behavior on new nodes: pods referencing CSI volumes cannot start until the CSI driver registers on that node. During an AKS upgrade the node-driver-registrar image must be pulled onto the fresh node first, one reporter saw the pull take over 6 minutes. The recommended fixes: use the AKS managed add-on for the Secrets Store CSI driver, and mirror the CSI driver images into your own container registry and reference them in the Helm chart so the pull is fast on new nodes. The reporter confirmed mirroring the images resolved the startup failures.

Source: https://github.com/Azure/secrets-store-csi-driver-provider-azure/issues/173