Azure Key Vault CSI driver driver name secrets-store.csi.k8s.io not found after AKS node upgrade

What this fixes

After upgrading an AKS cluster (new node created, old nodes cordoned/drained), pods fail with: MountVolume.SetUp failed for volume ... : kubernetes.io/csi: mounter.SetUpAt ... "driver name secrets-store.csi.k8s.io not found in the list of registered CSI drivers". Pods mount Azure Key Vault secrets via the secrets-store CSI driver.

The fix

This is expected behavior on new nodes: pods referencing CSI volumes cannot start until the CSI driver registers on that node. During an AKS upgrade the node-driver-registrar image must be pulled onto the fresh node first, one reporter saw the pull take over 6 minutes. The recommended fixes: use the AKS managed add-on for the Secrets Store CSI driver, and mirror the CSI driver images into your own container registry and reference them in the Helm chart so the pull is fast on new nodes. The reporter confirmed mirroring the images resolved the startup failures.

Source: https://github.com/Azure/secrets-store-csi-driver-provider-azure/issues/173

Why it happens

The thread above nails the cause, so the fix addresses that directly rather than symptoms. Adapt the version numbers and paths to your setup, then verify the behavior the thread confirmed.

Source

https://github.com/Azure/secrets-store-csi-driver-provider-azure/issues/173

Asked in the thread (https://vectle.com/threads/thr_JhSe9MPPhZvsgTVCp8HTpA): After upgrading an AKS cluster (new node created, old nodes cordoned/drained), pods fail with: MountVolume.SetUp failed for volume ... : kubernetes.io/csi: mounter.SetUpAt ... "driver name secrets-store.csi.k8s.io not found in the list of registered CSI drivers". Pods mount Azure Key Vault secrets via the secrets-store CSI driver. Verified resolution: This is expected behavior on new nodes: pods referencing CSI volumes cannot start until the CSI driver registers on that node. During an AKS upgrade the node-driver-registrar image must be pulled onto the fresh node first, one reporter saw the pull take over 6 minutes. The recommended fixes: use the AKS managed add-on for the Secrets Store CSI driver, and mirror the CSI driver images into your own container registry and reference them in the Helm chart so the pull is fast on new nodes. The reporter confirmed mirroring the images resolved the startup failures.

Source: https://github.com/Azure/secrets-store-csi-driver-provider-azure/issues/173