KServe deployment fails when using a cloud shared disk as PV even with model files already present

TL;DR

Maintainer supertetelman explained this is a known KServe issue (kserve/kserve#3687): the NIM cache has to exist before KServe mounts it, so you run the NIM once as a plain docker command, K8s Job, or Pod against the PV to download the artifacts and build a reusable cache.

Steps

1. If your KServe NIM deployment keeps trying to download models and fails despite the files being on the PV, the NIM cache was never initialized where KServe expects it. First run the NIM once outside KServe, as a docker command or a K8s Job or Pod mounted to the same PV, so it downloads the model artifacts and builds the cache. Then point your KServe InferenceService at that populated cache and the deployment proceeds. This is a known KServe-side issue, so also make sure you copied the cache contents into the right path rather than just the raw files.

When to use

You are seeing this: Following the KServe deployment steps with an Alibaba Cloud shared disk as the persistent volume, the NIM kept trying to download model files and errored out even though the files were already in the directory. Use this skill when you run into "KServe deployment fails when using a cloud shared disk as PV even with model files already present".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Sources

• https://github.com/NVIDIA/nim-deploy/issues/27

Someone asked: Following the KServe deployment steps with an Alibaba Cloud shared disk as the persistent volume, the NIM kept trying to download model files and errored out even though the files were already in the directory. The traceback died in inject_ngc_hub during apiserver startup. Maintainer supertetelman explained this is a known KServe issue (kserve/kserve#3687): the NIM cache has to exist before KServe mounts it, so you run the NIM once as a plain docker command, K8s Job, or Pod against the PV to download the artifacts and build a reusable cache. After the cache is populated, the KServe InferenceService workflow works. The verified resolution: If your KServe NIM deployment keeps trying to download models and fails despite the files being on the PV, the NIM cache was never initialized where KServe expects it. First run the NIM once outside KServe, as a docker command or a K8s Job or Pod mounted to the same PV, so it downloads the model artifacts and builds the cache. Then point your KServe InferenceService at that populated cache and the deployment proceeds. This is a known KServe-side issue, so also make sure you copied the cache contents into the right path rather than just the raw files. Thread: https://vectle.com/threads/thrtuFT1y_EJiTYN1JoAsM91g