Immutable solution version
KServe deployment fails when using a cloud shared disk as PV even with model files already present
Complete solution
What caused "KServe deployment fails when using a cloud shared disk as PV even with model files already present" and the fix that actually worked. Distilled from the thread where it was verified.
KServe deployment fails when using a cloud shared disk as PV even with model files already present
What was going on
Following the KServe deployment steps with an Alibaba Cloud shared disk as the persistent volume, the NIM kept trying to download model files and errored out even though the files were already in the directory. The traceback died in inject_ngc_hub during api_server startup. Maintainer supertetelman explained this is a known KServe issue (kserve/kserve#3687): the NIM cache has to exist before KServe mounts it, so you run the NIM once as a plain docker command, K8s Job, or Pod against the PV to download the artifacts and build a reusable cache. After the cache is populated, the KServe InferenceService workflow works.
The fix that worked
If your KServe NIM deployment keeps trying to download models and fails despite the files being on the PV, the NIM cache was never initialized where KServe expects it. First run the NIM once outside KServe, as a docker command or a K8s Job or Pod mounted to the same PV, so it downloads the model artifacts and builds the cache. Then point your KServe InferenceService at that populated cache and the deployment proceeds. This is a known KServe-side issue, so also make sure you copied the cache contents into the right path rather than just the raw files.
Where this came from
https://github.com/NVIDIA/nim-deploy/issues/27
Evidence
Someone asked: Following the KServe deployment steps with an Alibaba Cloud shared disk as the persistent volume, the NIM kept trying to download model files and errored out even though the files were already in the directory. The traceback died in inject_ngc_hub during apiserver startup. Maintainer supertetelman explained this is a known KServe issue (kserve/kserve#3687): the NIM cache has to exist before KServe mounts it, so you run the NIM once as a plain docker command, K8s Job, or Pod against the PV to download the artifacts and build a reusable cache. After the cache is populated, the KServe InferenceService workflow works. The verified resolution: If your KServe NIM deployment keeps trying to download models and fails despite the files being on the PV, the NIM cache was never initialized where KServe expects it. First run the NIM once outside KServe, as a docker command or a K8s Job or Pod mounted to the same PV, so it downloads the model artifacts and builds the cache. Then point your KServe InferenceService at that populated cache and the deployment proceeds. This is a known KServe-side issue, so also make sure you copied the cache contents into the right path rather than just the raw files. Thread: https://vectle.com/threads/thrtuFT1y_EJiTYN1JoAsM91g
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_8oRupJ4djJ3xm0ADL7RMlw" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">KServe deployment fails when using a cloud shared disk as PV even with model files already present <span>— Vectle</span></a>