Skip to content
← Back to activity

Immutable solution version

KServe deployment fails when using a cloud shared disk as PV even with model files already present

Version 1Sign in to save

Complete solution

What caused "KServe deployment fails when using a cloud shared disk as PV even with model files already present" and the fix that actually worked. Distilled from the thread where it was verified.

KServe deployment fails when using a cloud shared disk as PV even with model files already present

What was going on

Following the KServe deployment steps with an Alibaba Cloud shared disk as the persistent volume, the NIM kept trying to download model files and errored out even though the files were already in the directory. The traceback died in inject_ngc_hub during api_server startup. Maintainer supertetelman explained this is a known KServe issue (kserve/kserve#3687): the NIM cache has to exist before KServe mounts it, so you run the NIM once as a plain docker command, K8s Job, or Pod against the PV to download the artifacts and build a reusable cache. After the cache is populated, the KServe InferenceService workflow works.

The fix that worked

If your KServe NIM deployment keeps trying to download models and fails despite the files being on the PV, the NIM cache was never initialized where KServe expects it. First run the NIM once outside KServe, as a docker command or a K8s Job or Pod mounted to the same PV, so it downloads the model artifacts and builds the cache. Then point your KServe InferenceService at that populated cache and the deployment proceeds. This is a known KServe-side issue, so also make sure you copied the cache contents into the right path rather than just the raw files.

Where this came from

https://github.com/NVIDIA/nim-deploy/issues/27

Evidence

Someone asked: Following the KServe deployment steps with an Alibaba Cloud shared disk as the persistent volume, the NIM kept trying to download model files and errored out even though the files were already in the directory. The traceback died in inject_ngc_hub during apiserver startup. Maintainer supertetelman explained this is a known KServe issue (kserve/kserve#3687): the NIM cache has to exist before KServe mounts it, so you run the NIM once as a plain docker command, K8s Job, or Pod against the PV to download the artifacts and build a reusable cache. After the cache is populated, the KServe InferenceService workflow works. The verified resolution: If your KServe NIM deployment keeps trying to download models and fails despite the files being on the PV, the NIM cache was never initialized where KServe expects it. First run the NIM once outside KServe, as a docker command or a K8s Job or Pod mounted to the same PV, so it downloads the model artifacts and builds the cache. Then point your KServe InferenceService at that populated cache and the deployment proceeds. This is a known KServe-side issue, so also make sure you copied the cache contents into the right path rather than just the raw files. Thread: https://vectle.com/threads/thrtuFT1y_EJiTYN1JoAsM91g

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_8oRupJ4djJ3xm0ADL7RMlw" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">KServe deployment fails when using a cloud shared disk as PV even with model files already present <span>— Vectle</span></a>