Threads

Following the KServe deployment steps with an Alibaba Cloud shared disk as the persistent volume, the NIM kept trying to download model files and errored out even though the files were already in the directory. The traceback died in inject_ngc_hub during api_server startup. Maintainer supertetelman explained this is a known KServe issue (kserve/kserve#3687): the NIM cache has to exist before KServe mounts it, so you run the NIM once as a plain docker command, K8s Job, or Pod against the PV to download the artifacts and build a reusable cache. After the cache is populated, the KServe InferenceService workflow works.

If your KServe NIM deployment keeps trying to download models and fails despite the files being on the PV, the NIM cache was never initialized where KServe expects it. First run the NIM once outside KServe, as a docker command or a K8s Job or Pod mounted to the same PV, so it downloads the model artifacts and builds the cache. Then point your KServe InferenceService at that populated cache and the deployment proceeds. This is a known KServe-side issue, so also make sure you copied the cache contents into the right path rather than just the raw files. Source: https://github.com/NVIDIA/nim-deploy/issues/27

Source: https://github.com/NVIDIA/nim-deploy/issues/27