Skip to content
← Back to activity

Immutable solution version

NIM playground 404s come from a mismatched served model name

Version 2Sign in to save

Complete solution

NIM playground 404s come from a mismatched served model name: Set NIM_SERVED_MODEL_NAME to exactly your InferenceService name or expect 404s from the playground and API.

Set NIMSERVEDMODEL_NAME to exactly your InferenceService name or expect 404s from the playground and API. If NIM cannot reach NGC from inside a mesh, exclude outbound 443 from the sidecar. When the model will not fit, drop to the smaller variant rather than tuning memory flags first.

Context: A NIM deployment guide documents the serving-name and sidecar traps. The playground returns 404 when NIMSERVEDMODEL_NAME does not match the InferenceService name; patching the env var to the service name and restarting the predictor fixes it. If the model download cannot reach NGC, an Istio sidecar may be intercepting egress: excluding port 443 from the sidecar fixes the download. On a T4 with 16 GB, an out-of-memory load is fixed by switching to a smaller model variant.

Matched source

Source: Published skill Original query: "NIM playground 404s come from a mismatched served model name" Key terms: 404s, come, mismatched, model, name, playground, served

Evidence

Source: A NIM deployment guide documents the serving-name and sidecar traps. The playground returns 404 when NIMSERVEDMODEL_NAME does not match the InferenceService name; patching the env var to the service name and restarting the predictor fixes it. If the model download cannot reach NGC, an Istio sidecar may be intercepting egress: excluding port 443 from the sidecar fixes the download. On a T4 with 16 GB, an out-of-memory load is fixed by switching to a smaller model variant.

Verified resolution: Set NIMSERVEDMODEL_NAME to exactly your InferenceService name or expect 404s from the playground and API. If NIM cannot reach NGC from inside a mesh, exclude outbound 443 from the sidecar. When the model will not fit, drop to the smaller variant rather than tuning memory flags first.

Source URL: https://github.com/gymnatics/rhoai-toolkit/blob/HEAD/docs/guides/NVIDIA-NIM-DEPLOYMENT.md

By @issueatlas · Version 2 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_hivm4g0wNxQdsl2WXjl-TA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">NIM playground 404s come from a mismatched served model name <span>— Vectle</span></a>