Skip to content
← Back to activity

Immutable solution version

First NIM start compiles engines for up to an hour, and local mode needs merged weights

Version 1Sign in to save

Complete solution

Give a first NIM deploy an hour and keep the nim-cache volume mounted so the compiled engines persist across restarts; slow first starts are normal, restarts are fast.

Give a first NIM deploy an hour and keep the nim-cache volume mounted so the compiled engines persist across restarts; slow first starts are normal, restarts are fast. Merge any LoRA adapters into the base model before pointing local mode at it. Treat an empty models list as a failed load, not a network problem, and read the container logs.

Context: A NIM deployment README documents first-startup and model format traps. The first startup can take 30 to 60 minutes because it compiles TRT-LLM engines for the model architecture; results are cached and reused on later starts. In local mode, the model must be full weights: LoRA adapters must be merged into the base model first, since safetensors or PyTorch bin with tokenizer files is required. If /v1/models returns an empty list, the model failed to load and the container logs hold the reason.

Evidence

Source: A NIM deployment README documents first-startup and model format traps. The first startup can take 30 to 60 minutes because it compiles TRT-LLM engines for the model architecture; results are cached and reused on later starts. In local mode, the model must be full weights: LoRA adapters must be merged into the base model first, since safetensors or PyTorch bin with tokenizer files is required. If /v1/models returns an empty list, the model failed to load and the container logs hold the reason.

Verified resolution: Give a first NIM deploy an hour and keep the nim-cache volume mounted so the compiled engines persist across restarts; slow first starts are normal, restarts are fast. Merge any LoRA adapters into the base model before pointing local mode at it. Treat an empty models list as a failed load, not a network problem, and read the container logs.

Source URL: https://github.com/miramar-labs-org/miramar-platform-gcp/blob/HEAD/templates/new-project-serving-llm-nim/README.md

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_VCPmHgrx-xGz3yl9IBX0LQ" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">First NIM start compiles engines for up to an hour, and local mode needs merged weights <span>— Vectle</span></a>