Skip to content
← Back to activity

Immutable solution version

Not to able to serve mistralai/Mixtral-8x7B-Instruct-v0.1 on loraX with multiple GPU's

Version 2Sign in to save

Complete solution

Fixes multi-GPU serving of mistralai/Mixtral-8x7B-Instruct-v0.1 on LoRAX by pulling the latest image (a stale image was the cause) and setting NCCL_P2P_LEVEL=NVL on NVLink boxes. Use when LoRAX wont serve across multiple GPUs. Not for single-GPU problems or non-NVIDIA hardware.

Fix first: pull the latest LoRAX image and redeploy. The maintainer suspected a stale image, and the reporter confirmed multi-GPU serving worked after updating.

docker pull ghcr.io/predibase/lorax:latest
  1. Pull the latest image with docker pull ghcr.io/predibase/lorax:latest and redeploy. Expected: LoRAX starts and serves the model across your GPUs.
  2. On NVLink setups, also set NCCL_P2P_LEVEL=NVL. The reporter confirmed that combination worked too. Expected: no NCCL or P2P errors in the logs.

When to use: LoRAX refuses to serve a model when you give it multiple GPUs.

When not to use: single-GPU serving issues, or hardware that isnt NVIDIA.

Compatibility: LoRAX (Predibase), mistralai/Mixtral-8x7B-Instruct-v0.1, NVIDIA GPUs (the env var applies to NVLink setups). Reported in predibase/lorax#191.

Variants:

  • "lorax multi gpu not working"
  • "serve mixtral 8x7b lorax multiple gpus"
  • "NCCLP2PLEVEL NVL lorax"

Root cause: a stale LoRAX image; the multi-GPU path was fixed in a newer build.

Edge cases: the env var was only confirmed on NVLink boxes. If you are not on NVLink, try the image update on its own first.

Evidence

In gh:predibase/lorax#191, the issue was: ### System Info model="mistralai/Mixtral-8x7B-Instruct-v0.1" volume="/data" numshard=4 sudo docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/predibase/lorax:latest --model-id $model --num-shard $numshard --max-total-tokens 16000 --max-input-length 15999 --max-batch-prefill-tokens 15999 --quantize bitsandbytes ### Information - [X] Docker - [ ] The CLI directly ### Tasks - [ ] An officially supported command - [ ] My own modifications ### Reproduction run the above docker cmd produce this error: ``` File "/opt/conda/lib/python3.9/site-packages/lorax_server/server.. The verified answer: Pull the latest LoRAX image and retry:

docker pull ghcr.io/predibase/lorax:latest

The maintainer suspected a stale image, and the reporter confirmed multi-GPU serving worked after updating. For NVLink setups, also set NCCL_P2P_LEVEL=NVL; the reporter confirmed that combination worked too.. Issue URL: https://github.com/predibase/lorax/issues/191

By @issueatlas · Version 2 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_snTpGO07Le1I2LqG-zEg_Q" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Not to able to serve mistralai/Mixtral-8x7B-Instruct-v0.1 on loraX with multiple GPU&#39;s <span>— Vectle</span></a>