LoRAX: TensorParallelHead object has no attribute base_layer when serving Mixtral with an adapter

TL;DR

A new Docker image was built and pushed, and the reporter confirmed the fix. This crash is a LoRAX bug in adapter loading for Mixtral's tensor-parallel head, fixed in PR #131.

Steps

1. This crash is a LoRAX bug in adapter loading for Mixtral's tensor-parallel head, fixed in PR #131. Pull a fresh LoRAX Docker image with the fix, since the maintainer confirmed the corrected image was built and pushed. If you pinned an image tag from before the fix, update it. The reporter confirmed the new image resolved the error.

Expected: Pull a fresh LoRAX Docker image with the fix, since the maintainer confirmed the corrected image was built and pushed.

When to use

You are seeing this: Starting LoRAX with mistralai/Mixtral-8x7B-Instruct-v0.1 on two A100s and sending a generate request with a LoRA adapter crashed with 'TensorParallelHead' object has no attribute 'base\layer' while loading the adapter weights. Use this skill when you run into "LoRAX: TensorParallelHead object has no attribute baselayer when serving Mixtral with an adapter".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: v0.1. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Asked in the thread (https://vectle.com/threads/thrkCvTKf2IG6ShmJEdh5jow): Starting LoRAX with mistralai/Mixtral-8x7B-Instruct-v0.1 on two A100s and sending a generate request with a LoRA adapter crashed with 'TensorParallelHead' object has no attribute 'base_layer' while loading the adapter weights. Maintainer tgaddair acknowledged it was an oversight on his side and put up PR #131. A new Docker image was built and pushed, and the reporter confirmed the fix. The verified answer: This crash is a LoRAX bug in adapter loading for Mixtral's tensor-parallel head, fixed in PR #131. Pull a fresh LoRAX Docker image with the fix, since the maintainer confirmed the corrected image was built and pushed. If you pinned an image tag from before the fix, update it. The reporter confirmed the new image resolved the error. Source: https://github.com/predibase/lorax/issues/130 Thread: https://vectle.com/threads/thrkCvTKf2IG6ShmJEdh5jow