LoRAX: TensorParallelHead object has no attribute base_layer when serving Mixtral with an adapter

LoRAX: TensorParallelHead object has no attribute base_layer when serving Mixtral with an adapter

The problem

Starting LoRAX with mistralai/Mixtral-8x7B-Instruct-v0.1 on two A100s and sending a generate request with a LoRA adapter crashed with 'TensorParallelHead' object has no attribute 'base_layer' while loading the adapter weights. Maintainer tgaddair acknowledged it was an oversight on his side and put up PR #131. A new Docker image was built and pushed, and the reporter confirmed the fix.

The verified fix

This crash is a LoRAX bug in adapter loading for Mixtral's tensor-parallel head, fixed in PR #131. Pull a fresh LoRAX Docker image with the fix, since the maintainer confirmed the corrected image was built and pushed. If you pinned an image tag from before the fix, update it. The reporter confirmed the new image resolved the error. Source: https://github.com/predibase/lorax/issues/130

Asked in the thread (https://vectle.com/threads/thrkCvTKf2IG6ShmJEdh5jow): Starting LoRAX with mistralai/Mixtral-8x7B-Instruct-v0.1 on two A100s and sending a generate request with a LoRA adapter crashed with 'TensorParallelHead' object has no attribute 'base_layer' while loading the adapter weights. Maintainer tgaddair acknowledged it was an oversight on his side and put up PR #131. A new Docker image was built and pushed, and the reporter confirmed the fix. The verified answer: This crash is a LoRAX bug in adapter loading for Mixtral's tensor-parallel head, fixed in PR #131. Pull a fresh LoRAX Docker image with the fix, since the maintainer confirmed the corrected image was built and pushed. If you pinned an image tag from before the fix, update it. The reporter confirmed the new image resolved the error. Source: https://github.com/predibase/lorax/issues/130 Thread: https://vectle.com/threads/thrkCvTKf2IG6ShmJEdh5jow