LoRAX fails to start with microsoft/Phi-3-mini-128k-instruct

TL;DR

If LoRAX refuses to start with Phi-3-mini-128k-instruct and the logs mention RoPE scaling or model type detection, you need the fix from PR #499. Pull the latest main Docker image, which the reporter confirmed starts the model successfully. The remaining warnings about special token IDs are cosmetic and do not block startup. If you are on an older image tag, upgrade before debugging your config further.

When to use

You are seeing this: Pull the latest main Docker image, which the reporter confirmed starts the model successfully. Use this skill when you run into "LoRAX fails to start with microsoft/Phi-3-mini-128k-instruct".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Steps

1. If LoRAX refuses to start with Phi-3-mini-128k-instruct and the logs mention RoPE scaling or model type detection, you need the fix from PR #499. Pull the latest main Docker image, which the reporter confirmed starts the model successfully. The remaining warnings about special token IDs are cosmetic and do not block startup. If you are on an older image tag, upgrade before debugging your config further.

Asked in the thread (https://vectle.com/threads/thr2ON0F7QZ3d8dmTNRSyHL8g): Starting LoRAX with microsoft/Phi-3-mini-128k-instruct failed during shard startup with errors around special tokens and model type detection, including You are using a model of type phi3 to instantiate a model of type . Maintainer tgaddair identified it as an issue with auto-detecting RoPE scaling for Phi-3. The reporter was stuck on the stock main image while the fix was still being built and pushed. The fix went into PR #499, and the reporter confirmed the model started successfully on the new main image, though a tokenizer serialization warning about special token IDs remained. The verified answer: If LoRAX refuses to start with Phi-3-mini-128k-instruct and the logs mention RoPE scaling or model type detection, you need the fix from PR #499. Pull the latest main Docker image, which the reporter confirmed starts the model successfully. The remaining warnings about special token IDs are cosmetic and do not block startup. If you are on an older image tag, upgrade before debugging your config further. Source: https://github.com/predibase/lorax/issues/485 Thread: https://vectle.com/threads/thr2ON0F7QZ3d8dmTNRSyHL8g