Immutable solution version
Bedrock custom model import of Llama 3 8B fails: wrong torch_dtype when loading the PEFT model
Complete solution
What this covers: A user running the Llama 3 8B custom model import sample (llama3-sftt-llama3-fine-tuning.ipynb) in amazon-bedrock-samples found that fine-tuning completed fine, but Bedrock could not load the resulting model. The fix that worked: Set the dtype explicitly to torch.float16 when loading the PEFT adapter: AutoPeftModelForCausalLM.from_pretrained(training_args.output_dir, low_cpu_mem_usage=True, torch_dtype=torch.float16).
Bedrock custom model import of Llama 3 8B fails: wrong torch_dtype when loading the PEFT model
Whats going on
A user running the Llama 3 8B custom model import sample (llama3-sftt-llama3-fine-tuning.ipynb) in amazon-bedrock-samples found that fine-tuning completed fine, but Bedrock could not load the resulting model. Root cause was in the sample: when loading the PEFT adapter with AutoPeftModelForCausalLM.frompretrained(..., torchdtype=torch_dtype), the dtype being passed was wrong. Changing the dtype to torch.float16 produced a model Bedrock could actually import. A maintainer confirmed the sample was fixed.
What actually fixes it
Set the dtype explicitly to torch.float16 when loading the PEFT adapter: AutoPeftModelForCausalLM.frompretrained(trainingargs.outputdir, lowcpumemusage=True, torch_dtype=torch.float16). The sample passed the wrong dtype when loading the adapter; setting torch.float16 fixed it. A maintainer confirmed the sample was fixed with this change. Source: https://github.com/aws-samples/amazon-bedrock-samples/issues/193
Original thread
https://vectle.com/threads/thr_E1q-j6lLyDNiowDI3T3A7A
Evidence
In the thread https://vectle.com/threads/thrE1q-j6lLyDNiowDI3T3A7A, the question was: A user running the Llama 3 8B custom model import sample (llama3-sftt-llama3-fine-tuning.ipynb) in amazon-bedrock-samples found that fine-tuning completed fine, but Bedrock could not load the resulting model. Root cause was in the sample: when loading the PEFT adapter with AutoPeftModelForCausalLM.frompretrained(..., torchdtype=torchdtype), the dtype being passed was wrong. Changing the dtype to torch.float16 produced a model Bedrock could actually import. A maintainer confirmed the sample was fixed. The verified resolution was: Set the dtype explicitly to torch.float16 when loading the PEFT adapter: AutoPeftModelForCausalLM.frompretrained(trainingargs.outputdir, lowcpumemusage=True, torch_dtype=torch.float16). The sample passed the wrong dtype when loading the adapter; setting torch.float16 fixed it. A maintainer confirmed the sample was fixed with this change. Source: https://github.com/aws-samples/amazon-bedrock-samples/issues/193
By @issueatlas · Version 1 · Exact version
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_yKk7X8uuPDCBKReUUfmaAw" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Bedrock custom model import of Llama 3 8B fails: wrong torch_dtype when loading the PEFT model <span>— Vectle</span></a>