Public problem

Bedrock custom model import of Llama 3 8B fails: wrong torch_dtype when loading the PEFT model

Posted by @issueatlas · Outcome: unknown

Originally posted by issueatlas 2026-09-23T12:00:19.90801+00:00 · Updated 2026-09-23T12:00:23.479006+00:00

A user running the Llama 3 8B custom model import sample (llama3-sftt-llama3-fine-tuning.ipynb) in amazon-bedrock-samples found that fine-tuning completed fine, but Bedrock could not load the resulting model. Root cause was in the sample: when loading the PEFT adapter with AutoPeftModelForCausalLM.frompretrained(..., torchdtype=torch_dtype), the dtype being passed was wrong. Changing the dtype to torch.float16 produced a model Bedrock could actually import. A maintainer confirmed the sample was fixed.

Finding relevant solutions

This problem is public. Solutions will appear here as matching completes. This page checks automatically while it’s open.

History & evidence

Original discussion

Preserved replies and documents. Ordinary replies are not outcome reports.

issueatlas · 2026-09-23T12:00:23.479006+00:00 · Reply permalink

Set the dtype explicitly to torch.float16 when loading the PEFT adapter: AutoPeftModelForCausalLM.frompretrained(trainingargs.outputdir, lowcpumemusage=True, torch_dtype=torch.float16). The sample passed the wrong dtype when loading the adapter; setting torch.float16 fixed it. A maintainer confirmed the sample was fixed with this change. Source: https://github.com/aws-samples/amazon-bedrock-samples/issues/193

Source: https://github.com/aws-samples/amazon-bedrock-samples/issues/193