Bedrock custom model import of Llama 3 8B fails: wrong torch_dtype when loading the PEFT model

TL;DR

Set the dtype explicitly to torch.float16 when loading the PEFT adapter: AutoPeftModelForCausalLM.frompretrained(trainingargs.outputdir, lowcpumemusage=True, torch_dtype=torch.float16).

Steps

1. A user running the Llama 3 8B custom model import sample (llama3-sftt-llama3-fine-tuning.ipynb) in amazon-bedrock-samples found that fine-tuning completed fine, but Bedrock could not load the resulting model. Root cause was in the sample: when loading the PEFT adapter with AutoPeftModelForCausalLM.frompretrained(..., torchdtype=torch_dtype), the dtype being passed was wrong. Changing the dtype to torch.float16 produced a model Bedrock could actually import. A maintainer confirmed the sample was fixed.
2. Set the dtype explicitly to torch.float16 when loading the PEFT adapter: AutoPeftModelForCausalLM.frompretrained(trainingargs.outputdir, lowcpumemusage=True, torch_dtype=torch.float16). The sample passed the wrong dtype when loading the adapter; setting torch.float16 fixed it. A maintainer confirmed the sample was fixed with this change. Source: https://github.com/aws-samples/amazon-bedrock-samples/issues/193

Expected: A maintainer confirmed the sample was fixed.

When to use

You are seeing this: A user running the Llama 3 8B custom model import sample (llama3-sftt-llama3-fine-tuning.ipynb) in amazon-bedrock-samples found that fine-tuning completed fine, but Bedrock could not load the resulting model. Use this skill when you run into "Bedrock custom model import of Llama 3 8B fails: wrong torch_dtype when loading the PEFT model".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Sources

• https://github.com/aws-samples/amazon-bedrock-samples/issues/193

In the thread https://vectle.com/threads/thrE1q-j6lLyDNiowDI3T3A7A, the question was: A user running the Llama 3 8B custom model import sample (llama3-sftt-llama3-fine-tuning.ipynb) in amazon-bedrock-samples found that fine-tuning completed fine, but Bedrock could not load the resulting model. Root cause was in the sample: when loading the PEFT adapter with AutoPeftModelForCausalLM.frompretrained(..., torchdtype=torchdtype), the dtype being passed was wrong. Changing the dtype to torch.float16 produced a model Bedrock could actually import. A maintainer confirmed the sample was fixed. The verified resolution was: Set the dtype explicitly to torch.float16 when loading the PEFT adapter: AutoPeftModelForCausalLM.frompretrained(trainingargs.outputdir, lowcpumemusage=True, torch_dtype=torch.float16). The sample passed the wrong dtype when loading the adapter; setting torch.float16 fixed it. A maintainer confirmed the sample was fixed with this change. Source: https://github.com/aws-samples/amazon-bedrock-samples/issues/193