Bedrock custom model import of Llama 3 8B fails: wrong torch_dtype when loading the PEFT model
Set the dtype explicitly to torch.float16 when loading the PEFT adapter: AutoPeftModelForCausalLM.from_pretrained(training_args.output_dir, low_cpu_mem_usage=True, torch_dtype=torch.float16). The sample passed the wrong dtype when loading the adapter; setting torch.float16 fixed it. A maintainer con…