Immutable solution version
TensorRT LLM is not working.
Complete solution
How to tensorRT LLM is not working.. Verified in gh:modal-labs/modal-client#1318.
What this teaches
How to tensorRT LLM is not working..
The problem
Issue modal-labs/modal-client#1318 (closed, 6 comments): ### Steps to reproduce Here are the steps to reproduce the TensorRT 1. I am using this custom docker image named anindyadeep/tensorrt_llm:release that I build through TensorRT-LLM. 2.
What worked (verified answer)
TensorRT-LLM custom image failing on Modal: pass the gpu argument to runfunction (and pipinstall), not to fromimage(). In your code: .runfunction(helloworld, secrets=[Secret.fromname("your-huggingface-secret")], gpu=GPU_CONFIG). The gpu kwarg belongs on the function call that executes, not on the image builder.
Source
gh:modal-labs/modal-client#1318 - https://github.com/modal-labs/modal-client/issues/1318
Evidence
In gh:modal-labs/modal-client#1318, the reporter ran into: Issue modal-labs/modal-client#1318 (closed, 6 comments): ### Steps to reproduce Here are the steps to reproduce the TensorRT 1. I am using this custom docker image named anindyadeep/tensorrt_llm:release that I build through TensorRT-LLM. 2. The verified answer was: TensorRT-LLM custom image failing on Modal: pass the gpu argument to runfunction (and pipinstall), not to fromimage(). In your code: .runfunction(helloworld, secrets=[Secret.fromname("your-huggingface-secret")], gpu=GPU_CONFIG). The gpu kwarg belongs on the function call that executes, not on the image builder. Source issue: https://github.com/modal-labs/modal-client/issues/1318
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl__2N3t4Zqd1eQ2sxLmtzv3Q" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">TensorRT LLM is not working. <span>— Vectle</span></a>