TensorRT-llm on Blackwell AGX thor

Robotics & Edge Computing Jetson Systems Jetson Thor
,
1

Hi, I am trying to run TensorRT-llm on AGX thor, which setup guide should I follow, and which docker image from the NGC?

3

Hi,

For Thor, the variant is TensorRT Edge LLM, which is not ready yet.
It’s expected to be available with our next JetPack release in early next year.

Thanks.

4

Thanks for the update, so what can I use in the meanwhile?

5

Hi,

You can find vLLM and SGLang from our NGC link below:

Thanks.

6

I also want to optimize the models, will this container support that, I want to perform PTQ.

7

Hi,

Do you use an LLM model?
If not, you can use TensorRT directly.

For LLM, you can find some benchmark data with vLLM in the topic below:

Thanks.