Can TensorRT-LLM be used on Jetson Orin NX with JetPack 6.1?

[yijun0404]

I am using a Jetson Orin NX with JetPack 6.1, and I am trying to set up TensorRT-LLM. I followed the instructions on the TensorRT-LLM installation page, but I was unable to get it working.

TensorRT-LLM is supported on Jetson Orin NX with JetPack 6.1? If it is, what are the correct steps to get it running?

[AastaLLL]

Hi,

We are checking the latest status internally.
Will provide more info to you later.

Thanks.

[AastaLLL]

Hi,

We are working on supporting TRT-LLM on Jetson. Stay tuned.

Thanks.

[bigbighuang]

@AastaLLL

.

Hi,
Any updates on the progress?

[AastaLLL]

Hi,

Our internal team is actively working on this.
Please pay attention to our announcement for the latest info.

Thanks.

[AastaLLL]

Hi,

Please find the below link for the info:

TensorRT-LLM for Jetson

TensorRT-LLM is a high-performance LLM inference library with advanced quantization, attention kernels, and paged KV caching. Initial support for TensorRT-LLM in JetPack 6.1 has been included in the v0.12.0-jetson branch of the TensorRT-LLM repo for Jetson AGX Orin. We’ve made pre-compiled TensorRT-LLM wheels and containers available, along with these guides and additional documentation: > TensorRT-LLM Deployment on Jetson Orin

Thanks.