Nvidia Jetson Orin Nano tensorrt llm

Autonomous Machines Jetson & Embedded Systems Jetson Orin Nano
,
1

Hi,

I had a jetson nano and i was unable to run tensorrt or tensorrt llm on it. I am planning to buy a Jetso Orin Nano. I would like to know whether the jetson orin nano supports tensorrt llm? I could not find the information in the documents

Thank you

3

Yes, you can refer to Small LLMs and Mini VLMs on Orin Nano - Jetson & Embedded Systems / Jetson Projects - NVIDIA Developer Forums

4

@kayccc thank you for your reply.
i wanted to run Bert QA model, for example given in

It needed tensorrt 10, which not coming with jetpack 6.

6

Hi,

TensorRT 10 can work with JetPack 6.
For example, TensorRT 10.1 GA for JetPack

But TensorRT-LLM doesn’t officially support Jetson due to some dependency issues.

Thanks.

7

@AastaLLL Thankyou for the message.
with JetPack 6, in Jetson Orin Nano, will the examples given here work? (I understood this is not tensorrt llm)

In the examples i cannot find a suitable version for jetpack 6.

8

Hi,

The sample is also available for TensorRT 8.6 (from JetPack 6):

So you can give it a try.
But usually, LLMs require large memory so you might need to apply the below step:

Thanks.

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.