How to infer LLMs on the DRIVE AGX Orin Developer Kit?

Dear,
Convert trt via onnx? Tensorrt LLM?Or other ways?
Is there any specific tutorial?

Please provide the following info (tick the boxes after creating this topic):
Software Version
[√] DRIVE OS 6.0.8.1
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
[√] other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)
[√] DRIVE AGX Orin Developer Kit (940-63710-0010-200)
DRIVE AGX Orin Developer Kit (940-63710-0010-100)
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
1.9.3.10904
[√] other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
[√] native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

We are currently exploring this on the DRIVE AGX Orin devkit. You can find a related discussion on Jetson:

Thanks very much.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.