What is the way to run ONNX models with triton-server for GPU inferencing with Jetson Nano Orin NX with 16GB with Jetpack 5.1.1?

[hyates]

Goal

I want to get onnx runtime and cuda on the Jetson Nano Orin NX with 16GB so I can load ONNX models into triton-server and take the model(s) for a spin with perf-analyzer with the GPU

Question

Things move quickly with these devices, so I want to be sure I am doing an update that will not get me in trouble with my device. There are directions here for onnx runtime, but I think it is written more for Jetson 6.0?

I am thinking that I need onnx runtime engine and cuda (updated). Also any tips on config.pbtxt would be appreciated.

Here is the version of Jetpack

dpkg-query --showformat='${Version}' --show nvidia-jetpack
5.1.1-b56

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Sun_Oct_23_22:16:07_PDT_2022
Cuda compilation tools, release 11.4, V11.4.315
Build cuda_11.4.r11.4/compiler.31964100_0

I read this thread here, but this is for jetpack 5.1.2. I saw this open issue on github, but no real update on the status. Another weird thing is nvcc --version is not showing anything, but I thought this came with Jetpack?

I appreciate folks patience in their answers because I am trying to avoid messing up the device and having to reflash. Thanks!

[AastaLLL]

Hi,

Do you have any dependency on JetPack 5.1.1?
If not, it’s recommended to upgrade to Jetpack 6.0.

Triton started to support Jetson with containers from JetPack 6.0.
This should save a lot of installation effort:

Thanks.