Latest Supported Triton Inference Server for Jetson Xavier NX (JetPack 5.1.3)

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hello NVIDIA community,

I’m working on a project with a Jetson Xavier NX running JetPack 5.1.3. My goal is to deploy the latest compatible NVIDIA Triton Inference Server within a Docker container for optimal performance.

Here’s what I’ve tried and the challenges faced:

My Questions:

  1. What is the most up-to-date Triton Inference Server version officially supported on JetPack 5.1.3?
  2. Are there any specific Docker images or tags I should be aware of for best results with my Jetson setup?
  3. If there are compatibility limitations, can you recommend workarounds or alternative approaches to achieve my goal?

Additional Notes:

I’m open to exploring custom container builds if necessary, provided there’s some clear guidance available.

Thank you for your expertise and insights!

5
  1. please refer to this compatibiltiy table. DeepStream6.3 needs 5.1.2 GA. please refer to the first sentence in 6.3 nvinferserver.
  2. about building docker, please refer to the doc.
6

Hello and thank you for the provided links:
The documentation of nvinferserver for DS6.3 says that the supported Triton is:
Release 2.30.0, NGC Container 23.01 for Jetson

However when I take a look at Triton server GitHub
Especially at the 2.35.0 release link here

In the Jetson Jetpack Support section it says:

A release of Triton for JetPack is provided in the attached tar file: tritonserver2.35.0-jetpack5.1.2.tgz.
Release 2.35.0 corresponding to NGC container 23.06
This release supports TensorFlow 2.12.0, TensorRT 8.5.2.2, Onnx Runtime 1.15.0, PyTorch 2.1.0a0+41361538, Python 3.8 and as well as ensembles.

To me it seems like a conflicting information because the documentation says something different than description of Triton server release on GitHub (Release 2.30.0 vs 2.35.0). And of course I’d like to use the latest compatible version.

There isn’t JP5.1.3 explicitly mentioned in the 2.35.0 release, but since they both use same library versions I suppose that Triton 2.35.0 should work on JP5.1.3 as well.

Does this mean that I should create a GitHub ticket about 2.35.0 giving the CUDA 35 error on Jetpack 5.1.3 ?

Thank you
Tomas

7
  1. if you want to use the newer triton version, you can use Deepstream6.4, which will use “Release 2.40.0, NGC Container 23.11 for Jetson”, please refer to 6.4 nvinferserver. what do you mean about “the documentation says something different than description of Triton server release on GitHub”?

yes, it would be outside of DeepStream.

8

Thank you for the quick reply, but this is the thread about Jetson Xavier NX, so your advice to use Deepstream 6.4 isn’t very useful since it is not supported on this HW.

Please stick to the latest supported Deepstream, which is 6.3.

what do you mean about “the documentation says something different than description of Triton server release on GitHub”?

I am addressing the fact that the Deepstream documentation refers to Release 2.30.0, NGC Container 23.01 for Jetson
But on the other hand, according to the Triton GitHub repo the most recent release of Triton, which supports the Jetpack 5.1.x is Release 2.35.0 corresponding to NGC container 23.06.

But NGC container for this release is giving me CUDA 35 error during initialization.

And since this this is a NVIDIA forum, I am reporting this behaviour here. Could you please try to replicate the described behaviour? I think you have all the required information.

If would encounter the same behaviour, it means that there is probably a bug in the Triton container 23.06.

Thank you in advance for confirming or denying that you’ve encoutered the same error.

Thank you

9

There is no update from you for a period, assuming this is not an issue any more. Hence we are closing this topic. If need further support, please open a new one. Thanks.
From the doc, Release 2.35.0 can support jetpack5.1.2. you can ask if it can support 5.1.3 in triton github.
About “CUDA 35 error", from the doc, “Release 2.35.0” requires CUDA 12.1.1.

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.