Triton on Jetson Orin

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
1

I was wondering how can i run Triton inference server on my Jetson Orin device. Is there any docker image already available. As of what i have understood by now that Triton is not yet available for Jetson devices. Please help me with this. I want to run Triton on my Jetson orin

2

Hi,
Do you use AGX Orin or Orin Nano. This topic is in Orin Nano category but it looks like you are asking about AGX Orin. Would like to confirm which platform you are using.

4

Hi,

Please check below:

Thanks.

5

Hi , I just verified its AGX orin. I want to use Trition inference server on it. Can i directly pull docker image?. As of what i checked there are very few to no docs about Triton on Jetson

6

HI, Thanks for the reply. But i want to use Triton inference server on AGX Orin. I want to run multiple (5 deep learning inferences) in parallel on AGX orin.

7

Hi,

You can start with below tutorial:

Thanks.

8

I think the deepstream is limited to Multi object detection, Multi Image classification and Segmentation. But i want to run any model in parallel. Lets say i have machine learning models or any deep learning models (Speech enhancements, Super resolution ect…). I want to run multiple models and any sort of models in parallel on my AGX Orin device.

9

Hi,

You can refer to the Triton document for the examples:

Thanks.

10

Hi, thanks for the reply.

I dont really think you are getting me. I need parallel model inferences on AGX orin. I already mentioned that Triton provides dynamic batching feature for parallel model inference. But jetpack versions dont have Triton support. I need solution to use Triton( or any alternative) to run mutiple models in parallel at single time. Triton for jetpack doesnt have docker images. So do you have any inputs on this?

11

Hi @utkarsh.tiwariblr47, check the Triton releases page on GitHub for the Jetson releases:

See here for the containers:

12

Thanks for your response.

If i follow the steps of installation in the page mentioned, do i need to do any configuration after the installation.

Doubt 2:

In this page it is mentioned " For Jetson devices which support Jetpack 6.0 and above, Triton now publishes containers, based on the latest version of Jetpack, on NGC with the suffix -igpu".

So does that mean i can simply pull the container image according to the following link. Would that avoid manual installation of Triton inference server on Jetson?

13

Hi,

Yes, please use the container with -igpu tag and it should have Triton preinstalled.

Thanks.

15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.