How to parallel inference model in nvidia Xavier nx platform

[AastaLLL]

Hi,

You can launch trtexec in the different consoles for parallel inference.

$ /usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/mnist/mnist.onnx --useDLACore=0 --allowGPUFallback
$ /usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/mnist/mnist.onnx --useDLACore=1 --allowGPUFallback

If you are finding a multi-thread inference example, please check the below comment:

How to use TensorRT by the multi-threading package of python

Hi, The sample is ready. The workflow looks like this: Master: create TensorRT engine and buffer, store the created CUDA context. Thread1: restore master’s CUDA context, run inference, destroy its CUDA context. Master: destroy the stored CUDA context. The sample is created with TensorRT default MNIST model. 1. Prepare data $ /usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/mnist/mnist.onnx --saveEngine=mnist.trt $ cd /usr/src/tensorrt/data/mnist/ $ sudo pip3 install pillow $ pyt…

We create two engines for the same model with the corresponding context to make it parallel.

Thanks