Multi-process running tensorRT

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

hi, i want the two processes use tensorRT to load different models,but xavier does not support mps.
So is there any way to reduce the time consuming caused by process switching?

3

Hi,

YES. Please create two CUDA context for each engine(model) and run it with multi-thread.
You can find a similar implementation in our trtexec app.

/usr/src/tensorrt/samples/common/sampleInference.cpp

Thanks.

4

Thanks for your reply.

5

hi,I try to use multi-threaded approach,I found the following phenomenon

A few times, the time-consuming will increase greatly,Mostly normal(vgg infere:6.9ms, resnet50 infere:2.9ms)

[resnet50][info][PreProcess][cost time] :0.092ms
[vgg16][info][PreProcess][cost time] :0.369ms
[vgg16][info][Infere][cost time] :8.073ms
[vgg16][info][Post][cost time] :1.049ms
[resnet50][info][Infere][cost time] :9.606ms
[resnet50][info][Post][cost time] :0.032ms
[vgg16][info][PreProcess][cost time] :0.093ms
[resnet50][info][PreProcess][cost time] :0.094ms
[vgg16][info][Infere][cost time] :8.319ms
[vgg16][info][Post][cost time] :1.166ms
[resnet50][info][Infere][cost time] :9.564ms
[resnet50][info][Post][cost time] :0.032ms
[vgg16][info][PreProcess][cost time] :0.09ms
[resnet50][info][PreProcess][cost time] :0.47ms
[vgg16][info][Infere][cost time] :8.127ms
[vgg16][info][Post][cost time] :1.32ms
[resnet50][info][Infere][cost time] :9.051ms
[resnet50][info][Post][cost time] :0.031ms
[vgg16][info][PreProcess][cost time] :0.09ms
[resnet50][info][PreProcess][cost time] :0.503ms
[vgg16][info][Infere][cost time] :8.23ms

6

Hi,

Here are two possible causes for your reference.

1.
The default clock mode is dynamic and will cause the unstable performance.
Please maximize the device performance first.

$ sudo nvpmodel -m 0
$ sudo jetson_clocks

2.
Would you mind to check if there is any non-release memory inside your app?
This can be verified with tegrastats directly.

$ sudo tegrastats

Thanks.