Inference Time Scales Linearly With Batch Size

Robotics & Edge Computing Jetson Systems Jetson AGX Xavier
1

I am trying to deploy YOLOv7 model onto NVIDIA Xavier AGX, and everything works well with batch size 1. I utilize dynamic batch and deploy models on batch>1 images, and the performance is nominal. However, the time it takes to do inference scales with batch size.

trt_inference.txt (4.8 KB)

3

Hi,

Could you recheck the log since the output is not expected.

GPU utilization can be monitored by the GR3D_FREQ score from tegrastats.
In your experiment, only batchsize=8 runs with full GPU resources. Batchsize=1~4 only takes around 1%~4x%.
Please check if you do capture the inference period correctly.

For batching optimization, please find below topic for introduction and guidance:

Thanks.

4

GR3D_FREQ seems to oscillate a lot, at batch 4, between 4% to 50%. Sometimes, the values spike to 99% but also go all the way down to 0%. When I use jtop, these values seem to be a lot more stable.

What is the best way to record GPU utilization? For reference, I use TensorRT 7.1.3 and enqueueV2() to run inference. My engine is optimized for running batches of 4, with min=1 and max=8.

5

Hi,

Have you maximized the device’s performance?

$ sudo nvpmodel -m 0
$ sudo jetson_clocks

It sounds like your inference task is more IO tasks rather than computational jobs (GPU idle sometimes).
Could you profile the pipeline with Nsight System and share the result with us?

More, the latest TensorRT for Xavier is 8.5.
It’s recommended to upgrade your device and try it again.

Thanks.

6

I was unable to get my code working with Nsight System. I profiled it using with nvprof, but I am unable to upload the results. How can i send them to you?

7

Hi,

Is it limited by the file type or file size?
If yes, could you put it on some online DRIVE and share the link here?

Thanks.

8
9

Hi,

Do you run this with trtexec or custom software?
There is some idle period in between. Do you keep enqueue jobs to TensorRT?

image
image3614×1316 297 KB

More, could you try if increasing the CUDA queue can help?

$ export CUDA_DEVICE_MAX_CONNECTIONS=32

Thanks.

11

That was my custom software. I tried to use trtexec on my ONNX model file, and I test out the different batch_sizes=1, 2, 4, 8

Here are the output logs.
report_bs=1.txt|attachment (1.1 MB)
report_bs=2.txt|attachment (1.1 MB)
report_bs=4.txt|attachment (1.1 MB)
report_bs=8.txt|attachment (1.1 MB)