Inference speed is very slow, while the hardware ability is not been fully utilized

[Mariaa]

Hello,

I am running neural network inference on Jetson Orin NX using Docker containers. My neural network is a regularizing flow with 12 coupling layers. My inference speed is very slow, and the ability of the device has not been fully utilized. The CPU usage rate is about 12%, while the GPU usage rate is about 40%. Could you give me some suggestions to help me improve my inference speed?

PS:

I used – cpus=8-- gpus=all – memory=14g – memory swap=16g to run the container.
Equipment usage:
Before running inference:

After running the inference:

[Mariaa]

I tried running it directly without using Docker. There is no difference in inference speed.

[AastaLLL]

Hi,

It looks like the dynamic frequency is used in your environment.
Please try the below command to lock the process clocks to the maximum first.

$ sudo jetson_clocks

Based on the jtop info, the inference is blocked by other tasks rather than CPU or GPU resources.
Which framework do you use? Is there any preprocessing with OpenCV that can be the bottleneck of the pipeline?

Thanks.