About GPU using

Robotics & Edge Computing Jetson Systems Jetson Orin NX
1

Hello, recently I have been deploying the YOLO11S model on Jetson using TensorRT.

The GPU usage of the model is 428M and the memory usage is 157M. It was found that the GPU usage rate was extremely high. The maximum performance mode was enabled, and the GPU reached a frequency of 756 MHz. The occupancy rate fluctuated between 0% and 90%, and it was mostly at 60% (as this is an instantaneous observation value, it may not be entirely accurate).

I wonder why my model is so small, but the GPU utilization rate is so high?

Platform Infor strong text mation:

  • Jetson Orin NX 8G
  • Jetpack 5.1.5
  • TensorRT 8.5.2.2
  • CUDA 11.4
  • cudnn 8.6

Model Information:

  • Model: yolo11s
  • Inference Size: 864*864

Quantization Precision:

  • FP16
  • The floating-point operation volume of the model corresponding to ONNX is 21.3 GFLOPs.

Workflow information:

  • Camera frame rate: 30

Therefore, the computing capacity of this model per second is:

21.3 * 10^9 * 30 / 10^12 = 0.639 TFLOPS

The Jetson Orin NX 8G claims that its INT8 sparse computing power reaches 70 TOPS, and the corresponding FP16 dense computing power can be estimated at 17.5 TFLOPS. My model only occupied 3.65% of its TOPS during operation.

Why is the usage rate so high? How to optimize?

4

Hi,

Please upgrade your device to JetPack 6 first.
We enable the lazy loading from recent CUDA version so the CUDA lib is only loaded when required.

Thanks.

6

Thank you for your prompt reply!

May I ask if Nvidia has already optimized this issue in Jetpack 6? I am currently installing Jetpack 6.2.2 on another Jetson Orin NX 16G board. I will give it a try as soon as possible and provide you with the feedback!
Thank you!

7

Hi,

On JetPack 5, the CUDA library will be loaded at the initial time and consume lots of memory.
We solve this by the lazy loading feature.

Please try it on JetPack 6 to see if this can fix your issue.
Thanks.

9

Since the GPU usage rate cannot directly reflect the TOPS of the model, I would like to know how to calculate the TOPS of the model on the Jetson Orin NX 8G? Is the calculation method I used at the beginning correct?

Platform Infor strong text mation:
Jetson Orin NX 8G
Jetpack 5.1.5
TensorRT 8.5.2.2
CUDA 11.4
cudnn 8.6

Model Information:
Model: yolo11s
Inference Size: 864*864

Quantization Precision:
FP16
The floating-point operation volume of the model corresponding to ONNX is 21.3 GFLOPs.

Workflow information:
Camera frame rate: 30

Therefore, the computing capacity of this model per second is:
21.3 * 10^9 * 30 / 10^12 = 0.639 TFLOPS

10

Hi, I am currently using Jetson Orin NX 8G and TensorRT 8.5.2.2. I want to deploy the VLM model, but due to the low version of TensorRT, I always encounter the problem of INT64 truncation. What should I do? Which models can be directly deployed using TensorRT 8.5.2.2?