GPU memory usage got by using cudaMemGetInfo() is different with "tegrastats" command

[chien.pv3723]

My device is Jetson AGX Xavier having 32GB memory. I use cudaMemGetInfo() to query gpu memory usage information (source here: Caffe failed with py-faster-rcnn demo.py on TX1 - #10 by AastaLLL),
Output shows:

GPU memory usage: used = 6321.13, free = 25599.4 MB, total = 31920.6 MB
GPU memory usage: used = 6327.77, free = 25592.8 MB, total = 31920.6 MB
GPU memory usage: used = 6333.98, free = 25586.6 MB, total = 31920.6 MB
GPU memory usage: used = 6343.41, free = 25577.2 MB, total = 31920.6 MB
GPU memory usage: used = 6352.66, free = 25567.9 MB, total = 31920.6 MB
GPU memory usage: used = 6361.99, free = 25558.6 MB, total = 31920.6 MB
GPU memory usage: used = 6369.29, free = 25551.3 MB, total = 31920.6 MB

but when i run sudo tegrastats command in the same time, output shows:

RAM 4204/31921MB (lfb 6390x4MB) SWAP 0/15960MB (cached 0MB) CPU [23%@2265,24%@2265,26%@2265,22%@2265,36%@2265,19%@2265,18%@2265,81%@2265] EMC_FREQ 58%@2133 GR3D_FREQ 98%@1377 NVENC 115 NVENC1 115 NVDEC 1190 NVDEC1 1190 VIC_FREQ 65%@409 APE 150 MTS fg 0% bg 14% AO@52.5C GPU@60C Tdiode@56.25C PMIC@50C AUX@50.5C CPU@54.5C thermal@54.35C Tboard@49C GPU 22382/20318 CPU 4385/4172 SOC 8771/8178 CV 0/0 VDDRQ 3175/2931 SYS5V 3702/3563
RAM 4204/31921MB (lfb 6388x4MB) SWAP 0/15960MB (cached 0MB) CPU [23%@1156,22%@1156,31%@1157,26%@1165,20%@1608,19%@1632,19%@1841,83%@1739] EMC_FREQ 58%@2133 GR3D_FREQ 97%@1377 NVENC 115 NVENC1 115 NVDEC 1190 NVDEC1 1190 VIC_FREQ 38%@115 APE 150 MTS fg 0% bg 14% AO@53C GPU@60.5C Tdiode@56.5C PMIC@50C AUX@50.5C CPU@53.5C thermal@54.7C Tboard@49C GPU 22684/20426 CPU 3933/4161 SOC 8771/8205 CV 0/0 VDDRQ 3177/2942 SYS5V 3702/3570
RAM 4209/31921MB (lfb 6386x4MB) SWAP 0/15960MB (cached 0MB) CPU [39%@2265,21%@2265,31%@2265,31%@2265,39%@2265,18%@2265,20%@2265,19%@2265] EMC_FREQ 58%@2133 GR3D_FREQ 98%@1377 NVENC 115 NVENC1 115 NVDEC 1190 NVDEC1 1190 VIC_FREQ 78%@268 APE 150 MTS fg 0% bg 10% AO@53C GPU@60.5C Tdiode@56.5C PMIC@50C AUX@50.5C CPU@54C thermal@54.7C Tboard@49C GPU 22694/20524 CPU 3178/4119 SOC 8778/8230 CV 0/0 VDDRQ 3177/2952 SYS5V 3702/3575

1. Can you explain why there is such a big difference between “GPU memory usage: used = 6321.13” and “RAM 4204/31921MB”? I know both the CPU and the GPU share SoC DRAM memory in tegra device.
2. How i can calculate percentage of GPU and CPU memory usage when i running DeepStream app?
   Thank you very much

[AastaLLL]

Hi,

1. cudaMemGetInfo() returns the available memory amount for GPU.
tegrastats catch the physical memory hardware info.

Some memory is preoccupied with the system and cannot be accessed by GPU.

2. Since Jetson is a shared memory system, we don’t separable the memory usage between CPU and GPU.

Thanks.