Does utilising Jetson NX at full load make drivers memory leak?

Graphics / Linux Linux
, , , ,
1

Hardware Platform (Jetson / GPU) - Jetson Orin NX
• DeepStream Version - DS-7.0
• JetPack Version (valid for Jetson only) - JP6.0
• TensorRT Version - 8.6.2.3-1+cuda12.2
• Issue Type - questions
**• How to reproduce the issue ? Run 10 rtsp stream with yolo v9 detector and some post processing

Hi.
I have Jetson NX (8GB RAM and 4GB ZRAM)**

I am running 10 rtsp streams with some post processing in different docker containers. Out of which Deepstream is one of the container as well.
I found all of my docker containers are consuming around 6.4GB RAM (PSS memories) but still checking the jtop, memory goes to full and ZRAM starts increasing gradually and later both gets full.

I found following observations and questions:

  1. I assume rest 1 GB offset is what GPU pinned memory and other kernel related thing? Am i correct?

  2. I checked the behaviour for 4 days ZRAM increases from 0 → 2GB but if i keep it running, it increases further. I am sure that there is no memory leakage in our Post Processing and Deepstream memory goes down and ZRAM increases same time. Attached snapshot is for reference.

  3. I also checked “grep -E ‘MemAvailable|Cached|Slab|SReclaimable|SUnreclaim’ /proc/meminfo“ and found SUnreclaim is increasing gradually even if i restart deepstream docker. In 4 days it went from 349448 kB → 1040744 kB. Does it relate to Driver level memory leakage?

  4. If i reboot the device then memory increment is gradually slow and clean but if i don’t reboot it, after 1 crash the frequency gets increases and both RAM and ZRAM aggressively increases and causing more frequent crashes.

  5. Is it good practice to occasionally reboot the device when the device is under continuous heavy load? or any command is there to make everything clean once docker container stops?

  6. I checked chatGPT and other LLM, they suggested the same and says it’s kernel behaviour which pushes less active pages from RAM to ZRAM and thus even with 80-90% of load it will bound to happen and thus memory overload at some point. Is this true?

  7. What if i run the device under normal load (RAM - 6GB), Still kernel pushes some less active pages to ZRAM but at very slow rate?

Would appreciate to get response on all of above points. It will also help others to determine the true potential of jetson devices.

Few images for reference in which you can see memory profile for a days:
Blue line : Deepstream
Red: ZRAM

image
image1172×610 116 KB

2

Hi.

It’s been long time. Any update or suggestion?

3

Anyone from nvidia can suggest? Or i can close this ticket.

4

You might have better luck postng on the Jetson NX forum.

5

Yes, I would repost this question on the Jetson forum. This forum topic is heavily focused on Nvidia dGPUs.

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.