Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU)
Jetson Orin Nano
• DeepStream Version
Deepstream 7
• JetPack Version (valid for Jetson only)
Jetpack 6
• TensorRT Version
libnvinfer-dev 8.6.2.3-1+cuda12.2
• NVIDIA GPU Driver Version (valid for GPU only)
nvidia-smi: NVIDIA-SMI 540.3.0 Driver Version: N/A CUDA Version: 12.2
• Issue Type( questions, new requirements, bugs)
Bug
Our device does inference on roadways, and with the new jetpack and L4T versions, we have noticed that long running processes will continue to grow the kmalloc-128 in the slab memory until failure.
The easiest way to test and reproduce this is to use the deepstream-app config_infer_primary.txt file that is installed with deepstream:
/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt
Using a simple mp4 recording of the roadway roughly 1.5 minutes long and looping it through a pretty simple pipeline with the inference configured in the sample file provided above gives roughly 50mb/hour leak on the slab kmalloc-128 as shown by running slabtop.
gst-launch-1.0 multifilesrc location=/home/omnisight/timeless_file.mp4 loop=true ! decodebin ! nvvideoconvert ! mux.sink_0 nvstreammux name=“mux” width=640 height=480 batch-size=1 gpu-id=0 live-source=true ! nvinfer gpu-id=0 unique-id=1 config-file-path=“config_infer_primary.txt” ! progressreport update-freq=1 ! fakesink
We ran this for various hours and the result is always the same… the slab will continue to grow until the system crashes from OOM.
Restarting nvargus-daemon or the gst pipeline does NOT free the memory.
If we take out the nvinfer, the leak does not appear to be there.
Following is the leak progress:
Timestamp,OBJS,ACTIVE,USE,OBJ_SIZE,SLABS,OBJSLAB,CACHE_SIZE,NAME
2024-09-17 22:22:55,83328,82960,99%,0.12K,2604,32,10416K,kmalloc-128
2024-09-17 22:43:37,90816,90595,99%,0.12K,2838,32,11352K,kmalloc-128
2024-09-18 01:44:49,2670272,2670272,100%,0.12K,83446,32,333784K,kmalloc-128
2024-09-18 07:36:32,7803392,7803391,99%,0.12K,243856,32,975424K,kmalloc-128
2024-09-18 15:05:43,14345024,14344804,99%,0.12K,448282,32,1793128K,kmalloc-128
2024-09-18 16:19:38,15423008,15422670,99%,0.12K,481969,32,1927876K,kmalloc-128
Not sure if this is related to:
Is there any support that Nvidia can provide?