Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU) NX • DeepStream Version 6.1 • JetPack Version (valid for Jetson only) • TensorRT Version 8.4.0 • NVIDIA GPU Driver Version (valid for GPU only) 11.4 • Issue Type( questions, new requirements, bugs) bugs
I am using deepstream 3d action recognition app samples. I got memory leak when model start inference lead to out of memory and app start freezing. I have to reboot immediately or unplug the source from board. I tried both example model created by TAO and my custom model converted by onnx and got same problem.
Hi @samvdh , how do you confirm the memory leak?
Did you just run our demo without any change?
Could you attach the log when the error happened? Thanks
Hi, thank you for your response. I changed a little bit, I haved change from 4 stream batch to 1 stream. The app start to freezing after first few seconds. With jtop I saw memory continuous increasing significantly until 7g. I tried deepstream_test1 and it worked normally on my jetson NX. I attach the images of log here, without and with debug flag. My nx cannot connect to internet right now so I only can capture image like this. Sorry for this inconvenience.
So is it ok when you run this demo with our 4 demo stream source?
Did you just change the config file from our 4 stream sources to your stream source?
Could you attach your stream source to us?
Could you help to dump the memory log when the demo is running by referring the link below: https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/14
Hi, It got freezing when run with 4 stream source as well so I changed to 1 stream. I use default stream source which is sample_run.mov, I have tried other stream sources and still got problem. I followed your link and put logs here, this time I run with default config with 4 default stream sources.
OK，Thanks. There seems to be a memory leak from the log. We’ll run the demo in our NX env.
Please reconfirm the environment:
1.run the deepstream-3d-action-recognition demo without any change
We can not repro your freeze issue. can you specify your device RAM size? you can check by df. the VmRSS increased to 2G in one minutes is reasonable, please check the mem states running longer than 30 minutes.
nvidia@ubuntu:/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-3d-action-recognition$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mmcblk0p1 30031808 26926404 1719092 94% /
Hi, thank you for your verification. My jetson nx have 7G RAM. My jetson nx got freezing after few second and I have to reboot system or unplugging the board. I attach the video show progress that my system got freezing, RAM usage increasing rapidly until reach full size, hope that give you more information.
Hi, I don’t run other process. I will check again to make sure that I don’t miss any detail. Add another information my deepstream on NX install with jetpack 5.0.1
Hi, I used fakesink(by set fakesink=1 in config file) but it still get freeze. I may find out the problem, I run this app on 2080ti in deepstream docker and it worked normally. My jetson NX is 8G ram version, the problem maybe solve when I run with 16G ram NX. Can you tell me which NX type your are using? Thank you for your support.
Hi, I have just fixed the config and the problem is solved, I changed from network-type=1 to network-type=100 in config_infer_primary_3d_action.txt and system run without freezing.
With network-type=100 the output is like this
And with network-type=1 the output is like this, the logs is printed rapidly and non-stop, it look like the inference step is in an endless loop and make image sequence cached until full of memory.
If you set network-type=100, it will skip the postprocess. So it’s postprocess that causes this problem. Since we cannot duplicate this problem in our same env, you can help to debug it in your own env.
1.make sure that no code has been changed
2.You can run other classifier model to see if it is the classifier problem
3.You can dump the fd status with the cli below when the app runs:
Hi, sorry for late reply, my device is not available recently, I will test again when I get the device. Thank you for your support. p/s the predict output seem to correct without post processing
