There appears to be some problem with running cuda-memcheck with any deepstream app. If I run with NvDCF I see the following error:
cuda-memcheck ./deepstream-nvdsanalytics-test file:///opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-nvdsanalytics-sagar/sample_1080p_h264.mp4
========= CUDA-MEMCHECK
Now playing: file:///opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-nvdsanalytics-sagar/sample_1080p_h264.mp4,
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_nvdcf.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is ON
========= Internal Memcheck Error: Initialization failed
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x13e08c]
========= Host Frame:/usr/local/cuda/lib64/libcufft.so.10 [0x3d887a]
========= Host Frame:/usr/local/cuda/lib64/libcufft.so.10 [0x3cb9a0]
========= Host Frame:/usr/local/cuda/lib64/libcufft.so.10 [0x3d7bca]
========= Host Frame:/usr/local/cuda/lib64/libcufft.so.10 [0x3db8cf]
========= Host Frame:/usr/local/cuda/lib64/libcufft.so.10 [0x3dc03a]
========= Host Frame:/usr/local/cuda/lib64/libcufft.so.10 [0x3cf66c]
========= Host Frame:/usr/local/cuda/lib64/libcufft.so.10 [0x3bf16e]
========= Host Frame:/usr/local/cuda/lib64/libcufft.so.10 [0x3f138c]
========= Host Frame:/usr/local/cuda/lib64/libcufft.so.10 [0x37b82]
========= Host Frame:/usr/local/cuda/lib64/libcufft.so.10 [0x38186]
========= Host Frame:/usr/local/cuda/lib64/libcufft.so.10 [0x39cd2]
========= Host Frame:/usr/local/cuda/lib64/libcufft.so.10 (cufftXtMakePlanMany + 0x63a) [0x4d2aa]
========= Host Frame:/usr/local/cuda/lib64/libcufft.so.10 (cufftMakePlanMany64 + 0xfd) [0x4e20d]
========= Host Frame:/usr/local/cuda/lib64/libcufft.so.10 (cufftMakePlanMany + 0x193) [0x4acd3]
========= Host Frame:/opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_nvdcf.so (_ZN5NvDCF10initializeEN2cv5Size_IiEEm + 0x1149) [0x610c9]
========= Host Frame:/opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_nvdcf.so (_ZN12NvMOTContextC1ERK12_NvMOTConfigR20_NvMOTConfigResponse + 0x35f) [0x8688f]
========= Host Frame:/opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_nvdcf.so (NvMOT_Init + 0x3b) [0x87c2b]
========= Host Frame:/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_tracker.so (_ZN13NvTrackerProc18initTrackerContextEv + 0x29b) [0x52c45]
========= Host Frame:/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_tracker.so (_ZN13NvTrackerProc4initERK13TrackerConfig + 0xd1) [0x51627]
========= Host Frame:/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_tracker.so [0x4c3b3]
========= Host Frame:/usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0 [0x44270]
========= Host Frame:/usr/lib/x86_64-linux-gnu/libgstbase-1.0.so.0 [0x44505]
========= Host Frame:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 [0x7a69b]
========= Host Frame:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 (gst_pad_set_active + 0xe6) [0x7b116]
========= Host Frame:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 [0x58f0d]
========= Host Frame:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 (gst_iterator_fold + 0x94) [0x6b874]
========= Host Frame:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 [0x59a16]
========= Host Frame:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 [0x5b95e]
========= Host Frame:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 [0x5bc8f]
========= Host Frame:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 (gst_element_change_state + 0x3e) [0x5dd5e]
========= Host Frame:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 [0x5e499]
========= Host Frame:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 [0x3ba02]
========= Host Frame:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 (gst_element_change_state + 0x3e) [0x5dd5e]
========= Host Frame:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 (gst_element_change_state + 0x325) [0x5e045]
========= Host Frame:/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 [0x5e499]
========= Host Frame:./deepstream-nvdsanalytics-test [0x3bca]
========= Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xe7) [0x21b97]
========= Host Frame:./deepstream-nvdsanalytics-test [0x249a]
=========
[NvDCF] Initialized
Deserialize yoloLayerV3 plugin: yolo_168
Deserialize yoloLayerV3 plugin: yolo_176
If I replace the same with KLT tracker, everything runs fine. Furthermore, I used heaptrack to track for memory leaks and it returns saying that there is 230 MB of memory leak in the deepstream-nvdsanalytics-test application. Is this a known issue? Should I trust the tool, or is it false positive?
• Hardware Platform (Jetson / GPU): T4
• DeepStream Version: 5
• TensorRT Version: 7
• NVIDIA GPU Driver Version (valid for GPU only): 440.33