DeepStream 6.3 NvTracker NvDCF freezes on first detection

• Hardware Platform (Jetson / GPU): Xavier NX Developer Kit
• DeepStream Version: 6.3
• JetPack Version: 5.1.2
• TensorRT Version: 8.5.2.1
• Issue Type: Bug

We have been upgrading our system from Jetpack 4.6 to Jetpack 5.1.2. Our DeepStream pipeline consists of 2 PGIE/NvInfer followed by a single NvTracker ( ... nvinfer ! queue ! nvinfer ! queue ! nvtracker ! ... ). We notice that the first detection ever (regardless of which PGIE made it) will cause the stream to freeze for 1-7 seconds. After the first detection, there is no more freezing (even after a long period of no detections).

We attached probes to the pads of the elements and were able to determine that the NvTracker is causing the freeze. Upon further debugging, we discovered that disabling the NvDCF ( visualTrackerType: 0 ) in the config, the freeze would go away. We tried tuning the NvDCF and some other parameters to see if anything could help but no success there. My only hypothesis has been that there is some sort of initialization issue in the NvDCF library that occurs on the first detection / tracklet it must process.

We do believe that we have seen this behavior on Jetpack 4.6 (from reviewing our old video logs) where we can see frame jumps / freezes right before the first detection. However, these were always short enough (~<2 seconds) that it was not a major issue. On ground testing with Jetpack 5.1.2 has shown the freezes can last up to 7 seconds which is not suitable for our realtime system.

• How to reproduce the issue ?

I managed to recreate this with the sample app deepstream-test2. Re-encode the sample video sample_walk.mov (from the sample streams) to H264 at 720p. For your convenience, you can download here. Change dstest2_config.yml to use the new video. No other modifications need to be made. Just compile and run the app.

Unlike the other example videos where frame0 has detections and you can’t determine if it’s the stream buffering or a bug causing it to freeze, this video starts off with the first 2-3 seconds containing 0 detections. You should see that as soon as the man walks into view, the stream will freeze for ~1-2 seconds.

Can you upgrate to the latest DeepStream version? Can you have a try the issue on the latest DeepStream? Can you share the nsys profile data of the issue?
$ sudo /opt/nvidia/nsight_systems/nsys profile -t cuda,nvtx,nvmedia,osrt --accelerator-trace=nvmedia --show-output=true --force-overwrite=true --delay=20 --duration=90 --output=%p $APP

Unfortunately, due to other software, we are restricted to Jetpack 5.1.2 / DeepStream 6.3 at the moment. The reproducible example I detailed in my original post at the bottom should work in the latest DeepStream version (assuming no breaking changes in deepstream-test2 sample application) if you want to test on your end.

I’ve generated the nsys report when running with $APP=./deepstream-test2-app dstest2_config.yml. I am sending it to you as a DM because it contains some information we prefer to keep private.

I don’t see any function consumed ~1-2 seconds in your nsys profile. I only see the vpiCreateExtractColorNameFeatures() consumed 168ms.

I’m not sure why it would not show up. However, when I ran the nsys profile on the deepstream-test2 application, the FPS dropped on the display window to ~1 FPS and was difficult to discern a freeze from the slow FPS.

I think your best method to debug this is just run deepstream-test2 application with the H264 encoded video I linked in the original post. We’ve seen this issue on two Jetpack version (4.6, 5.1.2) internal to our program and with Nvidia’s example application and have hundreds of saved videos from deploying the system that all display the initial freeze on first detection.

I tested your video on latest DeepStream. I don’t see freeze on the first detection. Can you share recorded video to show the freeze?

That’s good to hear that it does not appear reproducible on the latest DeepStream; hoping something got fixed along the way. For historical records, is that DeepStream 6.4 or 7.0?

Sorry for the delay. I was able to record the freeze occurring. I had to use my phone because the native screen recorder (Ctrl+Alt+Shift+r) was lagging the system so the issue could not be easily isolated. Link to the video.

In the meantime, since I am working with a C++ pipeline (in my own project), I was able to force the freeze to occur on initial boot by adding a fake detection (NvDsObjectMeta) to the first 3 Buffers (single batch). From testing, 3 was the magic number to prevent the first real detection from freezing (1-2 worked but still had a little freeze on first real detection). I’ll be using this solution until we are able to upgrade to a version of DeepStream that does not have the freezing issue.

I tested your video on DeepStream 7.0. I don’t see the freeze. Can you have a try on DeepStream 7.0?

I cannot try on DeepStream 7.0. The system we are working on must be kept on DeepStream 6.3 / Jetpack 5.1.2 for compatibility with other software.