Nvdsanalytics processing time increasing over time

Please provide complete information as applicable to your setup.

• Hardware Platform dGPU
• DeepStream Version 6.2
• TensorRT Version (From DS image)
• NVIDIA GPU Driver Version 525 server
• Issue Type bugs

We are running a proprietary DeepStream app with inference, tracking and analytics. For small deployments the pipeline works fine. However, for a big deployment we noticed a big GPU usage decrease after a few hours. We started profiling the processing time per plugin used and the nvdsanalytics plugin had a continuous increase right from the moment the pipeline started. The plugin takes longer and longer to process buffers, which prevents the nvv4l2decoder plugin to push more batches ( because the limit of 4 ). After the time spent in nvdsanalytics becomes equal to the time spent in nvinfer, the inference is not getting any new batches until the analytics plugin is done processing and pushes the batch forward.

Why is the processing time for nvdsanalytics exponentially increasing? Potential issues we thought about:

  • increasing detection history
  • no cleaning strategy implemented
  • too many detections with tracking ID’s

Pipeline graph

1 Like
  1. please refer to this FAQ for measuring component latency.
  2. nvdsanalytic plugin is opensource in DS6.3. you can add log to check. for example, monitor the time consumption of calling low-level interface.
  3. why is there no detection model in the pipeline?
  1. We do not need help in measuring latency. We already have a remote profiling data gathering system.
  2. The open source gst plugin does not help since the low level library is closed source.
  3. We suspected the deepstream nvinfer plugin at first and we switched it with our custom nvinfer implementation. All our elements are based on a skeleton GstTransform plugin and the inference plugin has “perception_mesh” in the name. However, after we switched the inference plugin the increase did not disappear and we started profiling per element.

Are there any potential reasons for the low level library to pause or have reduced performance?

  1. the FAQ can help to measure the time consumption of every plugin. could you share the gathered data?
  2. could you share the nvdsanalytics cfg? could you narrow down this issue by removing some settings? for example, which analytic has this issue?
  3. can you reproduce this issue on the latest DS6.4?

Solution: we implemented a proper analytics plugin.

We upgraded to 6.4 and the issue persisted. The key factors we noticed were that memory and processing time were constantly increasing. The increase in processing time suggesting that the plugin is iterating over the entire history for every buffer (potentially because using a list or an array instead of a map). Furthermore, the increase in used host memory suggests that the history is not managed and grows indefinitely, which would aggravate the first symptom. The symptoms were visible even when running a pipeline with only one camera on a dGPU.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.