Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU)
GPU (A5000)
• DeepStream Version
Deepstream 6.0, 6.0.1, 6.2
• JetPack Version (valid for Jetson only)
• TensorRT Version
TensorRT 8.0.1
• NVIDIA GPU Driver Version (valid for GPU only)
Version 515.67
• Issue Type( questions, new requirements, bugs)
Potential Bug
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
We are using deepstream-app sample as a baseline for our product. Below is the configuration file we are using for the system :
[application]
enable-perf-measurement=1
perf-measurement-interval-sec=30
[source1]
enable=1
type=6 # custom source type to accept h264 stream through tcp
uri=127.0.0.1
gpu-id=0
cudadec-memtype=2
nvbuf-memory-type=3
num-sources=100
drop-frame-interval=10
intra-decode-enable=0
low-latency-mode=0
num-extra-surfaces=0
[sink0]
enable=1
type=6
sync=0
source-id=0
msg-conv-config=config_msgconv.txt
msg-conv-payload-type=258
msg-broker-proto-lib=/opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so
msg-broker-conn-str=localhost;9092;test_topic
topic=test_topic
msg-broker-config=/opt/nvidia/deepstream/deepstream-6.0.appliance/sources/apps/sample_apps/deepstream-test4/cfg_kafka.txt
new-api=0
disable-msgconv=0
[sink2]
enable=1
type=1
sync=0
source-id=0
gpu-id=0
nvbuf-memory-type=3
[streammux]
gpu-id=0
live-source=1
batch-size=16
batched-push-timeout=30000
width=1280
height=720
enable-padding=0
nvbuf-memory-type=3
[primary-gie]
[tracker]
[secondary-gie1]
[secondary-gie2]
[secondary-gie3]
[secondary-gie4]
[secondary-gie5]
[ds-example]
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)
Description of environment setting:
- A5000 GPU, 32-core CPU (Intel(R) Xeon(R) Gold 6234 CPU @ 3.30GHz), 64GB RAM
- Use TCP stream and tcp_server src plugin
- 100 channels of H.264 streams HD at 30FPS
- Use drop-frame-interval of 10 → 2.7-3 FPS per source
- Primary detection model + multiple secondary classification models + ds_example post-processing + Kafka broker
Description of issue
The system is stable and runs without any issues for a couple of hours or sometimes for multiple hours with CPU usage of ~800-900% and ~14GB RAM. However, at some point the CPU usage shoots up to >2400~2500% within 3 seconds and stays at this level until system crashes and CPU usage goes down to 0%. Afterwards the TCP server is unable to connect to Deepstream sockets and send any frames. The behavior of Deepstream-app after this is two-fold : 1) it just hangs indefinitely, with perf_cb showing 0FPS for all channels 2) crashes completely.
When testing with 32 channels and lower drop-frame-interval, such behavior was never discovered, with the system running in a stable fashion for multiple days without outside intervention.
When analyzing the thread-wise CPU usage using htop, we discovered that nvv4l2decoder was using the vast majority of CPU resources upon crash. So, we believe nvv4l2decoder plugin to be the culprit in this issue.
We tested by adjusting num-extra-surfaces attribute of the plugin and found that the higher the num-extra-surfaces value is the faster the system crashes. However, even setting num-extra-surfaces to 0 does not eliminate CPU 100% problem and the system still crashes within 10-20 hours.
Upon searching for similar issues across this forum, we found a somewhat similar issue that had happened in an older version of Deepstream (version 4.0.1) for Jetson. However, we could not find any ideas or fixes that would apply in case of the most recent versions of dGPU Deepstream.
We can upload the GST_DEBUG_LEVEL = 4 logs as a separate file if needed.
