Hardware Platform (Jetson / GPU) : NVIDIA A2 • DeepStream Version : 6.4 • JetPack Version (valid for Jetson only) • TensorRT Version : 12.2 • NVIDIA GPU Driver Version (valid for GPU only) : 535.154.05 • Issue Type( questions, new requirements, bugs) • How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
I am testing deepstream_test3.py with 14 RTSP Cameras and after pipeline start it reaches to process 14 streams but after few minutes number of streams goes down one by one
Note : I am using RTSP simulator cvlc
why this happened and how to fix this issue?
output
Resetting source rtsp://10.1.118.105:6011/stream
Warning: gst-resource-error-quark: Could not write to resource. (10): …/gst/rtsp/gstrtspsrc.c(6607): gst_rtspsrc_try_send (): /GstPipeline:pipeline0/GstBin:source-bin-11/GstDsNvUriSrcBin:uri-decode-bin/GstRTSPSrc:src:
Could not send message. (Received end-of-file)
Warning: gst-resource-error-quark: Could not write to resource. (10): …/gst/rtsp/gstrtspsrc.c(9034): gst_rtspsrc_pause (): /GstPipeline:pipeline0/GstBin:source-bin-11/GstDsNvUriSrcBin:uri-decode-bin/GstRTSPSrc:src:
Could not send message. (Received end-of-file)
please set nvurlsrcbin’s rtsp-reconnect-interval property for RTSP reconnection. for example, uri_decode_bin.set_property(“rtsp-reconnect-interval”, 10).
please refer to this topic for improving performance.
OK I set this parameter and it can down and upstreams not not all streams working
I tested with 40 Cams and deepstream_test3.py and ffmpeg tool to convert videos to RTSP streams
Output from script
Frame Number= 2596 Time Stamp= 2024-02-01 15:04:15.123334 Cam ID: 0
Frame Number= 2596 Time Stamp= 2024-02-01 15:04:15.123338 Cam ID: 4
Frame Number= 2562 Time Stamp= 2024-02-01 15:04:15.123340 Cam ID: 7
Frame Number= 2589 Time Stamp= 2024-02-01 15:04:15.123342 Cam ID: 12
Frame Number= 2584 Time Stamp= 2024-02-01 15:04:15.123344 Cam ID: 13
Frame Number= 2581 Time Stamp= 2024-02-01 15:04:15.123346 Cam ID: 14
Frame Number= 2584 Time Stamp= 2024-02-01 15:04:15.123348 Cam ID: 17
Frame Number= 2581 Time Stamp= 2024-02-01 15:04:15.123350 Cam ID: 18
Frame Number= 2591 Time Stamp= 2024-02-01 15:04:15.123352 Cam ID: 19
Frame Number= 1751 Time Stamp= 2024-02-01 15:04:15.123354 Cam ID: 20
Frame Number= 1774 Time Stamp= 2024-02-01 15:04:15.123356 Cam ID: 23
Frame Number= 1725 Time Stamp= 2024-02-01 15:04:15.123358 Cam ID: 24
Frame Number= 1707 Time Stamp= 2024-02-01 15:04:15.123361 Cam ID: 25
Frame Number= 1682 Time Stamp= 2024-02-01 15:04:15.123363 Cam ID: 26
Frame Number= 1669 Time Stamp= 2024-02-01 15:04:15.123365 Cam ID: 27
Frame Number= 1631 Time Stamp= 2024-02-01 15:04:15.123367 Cam ID: 29
Frame Number= 1453 Time Stamp= 2024-02-01 15:04:15.123369 Cam ID: 31
Frame Number= 1419 Time Stamp= 2024-02-01 15:04:15.123372 Cam ID: 32
Frame Number= 1374 Time Stamp= 2024-02-01 15:04:15.123373 Cam ID: 33
Frame Number= 1353 Time Stamp= 2024-02-01 15:04:15.123376 Cam ID: 34
Frame Number= 1335 Time Stamp= 2024-02-01 15:04:15.123379 Cam ID: 35
Frame Number= 1314 Time Stamp= 2024-02-01 15:04:15.123381 Cam ID: 36
Frame Number= 357 Time Stamp= 2024-02-01 15:04:15.123383 Cam ID: 37
Frame Number= 119 Time Stamp= 2024-02-01 15:04:15.123386 Cam ID: 38
Frame Number= 103 Time Stamp= 2024-02-01 15:04:15.123388 Cam ID: 39
No. of stream sources 25
No. objects in batch: 241
Output from ffmpeg container
2024/02/01 14:59:26 INF [RTSP] [conn 10.1.118.105:48266] opened
2024/02/01 14:59:26 INF [RTSP] [session 0979f910] created by 10.1.118.105:48266
2024/02/01 14:59:27 WAR [RTSP] [session cf109d5d] write queue is full
2024/02/01 14:59:27 INF [RTSP] [session 0979f910] is reading from path ‘stream11’, with TCP, 1 track (H264)
2024/02/01 14:59:28 WAR [RTSP] [session cf109d5d] write queue is full
2024/02/01 14:59:28 WAR [RTSP] [session 13af942b] write queue is full
2024/02/01 14:59:29 WAR [RTSP] [session cf109d5d] write queue is full
2024/02/01 14:59:29 WAR [RTSP] [session 13af942b] write queue is full
2024/02/01 14:59:30 WAR [RTSP] [session cf109d5d] write queue is full
2024/02/01 14:59:30 WAR [RTSP] [session 13af942b] write queue is full
2024/02/01 14:59:31 INF [RTSP] [session e0c29e84] destroyed: torn down by 10.1.118.105:53240
2024/02/01 14:59:31 INF [RTSP] [conn 10.1.118.105:53240] closed: EOF
2024/02/01 14:59:31 INF [RTSP] [conn 10.1.118.105:37330] opened
2024/02/01 14:59:31 INF [RTSP] [session b69f1bed] created by 10.1.118.105:37330
2024/02/01 14:59:31 WAR [RTSP] [session cf109d5d] write queue is full
2024/02/01 14:59:31 WAR [RTSP] [session 13af942b] write queue is full
2024/02/01 14:59:32 INF [RTSP] [session af528c60] destroyed: torn down by 10.1.118.105:53274
2024/02/01 14:59:32 INF [RTSP] [conn 10.1.118.105:53274] closed: EOF
2024/02/01 14:59:32 INF [RTSP] [conn 10.1.118.105:37340] opened
2024/02/01 14:59:32 INF [RTSP] [session b69f1bed] is reading from path ‘stream5’, with TCP, 1 track (H264)
2024/02/01 14:59:32 INF [RTSP] [session a164eebe] created by 10.1.118.105:37340
2024/02/01 14:59:32 WAR [RTSP] [session cf109d5d] write queue is full
2024/02/01 14:59:32 WAR [RTSP] [session 13af942b] write queue is full
I can see a lot of message mention queue is full which queue this message refer and how to fix this issue?
RTSP server is Gstreamer opensource code. please refer to this link. RTSP server uses rtp to send the encoded frames. please refer to the code and doc.