Handling Offline IP Cameras in running pipeline

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) Xavier AGX & Xavier NX
• DeepStream Version 5.0
• JetPack Version 4.4
• TensorRT Version 7.1.3
• Issue Type Question

Good Morning,

I am using the deepstream SDK with python Bindings, I have successfully created a working deepstream pipeline with multiple CCTV camera inputs, we have tuned the parameters to ensure minimal latency and maintain a stable FPS per stream.

The problem is that the System comes to a grinding halt if any of the cameras is disconnected or unstable.

What is the recommended approach for handling dodgy cameras (some are linked with long range radio extenders that can be temperamental in bad weather)?

I have already tried by specify nvstream.batched-push-timeout to be 1/FPS, but it had minimal effect

if a Camera is disconnected each stream drops down to 0.2 FPS so it basically chokes the entire pipeline

Regards Andrew

Can you share your python script? Theoretically, the pipeline will not be impacted even there is one stream broken. We need to know more information to analysis the problem.

okay let me test first with multistream example application and see if I can reproduce it first

Hi Fiona,

I can indeed reproduce the results with deepstream_imagedata-multistream.py

I just ran the deepstream_imagedata-multistream.py application with 3 rtsp sources
python3 deepstream_imagedata-multistream.py rtsp://admin:pass@192.168.0.1:554/profile1 rtsp://admin:pass@192.168.0.2:554/profile1 rtsp://admin:pass@192.168.0.3:554/profile1 test

It works perfectly, however when i disconnect the lan cable of 1 of the cameras all the streams drop to 0.2 FPS
below is some console output (trimmed some unecessary information like obj count)

Seting bufapi_version

Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
In cb_newpad

In cb_newpad

Frame Number= 0 Number of Objects= 0 Vehicle_count= 0 Person_count= 0
*
*
*
Frame Number= 129 Number of Objects= 2 Vehicle_count= 0 Person_count= 2
**********************FPS*****************************************
Fps of stream 1 is  25.8
Frame Number= 129 Number of Objects= 0 Vehicle_count= 0 Person_count= 0
**********************FPS*****************************************
Fps of stream 2 is  25.8
Frame Number= 129 Number of Objects= 0 Vehicle_count= 0 Person_count= 0
**********************FPS*****************************************
Fps of stream 0 is  25.8
Frame Number= 130 Number of Objects= 3 Vehicle_count= 1 Person_count= 2
*
*
*
*
Frame Number= 246 Number of Objects= 0 Vehicle_count= 0 Person_count= 0
**********************FPS*****************************************
Fps of stream 2 is  23.2
Frame Number= 246 Number of Objects= 3 Vehicle_count= 1 Person_count= 2
**********************FPS*****************************************
Fps of stream 1 is  23.2
Frame Number= 246 Number of Objects= 0 Vehicle_count= 0 Person_count= 0
**********************FPS*****************************************
Fps of stream 0 is  23.2
Frame Number= 247 Number of Objects= 3 Vehicle_count= 1 Person_count= 2
*
*
*
*
Vehicle_count= 0 Person_count= 0
Frame Number= 392 Number of Objects= 2 Vehicle_count= 0 Person_count= 2
**********************FPS*****************************************
Fps of stream 1 is  29.0
Frame Number= 392 Number of Objects= 0 Vehicle_count= 0 Person_count= 0
**********************FPS*****************************************
Fps of stream 2 is  29.0
Frame Number= 392 Number of Objects= 0 Vehicle_count= 0 Person_count= 0
**********************FPS*****************************************
Fps of stream 0 is  29.0
Frame Number= 393 Number of Objects= 3 Vehicle_count= 1 Person_count= 2
*
*
<------------------- Removed lan cabel------------------->
*
*
Frame Number= 405 Number of Objects= 2 Vehicle_count= 0 Person_count= 2
**********************FPS*****************************************
Fps of stream 1 is  2.4
Frame Number= 405 Number of Objects= 0 Vehicle_count= 0 Person_count= 0
**********************FPS*****************************************
Fps of stream 0 is  2.4
Frame Number= 406 Number of Objects= 3 Vehicle_count= 1 Person_count= 2
Frame Number= 406 Number of Objects= 0 Vehicle_count= 0 Person_count= 0
Frame Number= 407 Number of Objects= 2 Vehicle_count= 0 Person_count= 2
**********************FPS*****************************************
Fps of stream 1 is  0.2
Frame Number= 407 Number of Objects= 0 Vehicle_count= 0 Person_count= 0
**********************FPS*****************************************
Fps of stream 0 is  0.2
Frame Number= 408 Number of Objects= 3 Vehicle_count= 1 Person_count= 2
Frame Number= 408 Number of Objects= 0 Vehicle_count= 0 Person_count= 0
Frame Number= 409 Number of Objects= 3 Vehicle_count= 1 Person_count= 2
**********************FPS*****************************************
Fps of stream 1 is  0.2
Frame Number= 409 Number of Objects= 0 Vehicle_count= 0 Person_count= 0
**********************FPS*****************************************
Fps of stream 0 is  0.2
*
*
<------------------- Reconnected lan cabel------------------->
*
*
Frame Number= 556 Number of Objects= 0 Vehicle_count= 0 Person_count= 0
**********************FPS*****************************************
Fps of stream 0 is  29.2
Frame Number= 556 Number of Objects= 2 Vehicle_count= 0 Person_count= 2
**********************FPS*****************************************
Fps of stream 1 is  29.2
Frame Number= 557 Number of Objects= 0 Vehicle_count= 0 Person_count= 0
\

Is this not to do with the batch Inference?

Regards Andrew

@Fiona.Chen

Do you have any feedback on this topic?

Regards Andrew

We need to reproduce the problem.

Please try to modify the following code:
streammux.set_property(‘batched-push-timeout’, 4000000)

change it to
streammux.set_property(‘batched-push-timeout’, 40000)

1 Like