Deepstream primary inference stops after a random amount of time

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 5.1
• JetPack Version (valid for Jetson only)
• TensorRT Version 7.2.3
• NVIDIA GPU Driver Version (valid for GPU only) 460+
• Issue Type( questions, new requirements, bugs) Question

Hi,

I am currently running a facial recognition system from multiple RTSP cameras. My pipeline is as follows:

SrcBin (RTSP->Decode) X 4 → Streammux → Nvinfer( primary(Face Detector) , secondary(Face classifier)) → NvVidconv

The issue I am facing is that after a random amount of time the processing stops and it doesn’t throw an error as well because the loop will exit if an error message is thrown, it is also not recieving a EOS signal. It just stops reading frames.

I am printing frame number from a probe to the sink of the Face Detector.

Is there a way to find out what the error actually is?

It depends. Gstreamer has provided some tools for debugging. Basic tutorial 11: Debugging tools

Hey Fiona

I used the GST_DEBUG at 3 and I still don’t see any errors when it stops. The only recurring warning is this:

2021-07-14T12:41:58: 0:01:43.798095034 e[331m 666e[00m 0x7f5c04006cf0 e[36mINFO e[00m e[00m rtpsource rtpsource.c:1155:update_receiver_stats:e[00m duplicate or reordered packet (seqnr 1566, expected 1568)

The warning is OK.

No one can tell the reason since there is no useful clue for this issue.

Going through the logs more I also find this warning, is this okay?

2021-07-14T11:00:08: 0:29:16.055217568 e[331m79308e[00m 0x7fdd78004770 e[33;01mWARN e[00m e[00m v4l2videodec gstv4l2videodec.c:1259:gst_v4l2_video_dec_loop:e[00m Decoder is producing too many buffers

Futher information while testing

So it seems that the rtsp streams are still incoming

Following is the output of the logs from the probe attached to the face_detector part of the pipeline:
2021-07-14T13:58:28: Frame Number=47330 Number of Objects=0 Person_count=0 Face Count=0
2021-07-14T13:58:28: Frame Number=46811 Number of Objects=0 Person_count=0 Face Count=0

And here is the output of the debug logs:

2021-07-14T14:15:29: 1:35:14.493543374 e[331m 666e[00m 0x7f5c04006cf0 e[36mINFO e[00m e[00m rtpsource rtpsource.c:1155:update_receiver_stats:e[00m duplicate or reordered packet (seqnr 34587, expected 34589)
2021-07-14T14:15:33: 1:35:18.947913708 e[331m 666e[00m 0x7f5bf8005a30 e[36mINFO e[00m e[00m rtpsource rtpsource.c:1155:update_receiver_stats:e[00m duplicate or reordered packet (seqnr 46319, expected 46321)
2021-07-14T14:15:34: 1:35:19.351853481 e[331m 666e[00m 0x7f5bf8005a30 e[36mINFO e[00m e[00m rtpsource rtpsource.c:1155:update_receiver_stats:e[00m duplicate or reordered packet (seqnr 46328, expected 46330)
2021-07-14T14:15:34: 1:35:19.649634909 e[331m 666e[00m 0x7f5bf8005a30 e[36mINFO e[00m e[00m rtpsource rtpsource.c:1155:update_receiver_stats:e[00m duplicate or reordered packet (seqnr 46332, expected 46334)
2021-07-14T14:15:35: 1:35:20.738998852 e[331m 666e[00m 0x7f5bf8005a30 e[36mINFO e[00m e[00m rtpsource rtpsource.c:1155:update_receiver_stats:e[00m duplicate or reordered packet (seqnr 46397, expected 46399)
2021-07-14T14:15:36: 1:35:20.951943525 e[331m 666e[00m 0x7f5bf8005a30 e[36mINFO e[00m e[00m rtpsource rtpsource.c:1155:update_receiver_stats:e[00m duplicate or reordered packet (seqnr 46403, expected 46405)
2021-07-14T14:15:36: 1:35:21.049557891 e[331m 666e[00m 0x7f5bf8005a30 e[36mINFO e[00m e[00m rtpsource rtpsource.c:1155:update_receiver_stats:e[00m duplicate or reordered packet (seqnr 46405, expected 46407)
2021-07-14T14:15:36: 1:35:21.762961237 e[331m 666e[00m 0x7f5c04006cf0 e[36mINFO e[00m e[00m rtpsource rtpsource.c:1155:update_receiver_stats:e[00m duplicate or reordered packet (seqnr 34699, expected 34701)

As it can be seen that input is still being recieved but it isn’t going into nvinfer

The warnings show that your network status is poor. Some packets transferring failed.

Is there a way to restart the process or wait until the connection becomes stable?

The restart function has been implemented in deepstream-app sample code. Please refer to the code.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.