Changing uridecodebin to nvurisrcbin creates jitter at pipeline RTSP output

• Hardware Platform (Jetson / GPU) Jetson Xavier AGX
• DeepStream Version 6.3.0
• JetPack Version (valid for Jetson only) 5.1
• TensorRT Version 8.5.2.2

I had problems with a pipeline freezing (Pipeline freeze?) and we came to a solution by changing uridecodebin to nvurisrcbin as pipeline source bin. It works fine when my target sink is a nv3dsink, but when the target is a udpsink there is a lot of jitter, mainly at the bounding boxes detections.

At first, I inspected my pipeline since it has many customization, but then I moved on to a more simple sandbox. Thus, I started using the deepstream_python_apps/apps/deepstream-rtsp-in-rtsp-out/deepstream_test1_rtsp_in_rtsp_out.py at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub . This sample works pretty well using the uridecodebin, its pipeline looks like this:

[uridecodebin] → nvstreammux → nvinfer → nvvideoconvert → nvmultistreamtiler → nvosd → nvvideoconvert → caps → nvv4l2h264enc → rtph264pay → udpsink

When I changed the uridecodebin to nvurisrcbin I noticed some jitter, but the mosaic image produce by tiler disguised it a little bit since the images of each camera becames to small. Therefore, in order to view only one source (for instance number 2, from 0,1,2,3) I changed the tiler (2x2 cameras) to a nvstreamdemux and linked the source pad 2 to the nvosd. Then the jitter became quite visible. The altered pipeline goes like this:

[nvurisrcbin] → nvstreammux → nvinfer → nvvideoconvert → nvstreamdemux [2] → nvosd → nvvideoconvert → caps → nvv4l2h264enc → rtph264pay → udpsink

This is the altered deepstream_test1_rtsp_in_rtsp_out.py.txt (15.4 KB). The config file didn’t changed but here it is dstest1_pgie_nvinfer_config.txt (3.1 KB). I ran the script by doing:
python3 deepstream_test1_rtsp_in_rtsp_out.py -i rtsp://xxxx:xxxx@10.21.45.19 rtsp://xxxx:xxxx@10.21.45.19 rtsp://xxxx:xxxx@10.21.45.19 rtsp://xxxx:xxxx@10.21.45.19
… and the output RTSP is catched by:
nvgstplayer-1.0 -i rtsp://10.21.45.31:8554/ds-test

Does anyone can give me a tip?

Thanks in advance.

Could you attach the recorded video with the jitter issue?

Sure, this is a sample with the jitter

Could you try to just use 1 source and deepstream-rtsp-in-rtsp-out without any change to repro this issue?

Did you just change this uri_decode_bin = Gst.ElementFactory.make(“uridecodebin”, “uri-decode-bin”) to uri_decode_bin = Gst.ElementFactory.make(“nvurisrcbin”, “uri-decode-bin”)?

Could you try to just use 1 source and deepstream-rtsp-in-rtsp-out without any change to repro this issue?

The original file, with no editing, runs fine.

Did you just change this uri_decode_bin = Gst.ElementFactory.make(“uridecodebin”, “uri-decode-bin”) to uri_decode_bin = Gst.ElementFactory.make(“nvurisrcbin”, “uri-decode-bin”)?

Yes, the problem occurs when I changed uridecodebin to nvurisrcbin. The first message from this post contains my script with changes marked with a “#CHANGED

    #CHANGED uri_decode_bin = Gst.ElementFactory.make("uridecodebin", "uri-decode-bin")
    #CHANGED - start
    uri_decode_bin = Gst.ElementFactory.make("nvurisrcbin", "uri-source-bin")
    uri_decode_bin.set_property("rtsp-reconnect-interval", 10)
    uri_decode_bin.set_property("latency", 100)
    uri_decode_bin.set_property("file-loop", 1)
    uri_decode_bin.set_property("select-rtp-protocol", 4)
    uri_decode_bin.set_property("drop-frame-interval", 2)
    #CHANGED - stop

OK. Could you take the following tests to narrow it down?

  1. do not set the drop-frame-interval parameter
  2. set the “latency” to 2000
  3. change the rtsp out to file out by reffering to the FAQ

I made the changes you suggested (1, 2 and 3), but the jitter is present at the generated file. I believe the Forum does not allow to upload .h264 file, so, this is an snapshot.