Changing uridecodebin to nvurisrcbin creates jitter at pipeline RTSP output

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

• Hardware Platform (Jetson / GPU) Jetson Xavier AGX
• DeepStream Version 6.3.0
• JetPack Version (valid for Jetson only) 5.1
• TensorRT Version 8.5.2.2

I had problems with a pipeline freezing (Pipeline freeze?) and we came to a solution by changing uridecodebin to nvurisrcbin as pipeline source bin. It works fine when my target sink is a nv3dsink, but when the target is a udpsink there is a lot of jitter, mainly at the bounding boxes detections.

At first, I inspected my pipeline since it has many customization, but then I moved on to a more simple sandbox. Thus, I started using the deepstream_python_apps/apps/deepstream-rtsp-in-rtsp-out/deepstream_test1_rtsp_in_rtsp_out.py at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub . This sample works pretty well using the uridecodebin, its pipeline looks like this:

[uridecodebin] → nvstreammux → nvinfer → nvvideoconvert → nvmultistreamtiler → nvosd → nvvideoconvert → caps → nvv4l2h264enc → rtph264pay → udpsink

When I changed the uridecodebin to nvurisrcbin I noticed some jitter, but the mosaic image produce by tiler disguised it a little bit since the images of each camera becames to small. Therefore, in order to view only one source (for instance number 2, from 0,1,2,3) I changed the tiler (2x2 cameras) to a nvstreamdemux and linked the source pad 2 to the nvosd. Then the jitter became quite visible. The altered pipeline goes like this:

[nvurisrcbin] → nvstreammux → nvinfer → nvvideoconvert → nvstreamdemux [2] → nvosd → nvvideoconvert → caps → nvv4l2h264enc → rtph264pay → udpsink

This is the altered deepstream_test1_rtsp_in_rtsp_out.py.txt (15.4 KB). The config file didn’t changed but here it is dstest1_pgie_nvinfer_config.txt (3.1 KB). I ran the script by doing:
python3 deepstream_test1_rtsp_in_rtsp_out.py -i rtsp://xxxx:xxxx@10.21.45.19 rtsp://xxxx:xxxx@10.21.45.19 rtsp://xxxx:xxxx@10.21.45.19 rtsp://xxxx:xxxx@10.21.45.19
… and the output RTSP is catched by:
nvgstplayer-1.0 -i rtsp://10.21.45.31:8554/ds-test

Does anyone can give me a tip?

Thanks in advance.

3

Could you attach the recorded video with the jitter issue?

4

Sure, this is a sample with the jitter

5

Could you try to just use 1 source and deepstream-rtsp-in-rtsp-out without any change to repro this issue?

Did you just change this uri_decode_bin = Gst.ElementFactory.make(“uridecodebin”, “uri-decode-bin”) to uri_decode_bin = Gst.ElementFactory.make(“nvurisrcbin”, “uri-decode-bin”)?

6

Could you try to just use 1 source and deepstream-rtsp-in-rtsp-out without any change to repro this issue?

The original file, with no editing, runs fine.

Did you just change this uri_decode_bin = Gst.ElementFactory.make(“uridecodebin”, “uri-decode-bin”) to uri_decode_bin = Gst.ElementFactory.make(“nvurisrcbin”, “uri-decode-bin”)?

Yes, the problem occurs when I changed uridecodebin to nvurisrcbin. The first message from this post contains my script with changes marked with a “#CHANGED”

    #CHANGED uri_decode_bin = Gst.ElementFactory.make("uridecodebin", "uri-decode-bin")
    #CHANGED - start
    uri_decode_bin = Gst.ElementFactory.make("nvurisrcbin", "uri-source-bin")
    uri_decode_bin.set_property("rtsp-reconnect-interval", 10)
    uri_decode_bin.set_property("latency", 100)
    uri_decode_bin.set_property("file-loop", 1)
    uri_decode_bin.set_property("select-rtp-protocol", 4)
    uri_decode_bin.set_property("drop-frame-interval", 2)
    #CHANGED - stop
7

OK. Could you take the following tests to narrow it down?

  1. do not set the drop-frame-interval parameter
  2. set the “latency” to 2000
  3. change the rtsp out to file out by reffering to the FAQ
8

I made the changes you suggested (1, 2 and 3), but the jitter is present at the generated file. I believe the Forum does not allow to upload .h264 file, so, this is an snapshot.

vlcsnap-2024-06-12-10h27m23s842
vlcsnap-2024-06-12-10h27m23s8421920×1080 158 KB

9

OK, Thanks. Judging from this image, all the bboxes also have jitter problems. So it could be the encoder caused this problem.
Could you try to display the video directly on the monitor like our deepstream_test_1.py? You can also try to tune the parameters of the encoder by referring to the Gst-nvvideo4linux2 plugin encoder gst properties.

10

By sending the nvvideoconvert directly to the nv3dsink, just like deepstream_test_1.py, I get no jitter.

During the last days I have made several attempts:

A) Changed the encoder from H264 to H265: no improvement

B) With the encoder, I changed these properties:
idrinterval: reduced from 256 to 15, I got a slight and marginal improvement
iframeinterval: always keeping it lower than idrinterval, no improvement
bitrate: when reducing it from 4,000,000 to 40,000 the resolution reduces a lot and it is difficult to assert that the jitter is gone because it is too much blur, but it may be gone …

C) Moreover, if I inspect nvgstplayer output, I see some metadata about the stream that is present at the original camera RTSP stream, and is not present at the RTSP stream produced by the pipeline. Those metadata is in bold at the following log:

ubuntu@ubuntu:~/EdgeServer$ nvgstplayer-1.0 -i rtsp://xxx:xxx@10.21.45.19:554
** (nvgstplayer-1.0:115944): WARNING **: 10:49:51.856: <bus_call:1310> failed to query duration in time
** Message: 10:49:54.129: <cb_newpad:2077> creating the pipe for “video/x-h265, stream-format=(string)byte-stream, alignment=(string)au, parsed=(boolean)true, width=(int)1920, height=(int)1080, framerate=(fraction)25/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, pixel-aspect-ratio=(fraction)1/1, profile=(string)main, tier=(string)main, level=(string)4”
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 279
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 279

ubuntu@ubuntu:~/EdgeServer$ nvgstplayer-1.0 -i rtsp://10.21.45.31:8554/stream2
** (nvgstplayer-1.0:116496): WARNING **: 11:05:26.820: <bus_call:1310> failed to query duration in time
** Message: 11:05:35.251: <cb_newpad:2077> creating the pipe for “video/x-h265, stream-format=(string)byte-stream, alignment=(string)au, parsed=(boolean)true, width=(int)1920, height=(int)1080, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, profile=(string)main, tier=(string)main, level=(string)4”
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 279
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 279

D) Finnally, I used ffprobe to get more information on the output stream, and it complains that PPS is out of range.

fmello@turing:~$ ffprobe -i rtsp://10.21.45.31:8554/stream2
ffprobe version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2007-2021 the FFmpeg developers
built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)
configuration: --prefix=/usr --extra-version=0ubuntu0.22.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
libavutil 56. 70.100 / 56. 70.100
libavcodec 58.134.100 / 58.134.100
libavformat 58. 76.100 / 58. 76.100
libavdevice 58. 13.100 / 58. 13.100
libavfilter 7.110.100 / 7.110.100
libswscale 5. 9.100 / 5. 9.100
libswresample 3. 9.100 / 3. 9.100
libpostproc 55. 9.100 / 55. 9.100
[hevc @ 0x58746322e280] PPS id out of range: 0
Last message repeated 1 times
[hevc @ 0x58746322e280] Error parsing NAL unit #0.
[hevc @ 0x58746322e280] PPS id out of range: 0
Last message repeated 1 times
[hevc @ 0x58746322e280] Error parsing NAL unit #0.
[rtsp @ 0x587463228ec0] max delay reached. need to consume packet
[rtsp @ 0x587463228ec0] RTP: missed 28 packets
[hevc @ 0x58746322e280] PPS id out of range: 0
Last message repeated 1 times
[hevc @ 0x58746322e280] Error parsing NAL unit #0.
[hevc @ 0x58746322e280] PPS id out of range: 0
Last message repeated 1 times
[hevc @ 0x58746322e280] Error parsing NAL unit #0.
[hevc @ 0x58746322e280] PPS id out of range: 0
Last message repeated 1 times
[hevc @ 0x58746322e280] Error parsing NAL unit #0.
[hevc @ 0x58746322e280] PPS id out of range: 0
Last message repeated 1 times
[hevc @ 0x58746322e280] Error parsing NAL unit #0.
[hevc @ 0x58746322e280] PPS id out of range: 0
Last message repeated 1 times
[hevc @ 0x58746322e280] Error parsing NAL unit #0.
[hevc @ 0x58746322e280] PPS id out of range: 0
Last message repeated 1 times
[hevc @ 0x58746322e280] Error parsing NAL unit #0.
[hevc @ 0x58746322e280] PPS id out of range: 0
Last message repeated 1 times
[hevc @ 0x58746322e280] Error parsing NAL unit #0.
[hevc @ 0x58746322e280] PPS id out of range: 0
Last message repeated 1 times
[hevc @ 0x58746322e280] Error parsing NAL unit #0.
[hevc @ 0x58746322e280] PPS id out of range: 0
Last message repeated 1 times
[hevc @ 0x58746322e280] Error parsing NAL unit #0.
[hevc @ 0x58746322e280] PPS id out of range: 0
Last message repeated 1 times
[hevc @ 0x58746322e280] Error parsing NAL unit #0.
[hevc @ 0x58746322e280] PPS id out of range: 0
Last message repeated 1 times
[hevc @ 0x58746322e280] Error parsing NAL unit #0.
[hevc @ 0x58746322e280] PPS id out of range: 0
Last message repeated 1 times
[hevc @ 0x58746322e280] Error parsing NAL unit #0.
[hevc @ 0x58746322e280] PPS id out of range: 0
Last message repeated 1 times
[hevc @ 0x58746322e280] Error parsing NAL unit #0.
[rtsp @ 0x587463228ec0] max delay reached. need to consume packet
[rtsp @ 0x587463228ec0] RTP: missed 11 packets
[hevc @ 0x58746322e280] PPS id out of range: 0
Last message repeated 1 times
[hevc @ 0x58746322e280] Error parsing NAL unit #0.
[hevc @ 0x58746322e280] PPS id out of range: 0
Last message repeated 1 times
[hevc @ 0x58746322e280] Error parsing NAL unit #0.
[rtsp @ 0x587463228ec0] max delay reached. need to consume packet
[rtsp @ 0x587463228ec0] RTP: missed 68 packets
Input #0, rtsp, from ‘rtsp://10.21.45.31:8554/stream2’:
Metadata:
title : Session streamed with GStreamer
comment : rtsp-server
Duration: N/A, start: 0.160000, bitrate: N/A
Stream #0:0: Video: hevc (Main), yuv420p(tv), 1920x1080, 12.50 tbr, 90k tbn, 90k tbc

11

OK. So We’re pretty sure it’s a problem with the encoder. It may be that there is something special about your stream that requires special configuration.

Could you take the following tests?
1.Use ffprobe to get the format of the original rtsp source
2.Try tuning the values of the following parameters

control-rate  idrinterval iframeinterval preset-level profile
12

All comments concern nvv4l2h265enc:

  1. By setting control-rate=0 (variable_bitrate, the default is constant_bitrate) all jitter disappeared.
    The preset-level was already set to1 in the original script.
    And I didn’t made any changes to idrinterval iframeinterval profile, using the default values.

Further improvements I made, that may be useful for future developers

  1. When using ffprobe to inspect my output stream I got few metadata about it:
    Stream #0:0: Video: hevc, none, 100 tbr, 90k tbn, 90k tbc
    So I set insert-vui=1 in order to add video information usability to the stream and get a more polite stream.
    Stream #0:0: Video: hevc (Main), yuv420p(tv, gbr/reserved/reserved), 1920x1080, 25 fps, 12.50 tbr, 90k tbn, 25 tbc
13

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.