Gstreamer udp decoding pipeline in Jetson TX2

Autonomous Machines Jetson & Embedded Systems Jetson TX2
#1

Hey everyone!

I’m trying to update a pipeline that works ok in Windows (and nvidia Jetson, just very very slowly) that decodes an udp stream to send it to webrtcbin from using vp8enc/vp8dec to using hardware acceleration and I’m having a lot of issues while doing so, the working cpu pipeline is the following:

pipe="udpsrc multicast-group=224.1.1.1 auto-multicast=true port=5000 ! queue ! application/x-rtp,media=video,clock-rate=90000,encoding-name=VP8,framerate=25/1 !  rtpvp8depay ! vp8dec ! videoconvert ! queue ! vp8enc deadline=1 ! rtpvp8pay ! queue ! capsfilter caps=application/x-rtp,media=video,encoding-name=VP8,payload=97 ! webrtcbin name=sendrecv";

I tried several combinations around the following pipeline (not working) :

pipeline = "udpsrc multicast-group=224.1.1.1 auto-multicast=true port=5000 ! queue ! application/x-rtp,media=video,clock-rate=90000,encoding-name=VP8,framerate=25/1 ! rtpvp8depay ! omxvp8dec ! video/x-raw, format=RGBA ! nvvidconv ! video/x-raw(memory:NVMM), formatt=I420 !  nvv4l2vp8enc ! video/x-vp8 ! rtpvp8pay !  queue !  capsfilter caps=application/x-rtp,media=video,encoding-name=VP8,payload=97 ! webrtcbin name=sendrecv";

but nothing seems to link without issues, I’m pretty sure the answer it’s close but I haven’t managed to find it, any advice?

EDIT: Little update, I’ve managed to make the following pipeline work:

pipelineHw = "udpsrc multicast-group=224.1.1.1 auto-multicast=true port=5000 ! queue ! "
        " application/x-rtp,media=video,clock-rate=90000,encoding-name=VP8,framerate=25/1 ! rtpvp8depay ! video/x-vp8 ! omxvp8dec ! "
        " nvvidconv !  video/x-raw(memory:NVMM), format=RGBA ! nvvidconv ! video/x-raw(memory:NVMM), formatt=I420 !  nvv4l2vp8enc ! "
        " video/x-vp8 ! rtpvp8pay !  queue ! "
        "capsfilter caps=application/x-rtp,media=video,encoding-name=VP8,payload=97 ! webrtcbin name=sendrecv";

now it’s streaming a lot faster than before but I’m getting a really weird waring each frame:

(exe:14223): GStreamer-CRITICAL **: 16:18:52.304: gst_buffer_resize_range: assertion 'bufmax >= bufoffs + offset + size' failed
0:03:15.562139672 14223   0x7f6c006ad0 WARN          v4l2bufferpool gstv4l2bufferpool.c:1480:gst_v4l2_buffer_pool_dqbuf:<nvv4l2vp8enc1:pool:sink> V4L2 provided buffer has bytesused 0 which is too small to include data_offset 0
0:03:15.562171128 14223   0x7f6c006ad0 WARN          v4l2bufferpool gstv4l2bufferpool.c:1480:gst_v4l2_buffer_pool_dqbuf:<nvv4l2vp8enc1:pool:sink> V4L2 provided buffer has bytesused 0 which is too small to include data_offset 0
0:03:15.562200056 14223   0x7f6c006ad0 WARN          v4l2bufferpool gstv4l2bufferpool.c:1480:gst_v4l2_buffer_pool_dqbuf:<nvv4l2vp8enc1:pool:sink> V4L2 provided buffer has bytesused 0 which is too small to include data_offset 0
0:03:15.562171928 14223   0x7f98017400 LOG            dtlssrtpdemux gstdtlssrtpdemux.c:129:sink_chain:<dtlssrtpdemux0> pushing rtp packet

I can’t find the reason behind this and don’t know if it affects performance in any way, anyone know the reason?

EDIT2:

I’ve noticed this waring has also poped up in the previous element of the app. This element is based on:
gst_gpumat

But I don’t really understand what I could have broken there to change the buffer size, I’m just drawing over the gpu matt and returning it to the buffer and that error wasn’t there before (I’ve updated from gstreamer 14.5 to 16.2 since writting this, the error appeared with the update)

#3

Hi,
There is a valid example of using UDP:

We are deprecating omx plugins. You may use v4l2 plugins such as nvv4l2h264enc.

The NVIDIA plugins are built and verified with gstreamer 1.14.5. Suggest you try the default version.

#4

Hey! Thanks for the tip with omx plugins, I’ll definetly try that.

I tried both with your pipeline and mine (I already got one working as posted above) and both get similar stream speed but the same weird buffer size errors I posted, do you have any idea on what might be causing those?

#5

Hi,
The warning messages are harmless and you may remove it. Please check

#6

Oh that great, do you have any guide in how to properly rebuild those libraries? I’ve found the source but I don’t see any build instructions anywere

#7

Hi,
Please look at

Linux_for_Tegra\source\public\gst-v4l2\README.txt
#8

Great, got it!

Any tips in improving fps with the pipeline I posted above?

#9

I was just running some test and while your fix to libgstnvvideo4linux2.so does fix the 3 warings around gstbufferpool, one of the warning it’s still there:

Is that one also meaningless? If so how can I remove it? I need to get rid of it because I get one every frame and clogs the log output

Increasing GStreamer stream framerate
#10

Hi,
Is the warning specific to udp decoding? We don’t see it in video file playback:

$ gst-launch-1.0 filesrc location= sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvoverlaysink
#11

No, the warning appears in the gpu_mat program that uses the following code:

with the following pipeline:

Nvv4l2decoder + RTSP can not display
#12

Hi,
We try UDP streaming and don’t observe the print.

// server
$ gst-launch-1.0 videotestsrc ! video/x-raw,width=1920,height=1080 ! nvvidconv ! nvv4l2h264enc insert-sps-pps=1 ! h264parse ! rtph264pay ! udpsink host=127.0.0.1 port=5000

// client
$ gst-launch-1.0 udpsrc port=5000 ! 'application/x-rtp,encoding-name=H264,payload=96' ! rtph264depay ! h264parse ! nvv4l2decoder ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvvidconv ! 'video/x-raw(memory:NVMM),width=1280,height=720,format=I420' ! nvv4l2vp8enc ! fakesink sync=false

Are you able to reproduce it with gst-launch?

#13

Hey! Yes, it shows up with gst-launch too but as I mentioned it’s with the rtsp stream not udp. Try with the following:

gst-launch-1.0 rtspsrc location=rtsp://192.168.0.138:8554/video ! queue ! rtph264depay ! video/x-h264, stream-format=byte-stream ! h264parse ! nvv4l2decoder !  nvvidconv name=myconv ! 'video/x-raw(memory:NVMM), formatt=I420' !  nvv4l2vp8enc maxperf-enable=1 ! video/x-vp8 ! rtpvp8pay ! udpsink host=224.1.1.1 port=5000 sync=false

I think this is an issue that appeared in gstreamer 16, since I got the problem when updating but seems like a big issue that gstreamer 14 might be overlooking (or maybe i changed the pipeline while updating and I didnt notice which seems feasible). I know the oficial supported version in Jetpack 4.4 is 1.14.5 but Gstreamer 1.16 is mandatory for working with webrtc, and everything in our program is working flawlessly except this error spam which clogs the ouput so changing to 1.14 is not an option

#14

Hi,
We try the pipeline and don’t see the issue:

$ gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! rtph264depay ! h264parse ! nvv4l2decoder ! nvvidconv ! 'video/x-raw(memory:NVMM),format=I420' ! nvv4l2vp8enc ! rtpvp8pay ! udpsink host=224.1.1.1 port=5000 sync=false

So it looks specific to 1.16. Users in gstreamer forum may have mor experience on this. Suggest you go there to get help.

#15

I’ve already tried there, but I figured this might be a issue linked to the Jetson board. Thank you for your help! At least this isn’t a critical error and we can ignore it.

One more question: Gstreamer 1.16 is really usefull and we plan on using it in the Jetson for a while, do you have any plan on updating the default version in future Jetpack releases?

#16

Please note that the gstallocator.c false positive warning issue has been fixed upstream (see link below). This warning was completely harmless.

As for the assertion, it’s often hit due to miss-implementation of buffer pool reset function. It was also broken until recently in the GstBufferPool base class. The consequence of this one would be that the pool fails to reset the buffer to it’s original size. In the good cases (and most case) the buffer will be discarded and given back to the allocator. This may lead to unwanted run-time allocation, even though V4L2 allocator is handling a memory pool mitigating this defect. In the bad case, a shorter buffer may be acquired from the pool, this would generally lead to errors, but should be rare.

I believe it has been recommended few times by NVidia to report upstream some issues. Be ware that we cannot deal with reported issues unless you do have way to reproduce that does not imply any forked code. We have analyzed NVidia GStreamer changes provided in a flat tarball and found that the amount of modified code is too important and the changes do not follow any of the upstream guidelines or even coding style requirement. We strongly encourage following an upstream work flow when enhancing GStreamer for your platform, this is the only way your users will fully benefit from upstream development.

Gstreamer 1.16.2 Error gst_buffer_resize_range
Increasing GStreamer stream framerate
Multiple output resolutions Gstreamer
Gstreamer: H264 encoder V4L2 provided buffer has bytesused 0 Crashing Error
#17

Hi,
Thanks for sharing the patches. We will check and evaluate to include them into the package.

We have modification in gst-v4l2 open source code to enable hardware acceleration on desktop GPUSs and Jetson platforms. Certain codes are specific and may not be general to other vendor’s platforms. We will try to reduce/eliminate the deviation and unify with the upstream code.

#19