Gstreamer nvvidconv memory conversion error causing rtspstream failure?

I am taking a udpsrc, processing it with a custom gstreamer element called qtoverlay, and then sending it to a udpsink to stream out via rtsp.
Pipeline A (udpsrc -->qtoverlay -->nvvidconv (NVMM) --> nvvidconv (raw) --> omxh264enc–>udpsink) works
gst-launch-1.0 udpsrc port=5007 buffer-size=60000000 ! application/x-rtp, media=video, clock-rate=90000, encoding-name=H265,playload=96 ! queue ! rtph265depay ! h265parse ! video/x-h265,alignment=au ! nvv4l2decoder disable-dpb=true enable-max-performance=1 ! queue max-size-bytes=0 ! videorate max-rate=30 ! queue ! nvvidconv ! video/x-raw,width=1920,height=1080,framerate=30/1 ! qtoverlay qml=main.qml ! tee name=ovl ovl. ! nvvidconv ! ‘video/x-raw, format=I420,width=1920,height=1080’ ! omxh264enc ! video/x-h264,stream-format=byte-stream ! h264parse ! rtph264pay pt=96 ! tee name=ovl1 ovl1. ! udpsink host=127.0.0.1 port=10006 sync=false async=false

Pipeline B (udpsrc -->qtoverlay -->nvvidconv (raw) --> omxh264enc–>udpsink) runs without throwing an error, but I see nothing on the vlc player. In fact, when I run ‘top’ on linux, I don’t even see the process running.
gst-launch-1.0 udpsrc port=5007 buffer-size=60000000 ! application/x-rtp, media=video, clock-rate=90000, encoding-name=H265,playload=96 ! queue ! rtph265depay ! h265parse ! video/x-h265,alignment=au ! nvv4l2decoder disable-dpb=true enable-max-performance=1 ! queue max-size-bytes=0 ! videorate max-rate=30 ! queue ! nvvidconv ! video/x-raw,width=1920,height=1080,framerate=30/1 ! qtoverlay qml=main.qml ! tee name=ovl ovl. ! nvvidconv ! ‘video/x-raw(memory:NVMM), format=I420,width=1920,height=1080’ ! nvvidconv ! ‘video/x-raw’ ! omxh264enc ! video/x-h264,stream-format=byte-stream ! h264parse ! rtph264pay pt=96 ! tee name=ovl1 ovl1. ! udpsink host=127.0.0.1 port=10006 sync=false async=false

In pipeline B, I could replace nvvidconv with videoconvert and then the videostream will work. But this uses up a lot of cpu. Is there something wrong with nvvidconv?

Hi,
In pipelien B, it is strange you run

nvvidconv ! 'video/x-raw' ! omxh264enc

video/x-raw(memory:NVMM) is supported in sink pad of omxh264enc

And we are deprecating omx plugins. Please use nvv4l2h264enc.

When I tried substituting omxh264enc with nvv4l2h264enc I no longer see any video when I run my player pipeline:

gst-launch-1.0 udpsrc port=10000 ! application/x-rtp, media=video, encoding-name=H264,playload=96 ! queue ! rtph264depay ! h264parse ! video/x-h264,alignment=au ! nvv4l2decoder disable-dpb=true enable-max-performance=1 ! queue max-size-bytes=0 ! videorate max-rate=30 ! queue ! nvvidconv ! nveglglessink window-width=720 window-height=480 sync=false

Also, I convert to video/x-raw even though omxh264enc support NVMM because if I run more than 2 instances of pipeline B without converting it to video/x-raw, the video on vlc begins to tear. I increment udscr and udpsink port for each pipeline, so it’s not a problem of the src and sink stomping on each other. This is another issue that I was going to ask for help on in a seperate thread.

Note that although this is not explicit looking at SRC and SINK caps, nvvidconv expects at least one of its input or output to be in NVMM memory (i.e. x-raw to x-raw is not supported, AFAIK).
Furthermore, there an issue with h264 parsing with nvv4l2decoder, where you may have to remove h264parse before decoder.

1 Like

The answer to my streaming corruption problem was nvv4l2h264enc.

When I initially replaced omxh264enc with the nvv4l2h264enc element, I saw no stream at all. The answer to that problem was found in the post I’m linking below. I had to enable insert-sps-pps. AND I had to remove the h264parse element, just as Honey_Patouceul had suggested.

1 Like

@DaneLLL

Even though I can now handle 4 streams without video tearing, the video is a bit choppy. As if it is dropping frames. This choppiness goes away if I replace nvv4l2h264enc with omxh264enc. (But if I use omxh264enc, I can’t run more than 2 pipelines before experiencing tearing )

Here is my bare-bones pipeline. Is there an additional setting i could use to smooth out the video?

sudo gst-launch-1.0 udpsrc port=5004 buffer-size=60000000 ! application/x-rtp, media=video, clock-rate=90000, encoding-name=H265,playload=96 ! queue ! rtph265depay ! h265parse ! video/x-h265,alignment=au ! nvv4l2decoder disable-dpb=true enable-max-performance=1 ! queue max-size-bytes=0 ! videorate max-rate=30 ! queue ! nvvidconv ! ‘video/x-raw(memory:NVMM), format=I420,width=1920,height=1080’ ! nvv4l2h264enc insert-sps-pps=1 ! video/x-h264,stream-format=byte-stream ! rtph264pay pt=96 ! udpsink host=127.0.0.1 port=10000 sync=false async=false

Not sure, but you may try if adding rtpjitterbuffer after udpsrc helps.