Pixelated video when using Gstreamer pipeline with OpenCV via RTSP

[BlindBoolean]

I’m working on a program that passes a Gstreamer pipeline to OpenCV via the cv2.VideoCapture() function on a Jetson Xavier NX board, but I’ve been running into issues. Every few seconds, the video becomes pixelated and distorted, like so:

Here is the pipeline I’ve been using:

"rtspsrc location=rtsp://local-ip/ latency=15 ! rtph265depay ! h265parse ! nvv4l2decoder ! "
"nvvidconv ! video/x-raw, width=640, height=480, format=BGRx ! videoconvert ! appsink"

After exploring posts for similar issues, it seems the likeliest culprit is packets getting lost across the rtsp network.

Looking through some options, I found that there is a plugin called rtpjitterbuffer, where:

If the “do-lost” property is set, lost packets will result in a custom serialized 
downstream event of name GstRTPPacketLost. The lost packet events are 
usually used by a depayloader or other element to create concealment data 
or some other logic to gracefully handle the missing packets.

I reworked my pipeline to have jitter buffer leading into a rtph265depay plugin in the hopes the above-mentioned data concealment would appear, but the pixelation still occurred at approximately the same rate. Here is that pipeline:

"rtspsrc location=rtsp://local-ip/ latency=15 ! rtpjitterbuffer do-lost=true ! rtph265depay ! h265parse ! "
"nvv4l2decoder ! nvvidconv ! video/x-raw, width=640, height=480, format=BGRx ! videoconvert ! appsink"

The next thing I found was that rtpjitterbuffer has two parameters stats and post-drop-messages that can output the number of packets lost during a stream. I was hoping to use one of these to verify my packet loss theory, but for the life of me I could not figure out how to set them up or save the output data.

stats info found here: link
post-drop-messages info found here: link

With that all out of the way, I was wondering if anyone knew either:

1. How to configure rtpjitterbuffer to conceal lost packets across an rtsp gstreamer pipeline

2. How to configure rtpjitterbuffer to print/write the number of lost packets in a stream to the command line or a file

Any help or advice would be greatly appreciated!

[Honey_Patouceul]

Not sure for your case, but I’d try to use TCP transport instead of UDP, If your RTSP server supports this, try something like:

rtspsrc location=rtspt://local-ip/ latency=15 ! rtph265depay ! h265parse ! nvv4l2decoder ! nvvidconv ! video/x-raw, width=640, height=480, format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink drop=1

#or
rtspsrc location=rtsp://local-ip/ protocols=tcp latency=15 ! rtph265depay ! h265parse ! nvv4l2decoder ! nvvidconv ! video/x-raw, width=640, height=480, format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink drop=1

Also note that 15 ms is very low latency (don’t know your framerate and encoding properties), it might work on localhost with h264 avc stream format, but you may investigate optimal value for your case (the lower may not be the better).

[BlindBoolean]

Hey @Honey_Patouceul, thank you so much for the fast reply!! Also for your other Gstreamer-related posts, they’ve been even more helpful than the documentation as I’ve been working on the project this week!

I just tried both of your solutions. The first one reduced the pixelation occurrences significantly, and the second one has eliminated the pixelation completely!! I’m going to run the stream overnight, and look through the recordings tomorrow morning just to be sure, but it’s looking like using the tcp protocol has done the trick!

I also increased the latency to 100 ms, and the stream quality has enormously improved. Thank you so much for your help! I’ll mark your reply as the solution tomorrow if everything worked ok.

[DaneLLL]

@Honey_Patouceul thanks for your quick suggestion.

[BlindBoolean]

@Honey_Patouceul TCP that totally did the job!! Thanks again for your solution, it absolutely saved me.

[duffjay]

I had a similar problem. I am a bit lazy and didn’t want to re-engineer my solution around a RTP server. I tried putting a timestamp on the source and using rtpjittter on the consuming pipeline. This eliminated my pixelation problems. I have not measured latency - my pipeline may be inferior in that regard - it’s not critical in my use case.

Here is my source pipeline (running on a Jetson Nano) . Note: do-timestamp=1. You need a timestamp on the source feed or the jitter element has nothing to work with. (bonus - in my application, the aeregion was a TREMENDOUS help)

gst-launch-1.0 -e nvarguscamerasrc do-timestamp=1 aeregion=“1500 2000 3000 3000 1” ! ‘video/x-raw(memory:NVMM), width=4032, height=3040, framerate=1/1’ ! nvvidconv flip-method=2 ! nvv4l2h264enc bitrate=8000000 insert-sps-pps=true ! queue ! rtph264pay mtu=1400 ! udpsink host=“10.0.0.32” port=5118 sync=false async=false

Here is my client side pipeline. (this is not a gstreamer - this is the string I passed into OpenCV video capture). I suspect the real problem is OpenCV dropping data. (using rtpjitterbuffer cleared it up).

udpsrc port=5118 ! application/x-rtp,media=video,encoding-name=H264 ! queue ! rtpjitterbuffer ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! appsink

Here is my GStreamer display pipeline:
gst-launch-1.0 udpsrc port=5118 ! application/x-rtp,encoding-name=H264,payload=96 ! rtpjitterbuffer ! rtph264depay ! h264parse ! queue ! avdec_h264 ! xvimagesink sync=false async=false -e

I never had pixelation in the display pipeline - only the OpenCV pipeline. The client computer is a very powerful computer (running 15 camera feeds) and you’ll notice I throttled to 1 fps. I barely know what I’m doing myself - any suggested improvements welcome!