Cannot open Gstreamer Pipeline with OpenCV on Jetson Nano

Autonomous Machines Jetson & Embedded Systems Jetson Nano
1

Hello,
I’m trying to use tracker algorithms of OpenCV which is coming from UDP port. Following pipeline is working good on two different PC and Raspberry Pi 4 but I can’t open it on Jetson Nano.

VideoCapture video("udpsrc port=5600 ! application/x-rtp,payload=96,encoding-name=H264 ! rtpjitterbuffer mode=1 ! rtph264depay ! h264parse ! decodebin ! videoconvert ! appsink emit-signals=true sync=false max-buffers=1 drop=true", CAP_GSTREAMER);

I’m getting following output when I execute the code.

Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261

FYI, I connected Jetson Nano to directly Monitor via HDMI output.

Thanks.

2

Hi,
Please try gst-launch-1.0 command:

$ gst-launch-1.0 udpsrc port=5600 ! application/x-rtp,payload=96,encoding-name=H264 ! rtpjitterbuffer mode=1 ! rtph264depay ! h264parse ! decodebin ! videoconvert ! video/x-raw,format=BGR ! fakesink

If the gst-launch command works, it should also work fine in OpenCV code.

3

Thanks for answer,

Unfortunately it doesn’t work. The command that you advised gives this output continuously:

Additional debug info:
gstbasetransform.c(1415): gst_base_transform_reconfigure (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter2:
not negotiated
WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter2: not negotiated

Also, I found command which is running without problem. I got advice by https://github.com/patrickelectric

udpsrc port=5600 ! application/x-rtp,payload=96,encoding-name=H264 ! rtpjitterbuffer mode=1 ! rtph264depay ! h264parse ! decodebin ! videoconvert ! autovideosink emit-signals=true sync=false max-buffers=1 drop=true

https://user-images.githubusercontent.com/44507545/74029500-db976480-49bd-11ea-8ee6-766f1303127c.jpg

But also there is problem with this command, because it is just opening without border and windows. I am not able click anything. When I run command. I just switched with Alt+Tab shortcut and closed command from terminal without seeing it. You can see on the screenshot.

Also yeah gstreamer work with command but it’s strange. I don’t get why Jetson Nano switch blocking mode when it starts. I think it’s related to strange start.

4

Hi,
Please check if software decoder works:

udpsrc port=5600 ! application/x-rtp,payload=96,encoding-name=H264 ! rtpjitterbuffer mode=1 ! rtph264depay ! h264parse ! avdec_h264 ! xvimagesink sync=false

If software decoder works, please then try hardware decoder:

udpsrc port=5600 ! application/x-rtp,payload=96,encoding-name=H264 ! rtpjitterbuffer mode=1 ! rtph264depay ! h264parse ! nvv4l2decoder ! nvoverlaysink sync=false
5

I’m able to run gstreamer with software decoder:

gst-launch-1.0 udpsrc port=5600 ! application/x-rtp,payload=96,encoding-name=H264 ! rtpjitterbuffer mode=1 ! rtph264depay ! h264parse ! avdec_h264 ! xvimagesink sync=false

Also second gstreamer with hardware decoder works well with some warnings:

gst-launch-1.0 udpsrc port=5600 ! application/x-rtp,payload=96,encoding-name=H264 ! rtpjitterbuffer mode=1 ! rtph264depay ! h264parse ! nvv4l2decoder ! nvoverlaysink sync=false

Output:

Setting pipeline to PAUSED ...
Opening in BLOCKING MODE 
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261

Both command works without any problem. But gstreamer in opencv needs “appsink” so I changed code to:

VideoCapture video("udpsrc port=5600 ! application/x-rtp, payload=96, encoding-name=H264 ! rtpjitterbuffer mode=1 ! rtph264depay ! h264parse ! avdec_h264 ! decodebin ! videoconvert ! appsink max-buffers=2 drop=true emit-signals=true sync=false", CAP_GSTREAMER);

I think problem was related to decoder option. “avdec_h264” option works while nvv4l2decoder doesn’t work.

But there is a problem with this pipeline, it makes too lag. Udp video has very latency while other functions of opencv works without delay. Delay is increasing time by time. I think problem is between “sink” and “decoder”. At least now I can receive video. I’m working to resolve this problem.

Thanks for helping @DaneLLL :)
After resolve this problem I’ll share guide about integration gazebo sim and jetson nano on github.

EDIT: Ubuntu-Desktop PC runs without latency or problem with this pipeline :/ How can I accelerate video decoding on Jetson Nano?

6

You may try something like:

VideoCapture video("udpsrc port=5600 ! application/x-rtp, payload=96, encoding-name=H264 ! rtph264depay ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink", CAP_GSTREAMER);

If not working you may also try adding queue between udpsrc and rtph264depay.