Freezing when opencv and gstreamer(with nvv4l2decoder) used on Xavier NX

[lasiyan]

I am developing a C++ application at Nvidia Jetson Xavier NX.

I’m using gstreamer and OpenCV(C++) with gstreamer to get frame from IP Camera.
(my CV version is 4.1.1)

It seems to well, but when I changed IP Camera’s frame size(ex. 1920x1080 to 1280x720),

My application was frozen on VideoCapture’s read function.

VideoCapture stream;

if (use_hw_decoding) // freezing when frame size was changed
{
    // url   rtsp://admin:qwer1234@192.168.0.100:554/video1
    auto url = format_string(
        "rtspsrc location=%s latency=0 buffer-mode=auto ! "
        "queue ! "
        "rtph264depay ! "
        "nvv4l2decoder ! "
        "nvvidconv ! "
        "video/x-raw, format=BGRx ! "
        "videoconvert ! "
        "video/x-raw, format=BGR ! "
        "appsink", __url.c_str());
    
    stream.open(url, cv::CAP_GSTREAMER);
}
else // It's always ok.
{
    stream.open(url, cv::CAP_FFMPEG);
}

while (is_stop == false)
{
    ....
    bool result = stream.grab();  // freezing occured here.
    if (result)
    {
        result = stream.retrieve(MyMatrix);
    }
    ...
}

I searched a lot for related information, but I couldn’t find anything similar.

Attempted to debug via gdb, but it seems to stop inside the nvvidconv or nvv4l2decoder thread.

The source could not be determined.

Use CPU resources when using FFMPEG.
I want to use a hardware decoder as much as possible because I send out a lot of streams.

Has anyone experienced anything similar to me?

[DaneLLL]

Hi,
Please try this sample:
Doesn't work nvv4l2decoder for decoding RTSP in gstreamer + opencv - #3 by DaneLLL

And then try:

    cap = cv2.VideoCapture("rtspsrc location=rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mov ! rtph264depay ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw, width=1280, height=720, format=(string)BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink ")

[lasiyan]

Thank you for your solution!

First, the h264parse keyword after “rtph264depay” resolves the freezing of the read() function !

BTW, do you know why printed the NvMMLiteOpen log twice using h264parse?

Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (896) open OpenCV | GStreamer warning: unable to query duration of stream
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (933) open OpenCV | GStreamer warning: Cannot query video position: status=1, value=0, duration=-1

Second, with size format can help to change IP camera’s frame size.

video/x-raw, width=1280, height=720 format=(string)BGRx !

However, it seems to resize the image in the process of converting or decoding.

So if I used this, I always get 1280 x 720 cv::Matrix.

I want to adjust the frame dynamically by IP camera setting.

Therefore, it works normally if the size format parameter isn’t used.

What are the benefits of using this?

[DaneLLL]

Hi,
Dynamic resolution change is not supported. Suggest set to fixed resolution in the RTSP source and do downscale to the frame data in OpenCV code.

There is a known memory leak and please apply this patch:
Jetson/L4T/r32.6.x patches - eLinux.org

[gstreamer] Memory leak in UDP streaming

With the patch, NvMMLiteOpen log shall be printed once.