Issues decoding RTSP stream using nvv4l2decoder with Jetpack 4.4

benky58un · July 10, 2020, 9:12am

Hi,

I just upgraded from L4T 32.3.1 to L4T 32.4.3 (Jetpack 4.4) and a GStreamer pipeline that used to work on 32.3.1 no longer works on 32.4.3. I believe this has something to do with the nvv4l2decoder filter.

The pipeline in question is

GST_DEBUG=3 gst-launch-1.0 -e rtspsrc location="rtsp://192.168.0.103:554/av0_0" is-live=true protocols=tcp ! rtph264depay ! h264parse ! nvv4l2decoder ! video/x-raw\(memory:NVMM\) ! nvvidconv ! video/x-raw\(memory:NVMM\),width=256,height=144 ! nvvidconv ! video/x-raw,format=GRAY8 ! multifilesink location="/tmp/%d.y"

When running on L4T 32.3.1 this pipeline works and I get a series of files being generated. However when running the same pipeline on L4T 32.4.3 no files are generated and it seems to not respond to the EOS event when stopping the pipeline using ctrl-c on the console.

The output when running the pipeline on 32.4.3 is

$ GST_DEBUG=3 gst-launch-1.0 -e rtspsrc location="rtsp://192.168.0.103:554/av0_0" is-live=true protocols=tcp ! rtph264depay ! h264parse ! nvv4l2decoder ! video/x-raw\(memory:NVMM\) ! nvvidconv ! video/x-raw\(memory:NVMM\),width=256,height=144 ! nvvidconv ! video/x-raw,format=GRAY8 ! multifilesink location="/tmp/%d.y"
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE
0:00:00.109288642  9922   0x55ae811ec0 WARN                    v4l2 gstv4l2object.c:4435:gst_v4l2_object_probe_caps:<nvv4l2decoder0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1
0:00:00.109362134  9922   0x55ae811ec0 WARN                    v4l2 gstv4l2object.c:2372:gst_v4l2_object_add_interlace_mode:0x55ae803710 Failed to determine interlace mode
Pipeline is live and does not need PREROLL ...
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://192.168.0.103:554/av0_0
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
Progress: (request) SETUP stream 0
Progress: (open) Opened Stream
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Progress: (request) Sending PLAY request
Progress: (request) Sending PLAY request
Progress: (request) Sent PLAY request
0:00:02.150779553  9922   0x7f7403a1e0 FIXME               basesink gstbasesink.c:3145:gst_base_sink_default_event:<multifilesink0> stream-start event without group-id. Consider implementing group-id handling in the upstream elements
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261

(gst-launch-1.0:9922): GStreamer-CRITICAL **: 17:00:29.932: gst_mini_object_unref: assertion 'mini_object != NULL' failed

It will stay like this “forever”.

If I kill the pipeline using ctrl-c, I get the following

^Chandling interrupt.
Interrupt: Stopping pipeline ...
EOS on shutdown enabled -- Forcing EOS on the pipeline
Waiting for EOS...

It will also stay like this “forever” unless I issue another ctrl-c whereupon I get the following

^Chandling interrupt.
Interrupt: Stopping pipeline ...
Interrupt while waiting for EOS - stopping pipeline...
Execution ended after 0:00:13.640278834
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
0:00:13.949095411  9922   0x55ae815800 WARN                 rtspsrc gstrtspsrc.c:5200:gst_rtspsrc_loop_interleaved:<rtspsrc0> warning: The server closed the connection.

^C

If I substitute the nvv4l2decoder filter with avdec_h264 then the pipeline works perfectly.

GST_DEBUG=3 gst-launch-1.0 -e rtspsrc location="rtsp://192.168.0.103:554/av0_0" is-live=true protocols=tcp ! rtph264depay ! h264parse ! avdec_h264 ! video/x-raw ! nvvidconv ! video/x-raw\(memory:NVMM\),width=256,height=144 ! nvvidconv ! video/x-raw,format=GRAY8 ! multifilesink location="/tmp/%d.y"

Interestingly this only seems to cause a problem if the pipeline is streaming from an RTSP source. If I substitute the RTSP source with a file then nvv4l2decoder appears to operate properly.

Can anyone offer me any assistance or guidance about why I am seeing this behaviour only on the latest Jetpack release? Any help would be greatly appreciated.

Regards
Ben

Honey_Patouceul · July 10, 2020, 11:59am

I’ve also seen something similar since R32.4. What if you :

use omxh264dec instead of nvv4l2decoder ?
or remove h264parse before nvv4l2decoder ?
or use double h264parse conversion through avc format: ... ! h264parse ! video/x-h264, stream-format=avc ! h264parse ! video/x-h264, stream-format=byte-stream ! nvv4l2decoder ! ... ?

LoveNvidia · July 10, 2020, 2:47pm

@Honey_Patouceul
This structural tested with gstreamer + opencv in python?

LoveNvidia · July 10, 2020, 2:50pm

@benky58un

GST_DEBUG=3 gst-launch-1.0 -e rtspsrc location="rtsp://192.168.0.103:554/av0_0" is-live=true protocols=tcp ! rtph264depay ! h264parse ! avdec_h264 ! video/x-raw ! nvvidconv ! video/x-raw\(memory:NVMM\),width=256,height=144 ! nvvidconv ! video/x-raw,format=GRAY8 ! multifilesink location="/tmp/%d.y"

How much RAM/CPU is consumed?

DaneLLL · July 13, 2020, 2:05am

Hi,
Please check if it works with nvoverlaysink.

$ GST_DEBUG=3 gst-launch-1.0 -e rtspsrc location="rtsp://192.168.0.103:554/av0_0" is-live=true protocols=tcp ! rtph264depay ! h264parse ! nvv4l2decoder ! nvoverlaysink

benky58un · July 13, 2020, 2:15am

Hi @DaneLLL,

No luck I am afraid. I get the same result. The output that I receive is as follows

~$ GST_DEBUG=3 gst-launch-1.0 -e rtspsrc location="rtsp://192.168.0.103:554/av0_0" is-live=true protocols=tcp ! rtph264depay ! h264parse ! nvv4l2decoder ! nvoverlaysink
0:00:00.105373146  9824   0x5572bd74f0 WARN                     omx gstomx.c:2826:plugin_init: Failed to load configuration file: Valid key file could not be found in search dirs (searched in: /home/benky58un/.config:/etc/xdg/xdg-unity:/etc/xdg as per GST_OMX_CONFIG_DIR environment variable, the xdg user config directory (or XDG_CONFIG_HOME) and the system config directory (or XDG_CONFIG_DIRS)
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE 
0:00:00.127210870  9824   0x5572bd74f0 WARN                    v4l2 gstv4l2object.c:4435:gst_v4l2_object_probe_caps:<nvv4l2decoder0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1
0:00:00.127270870  9824   0x5572bd74f0 WARN                    v4l2 gstv4l2object.c:2372:gst_v4l2_object_add_interlace_mode:0x5572bdb230 Failed to determine interlace mode
Pipeline is live and does not need PREROLL ...
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://192.168.0.103:554/av0_0
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
Progress: (request) SETUP stream 0
Progress: (open) Opened Stream
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Progress: (request) Sending PLAY request
Progress: (request) Sending PLAY request
Progress: (request) Sent PLAY request
0:00:02.190055841  9824   0x7f70005f20 FIXME               basesink gstbasesink.c:3145:gst_base_sink_default_event:<nvoverlaysink-nvoverlaysink0> stream-start event without group-id. Consider implementing group-id handling in the upstream elements
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 

(gst-launch-1.0:9824): GStreamer-CRITICAL **: 10:08:02.743: gst_mini_object_unref: assertion 'mini_object != NULL' failed
^Chandling interrupt.
Interrupt: Stopping pipeline ...
EOS on shutdown enabled -- Forcing EOS on the pipeline
Waiting for EOS...
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Interrupt while waiting for EOS - stopping pipeline...
Execution ended after 0:00:15.705092323
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
0:00:16.027118535  9824   0x5572b84a80 WARN                 rtspsrc gstrtspsrc.c:5200:gst_rtspsrc_loop_interleaved:<rtspsrc0> warning: The server closed the connection.
^C
$

LoveNvidia · July 13, 2020, 8:25am

@Honey_Patouceul
I used the below codes for decoding in jetapck 4.2.2 and opecnv 3.6.8, I get this error:

 def read_cam():
    cap = cv2.VideoCapture("rtspsrc location=rtsp://IP:PORT/1920x1080.264 ! h264parse ! \
    video/x-h264, stream-format=avc ! h264parse ! video/x-h264, stream-format=byte-stream ! nvv4l2decoder ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink ")
    if cap.isOpened():
        cv2.namedWindow("demo", cv2.WINDOW_AUTOSIZE)
        while True:
            ret_val, img = cap.read()
            cv2.imshow('demo',img)
            cv2.waitKey(10)
    else:
     print ("rtsp open failed")

    cv2.destroyAllWindows()


if __name__ == '__main__':
    read_cam()

nvbuf_utils: Could not get EGL display connection
Opening in BLOCKING MODE

(python3:30684): GStreamer-CRITICAL **: 12:52:06.921: gst_element_get_state: assertion ‘GST_IS_ELEMENT (element)’ failed
VIDEOIO ERROR: V4L: device rtspsrc location=rtsp://IP:PORT/1920x1080.264 ! h264parse ! video/x-h264, stream-format=avc ! h264parse ! video/x-h264, stream-format=byte-stream ! nvv4l2decoder ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink : Unable to query number of channels
rtsp open failed

benky58un · July 13, 2020, 8:37am

Hi @Honey_Patouceul,

Thanks for the suggestions!

I removed the h264parse filter before nvv4l2decoder and the pipeline now seems to behave correctly. My pipeline is now as follows,

GST_DEBUG=3 gst-launch-1.0 -e rtspsrc location="rtsp://192.168.0.103:554/av0_0" is-live=true protocols=tcp ! rtph264depay ! nvv4l2decoder ! video/x-raw\(memory:NVMM\) ! nvvidconv ! video/x-raw\(memory:NVMM\),width=256,height=144 ! nvvidconv ! video/x-raw,format=GRAY8 ! multifilesink location="/tmp/%d.y"

This also seems to work on L4T 32.3.1 so perhaps the filter has always been redundant? I am still puzzled why the behaviour is different between the two versions of L4T.

I tried to refer back to the Accelerated GStreamer User Guide (Release 28.2 - which seems to be the latest version available) to look for examples of the correct way to use the filter but surprisingly there are no examples for the nvv4l2decoder filter only the omxh264dec which is going to be deprecated.

Regards,

DaneLLL · July 13, 2020, 12:39pm

Hi,
We are investigating on this. Will update when there is new finding.

Honey_Patouceul · July 13, 2020, 1:29pm

@LoveNvidia,

LoveNvidia:

cap = cv2.VideoCapture("rtspsrc location=rtsp://IP:PORT/1920x1080.264 ! h264parse ! \
    video/x-h264, stream-format=avc ! h264parse ! video/x-h264, stream-format=byte-stream ! nvv4l2decoder ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink ")

Your pipeline lacks rtph264depay between rtspsrc and h264parse.

LoveNvidia · July 13, 2020, 4:54pm

Thanks a lot.
1- when I comment ‘video/x-h264, stream-format=byte-stream !’ also work, Is this need?
2- what’s mean when we use :
video/x-raw, format=BGR and video/x-raw(memory:NVMM),format=(string)NV12 in elements? there are used for converting? or stograge? if so, are GPU buffer?
3- appsrc is used cpu buffer?
4- what’s about video/x-h264, stream-format=byte-stream ? GPU-buffer??

Honey_Patouceul · July 14, 2020, 9:31pm

A gstreamer pipeline is a sequence of gstreamer plugins, from (at least) a source plugin to (at least) a sink plugin.
Each plugin has either SRC capabilities, or SINK capabilities, or both in most cases.
You would use gst-inspect-1.0 command for listing all plugins that your current gstreamer install knows:

gst-inspect-1.0

or for getting details about a given plugin, for example:

gst-inspect-1.0 nvvidconv

The latter will show SRC and SINK caps, so you would know depending on its SINK capabilities if a plugin can be plugged after another one (there should be at least one set of types/formats/… matching with previous SRC capabilities).
If no caps match previous plugin SRC capabilities and next plugin SINK caps, the pipeline would fail to link and start.
In some cases, various possibilities are available, and gstreamer framework could choose what it thinks is best, but that may not be the best solution for your case, so in this case you would specify wanted caps between plugins for ensuring gstreamer uses the types and formats you’re expecting.

If next plugin only expects video/x-h264, stream-format=byte-stream as SINK caps, this should make no difference.

The first case means BGR (888) video format in standard (CPU) allocated memory.
The second one means NV12 format in NVMM memory, that is contiguous DMA-able memory, suitable for argus, HW encoders/decoders, GPU processing.
nvvidconv plugin allow to copy/convert-format/rescale from NVMM to NVMM/CPU to NVMM/NVMM to CPU (but not CPU to CPU, although its caps don’t tell).

Yes, in most cases, especially if using opencv with a videoWriter, appsrc is a CPU application outputting BGR frames in CPU allocated memory.

No, it’s h264 encoded video, in CPU allocated memory, with byte-stream format suitable for RTP for example. Most containers would use avc format instead.

deemmirror · July 14, 2020, 10:10pm

Hello!

I have a bit similar issue. Please, can you change ... ! nvv4l2decoder ! ... to ... ! nvv4l2decoder enable-frame-type-reporting=true ! ... in your base pipeline. Does it works to you? And what in output?

Honey_Patouceul · July 14, 2020, 10:19pm

I have checked now that both:

gst-launch-1.0 -v nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=640, height=480, framerate=30/1' ! nvv4l2h264enc ! h264parse ! nvv4l2decoder enable-frame-type-reporting=true ! videoconvert ! fpsdisplaysink video-sink= fakesink text-overlay=false
#and
gst-launch-1.0 -v nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=640, height=480, framerate=30/1' ! nvv4l2h264enc ! h264parse ! nvv4l2decoder enable-error-check=true ! videoconvert ! fpsdisplaysink video-sink= fakesink text-overlay=false

still stall and make faults in argus upon double Ctrl-C. So it looks your issue is another one.

deemmirror · July 14, 2020, 10:29pm

May be… Also, when i tried to encode and decode in the same pipeline, it only works as:

 nvv4l2h264enc ! h264parse ! tsmpegmux ! queue ! tsdemux ! h264parse ! nvv4l2decoder

Also, isn’t nvvidconv should be between nvarguscamerasrc and caps?

Honey_Patouceul · July 14, 2020, 10:37pm

Not sure what you mean here…nvv4l2h264enc may expect input from NVMM memory, as nvarguscamerasrc provides, but adding nvvidconv in between should be harmless. Same, nvv4l2decoder would output into NVMM memory, so you would use nvvidconv after for copying back (may also convert format and resize) to CPU allocated memory.

deemmirror · July 14, 2020, 10:42pm

As i know, nvvidconv (unlike videoconvert), can use NVMM. So yes, it should be harmless.and can fix some video format conflicts. Also it can transfer buffers from CPU allocated memory to NVMM.

LoveNvidia · July 16, 2020, 2:27pm

@Honey_Patouceul
IN cv2.videoCapture + gstreamer, first the decoded frames pushed into NVVM buffer and then copied into CPU buffer, right? because of opencv only accept CPU buffer, So, In this mode, for one frame decoding we use double memory from 4GB of jetson nano? (NVVM buffer + CPU buffer), I want to know because the deep stream don’t use opencv for decoding, and only use gstreamer for decoding the frames, when I used opencv + gstreamer and decoding the frames with HW, in this case in the same situation, opencv + gstreamer decoder used twice memory usage than deep stream decoder?

and what’s main difference between opencv+gstreamer and deep-stream decoders? CPU/RAM usage.

Honey_Patouceul · July 16, 2020, 6:46pm

First, apologies to @benky58un for squatting this topic.

Only in case of a MIPI/CSI camera being accessed through ISP with argus. In most other cases such as with v4l2src not going through ISP, frames would be received in CPU allocated memory.

Yes, an application linked to opencv would only receive frames into CPU allocated cv::Mat using opencv videoio.
However, you can access NVMM buffers with gstreamer plugin nvivafilter. It is intended to perform CUDA operations on NVMM hosted frames, so you can use it with opencv CUDA. You would have to output RGBA frames from this plugin. You may have a look to this example.
Also note that you can directly access from gstreamer buffer.

There may be (from tail to head) a BGR buffer, a BGRx buffer, a NVMM NV12 buffer, and some H264 encoded/depayed/RTP buffers. For a 4K resolution, worst case being BGRx would be 33MB only, so it should be affordable to have a few copies (I have no Nano, but I think that even with Linux and Ubuntu you would have more than 3.5GB available for your application).

benky58un · July 17, 2020, 2:51am

Thanks for all your contributions to this community @Honey_Patouceul. You have assisted me personally a few times in the last few months - so a big thank you to you.

I don’t mind the activity on this thread. At least it keeps it active and at the top of the forums leading hopefully to a solution to the nvv4l2decoder issue.

@DaneLLL is there a public issue tracker that we can access to more closely track the progress of the investigation into the nvv4l2decoder issue?