Issues decoding RTSP stream using nvv4l2decoder with Jetpack 4.4

Please check if it works with nvoverlaysink.

$ GST_DEBUG=3 gst-launch-1.0 -e rtspsrc location="rtsp://" is-live=true protocols=tcp ! rtph264depay ! h264parse ! nvv4l2decoder ! nvoverlaysink

Hi @DaneLLL,

No luck I am afraid. I get the same result. The output that I receive is as follows

~$ GST_DEBUG=3 gst-launch-1.0 -e rtspsrc location="rtsp://" is-live=true protocols=tcp ! rtph264depay ! h264parse ! nvv4l2decoder ! nvoverlaysink
0:00:00.105373146  9824   0x5572bd74f0 WARN                     omx gstomx.c:2826:plugin_init: Failed to load configuration file: Valid key file could not be found in search dirs (searched in: /home/benky58un/.config:/etc/xdg/xdg-unity:/etc/xdg as per GST_OMX_CONFIG_DIR environment variable, the xdg user config directory (or XDG_CONFIG_HOME) and the system config directory (or XDG_CONFIG_DIRS)
Setting pipeline to PAUSED ...
0:00:00.127210870  9824   0x5572bd74f0 WARN                    v4l2 gstv4l2object.c:4435:gst_v4l2_object_probe_caps:<nvv4l2decoder0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1
0:00:00.127270870  9824   0x5572bd74f0 WARN                    v4l2 gstv4l2object.c:2372:gst_v4l2_object_add_interlace_mode:0x5572bdb230 Failed to determine interlace mode
Pipeline is live and does not need PREROLL ...
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
Progress: (request) SETUP stream 0
Progress: (open) Opened Stream
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Progress: (request) Sending PLAY request
Progress: (request) Sending PLAY request
Progress: (request) Sent PLAY request
0:00:02.190055841  9824   0x7f70005f20 FIXME               basesink gstbasesink.c:3145:gst_base_sink_default_event:<nvoverlaysink-nvoverlaysink0> stream-start event without group-id. Consider implementing group-id handling in the upstream elements
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 

(gst-launch-1.0:9824): GStreamer-CRITICAL **: 10:08:02.743: gst_mini_object_unref: assertion 'mini_object != NULL' failed
^Chandling interrupt.
Interrupt: Stopping pipeline ...
EOS on shutdown enabled -- Forcing EOS on the pipeline
Waiting for EOS...
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Interrupt while waiting for EOS - stopping pipeline...
Execution ended after 0:00:15.705092323
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
0:00:16.027118535  9824   0x5572b84a80 WARN                 rtspsrc gstrtspsrc.c:5200:gst_rtspsrc_loop_interleaved:<rtspsrc0> warning: The server closed the connection.

I used the below codes for decoding in jetapck 4.2.2 and opecnv 3.6.8, I get this error:

 def read_cam():
    cap = cv2.VideoCapture("rtspsrc location=rtsp://IP:PORT/1920x1080.264 ! h264parse ! \
    video/x-h264, stream-format=avc ! h264parse ! video/x-h264, stream-format=byte-stream ! nvv4l2decoder ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink ")
    if cap.isOpened():
        cv2.namedWindow("demo", cv2.WINDOW_AUTOSIZE)
        while True:
            ret_val, img =
     print ("rtsp open failed")


if __name__ == '__main__':

nvbuf_utils: Could not get EGL display connection

(python3:30684): GStreamer-CRITICAL **: 12:52:06.921: gst_element_get_state: assertion ‘GST_IS_ELEMENT (element)’ failed
VIDEOIO ERROR: V4L: device rtspsrc location=rtsp://IP:PORT/1920x1080.264 ! h264parse ! video/x-h264, stream-format=avc ! h264parse ! video/x-h264, stream-format=byte-stream ! nvv4l2decoder ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink : Unable to query number of channels
rtsp open failed

Hi @Honey_Patouceul,

Thanks for the suggestions!

I removed the h264parse filter before nvv4l2decoder and the pipeline now seems to behave correctly. My pipeline is now as follows,

GST_DEBUG=3 gst-launch-1.0 -e rtspsrc location="rtsp://" is-live=true protocols=tcp ! rtph264depay ! nvv4l2decoder ! video/x-raw\(memory:NVMM\) ! nvvidconv ! video/x-raw\(memory:NVMM\),width=256,height=144 ! nvvidconv ! video/x-raw,format=GRAY8 ! multifilesink location="/tmp/%d.y"

This also seems to work on L4T 32.3.1 so perhaps the filter has always been redundant? I am still puzzled why the behaviour is different between the two versions of L4T.

I tried to refer back to the Accelerated GStreamer User Guide (Release 28.2 - which seems to be the latest version available) to look for examples of the correct way to use the filter but surprisingly there are no examples for the nvv4l2decoder filter only the omxh264dec which is going to be deprecated.


We are investigating on this. Will update when there is new finding.


Your pipeline lacks rtph264depay between rtspsrc and h264parse.

Thanks a lot.
1- when I comment ‘video/x-h264, stream-format=byte-stream !’ also work, Is this need?
2- what’s mean when we use :
video/x-raw, format=BGR and video/x-raw(memory:NVMM),format=(string)NV12 in elements? there are used for converting? or stograge? if so, are GPU buffer?
3- appsrc is used cpu buffer?
4- what’s about video/x-h264, stream-format=byte-stream ? GPU-buffer??

A gstreamer pipeline is a sequence of gstreamer plugins, from (at least) a source plugin to (at least) a sink plugin.
Each plugin has either SRC capabilities, or SINK capabilities, or both in most cases.
You would use gst-inspect-1.0 command for listing all plugins that your current gstreamer install knows:


or for getting details about a given plugin, for example:

gst-inspect-1.0 nvvidconv

The latter will show SRC and SINK caps, so you would know depending on its SINK capabilities if a plugin can be plugged after another one (there should be at least one set of types/formats/… matching with previous SRC capabilities).
If no caps match previous plugin SRC capabilities and next plugin SINK caps, the pipeline would fail to link and start.
In some cases, various possibilities are available, and gstreamer framework could choose what it thinks is best, but that may not be the best solution for your case, so in this case you would specify wanted caps between plugins for ensuring gstreamer uses the types and formats you’re expecting.

If next plugin only expects video/x-h264, stream-format=byte-stream as SINK caps, this should make no difference.

The first case means BGR (888) video format in standard (CPU) allocated memory.
The second one means NV12 format in NVMM memory, that is contiguous DMA-able memory, suitable for argus, HW encoders/decoders, GPU processing.
nvvidconv plugin allow to copy/convert-format/rescale from NVMM to NVMM/CPU to NVMM/NVMM to CPU (but not CPU to CPU, although its caps don’t tell).

Yes, in most cases, especially if using opencv with a videoWriter, appsrc is a CPU application outputting BGR frames in CPU allocated memory.

No, it’s h264 encoded video, in CPU allocated memory, with byte-stream format suitable for RTP for example. Most containers would use avc format instead.


I have a bit similar issue. Please, can you change ... ! nvv4l2decoder ! ... to ... ! nvv4l2decoder enable-frame-type-reporting=true ! ... in your base pipeline. Does it works to you? And what in output?

I have checked now that both:

gst-launch-1.0 -v nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=640, height=480, framerate=30/1' ! nvv4l2h264enc ! h264parse ! nvv4l2decoder enable-frame-type-reporting=true ! videoconvert ! fpsdisplaysink video-sink= fakesink text-overlay=false
gst-launch-1.0 -v nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=640, height=480, framerate=30/1' ! nvv4l2h264enc ! h264parse ! nvv4l2decoder enable-error-check=true ! videoconvert ! fpsdisplaysink video-sink= fakesink text-overlay=false

still stall and make faults in argus upon double Ctrl-C. So it looks your issue is another one.

May be… Also, when i tried to encode and decode in the same pipeline, it only works as:

 nvv4l2h264enc ! h264parse ! tsmpegmux ! queue ! tsdemux ! h264parse ! nvv4l2decoder

Also, isn’t nvvidconv should be between nvarguscamerasrc and caps?

Not sure what you mean here…nvv4l2h264enc may expect input from NVMM memory, as nvarguscamerasrc provides, but adding nvvidconv in between should be harmless. Same, nvv4l2decoder would output into NVMM memory, so you would use nvvidconv after for copying back (may also convert format and resize) to CPU allocated memory.

As i know, nvvidconv (unlike videoconvert), can use NVMM. So yes, it should be harmless.and can fix some video format conflicts. Also it can transfer buffers from CPU allocated memory to NVMM.

IN cv2.videoCapture + gstreamer, first the decoded frames pushed into NVVM buffer and then copied into CPU buffer, right? because of opencv only accept CPU buffer, So, In this mode, for one frame decoding we use double memory from 4GB of jetson nano? (NVVM buffer + CPU buffer), I want to know because the deep stream don’t use opencv for decoding, and only use gstreamer for decoding the frames, when I used opencv + gstreamer and decoding the frames with HW, in this case in the same situation, opencv + gstreamer decoder used twice memory usage than deep stream decoder?

and what’s main difference between opencv+gstreamer and deep-stream decoders? CPU/RAM usage.

First, apologies to @benky58un for squatting this topic.

Only in case of a MIPI/CSI camera being accessed through ISP with argus. In most other cases such as with v4l2src not going through ISP, frames would be received in CPU allocated memory.

Yes, an application linked to opencv would only receive frames into CPU allocated cv::Mat using opencv videoio.
However, you can access NVMM buffers with gstreamer plugin nvivafilter. It is intended to perform CUDA operations on NVMM hosted frames, so you can use it with opencv CUDA. You would have to output RGBA frames from this plugin. You may have a look to this example.
Also note that you can directly access from gstreamer buffer.

There may be (from tail to head) a BGR buffer, a BGRx buffer, a NVMM NV12 buffer, and some H264 encoded/depayed/RTP buffers. For a 4K resolution, worst case being BGRx would be 33MB only, so it should be affordable to have a few copies (I have no Nano, but I think that even with Linux and Ubuntu you would have more than 3.5GB available for your application).

Thanks for all your contributions to this community @Honey_Patouceul. You have assisted me personally a few times in the last few months - so a big thank you to you.

I don’t mind the activity on this thread. At least it keeps it active and at the top of the forums leading hopefully to a solution to the nvv4l2decoder issue.

@DaneLLL is there a public issue tracker that we can access to more closely track the progress of the investigation into the nvv4l2decoder issue?

1 Like

We are investigating it internally. Will update when there is new finding.

You suggested this link, Is it efficient for decoding the rtsp? It use double memory from main RAM? NVVM buffer + CPU buffer

When the decoded frame pushed into NVMM buffer and then copied into CPU buffer, So that frame in NVMM buffer then removed from this buffer?

1- In your opinion, Is this correctly work for jetson xavier nx? It’s work for jetson nano.

2- Is it possible to use the decoded of frames from NVMM buffer for processing without copy to CPU buffer? If so, How?

3- If I want to use the decoded of frames for TPU dongle that connected to jetson nano via USB I need to copy the decoded frames from NVMM buffer to CPU buffer? If is possible to use the decoded frames from nvvm buffer directly into USB TPU?

4- If I want to use very efficient mode for decoding the streams, I have to use pure gstreamer? How I can use the buffers of gsteamer in numpy array for processing?

I don’t know if it helps, but I have the same issue. And during testing I found out that if I open a second connection to the problematic camera with for instance VLC on a laptop, then decoding with nvv4l2decoder works.
I have attached a log of a working (ok) and non working situation (wrong)
output_v_ok.txt (20.9 KB)
output_v_wrong.txt (17.3 KB)

It seems that the udpsrc caps are detected differently. Might that be the issue?

If there is anything else I can test, please let me know.