Issues decoding RTSP stream using nvv4l2decoder with Jetpack 4.4

Hi,
We are investigating on this. Will update when there is new finding.

@LoveNvidia,

Your pipeline lacks rtph264depay between rtspsrc and h264parse.

Thanks a lot.
1- when I comment ‘video/x-h264, stream-format=byte-stream !’ also work, Is this need?
2- what’s mean when we use :
video/x-raw, format=BGR and video/x-raw(memory:NVMM),format=(string)NV12 in elements? there are used for converting? or stograge? if so, are GPU buffer?
3- appsrc is used cpu buffer?
4- what’s about video/x-h264, stream-format=byte-stream ? GPU-buffer??

A gstreamer pipeline is a sequence of gstreamer plugins, from (at least) a source plugin to (at least) a sink plugin.
Each plugin has either SRC capabilities, or SINK capabilities, or both in most cases.
You would use gst-inspect-1.0 command for listing all plugins that your current gstreamer install knows:

gst-inspect-1.0

or for getting details about a given plugin, for example:

gst-inspect-1.0 nvvidconv

The latter will show SRC and SINK caps, so you would know depending on its SINK capabilities if a plugin can be plugged after another one (there should be at least one set of types/formats/… matching with previous SRC capabilities).
If no caps match previous plugin SRC capabilities and next plugin SINK caps, the pipeline would fail to link and start.
In some cases, various possibilities are available, and gstreamer framework could choose what it thinks is best, but that may not be the best solution for your case, so in this case you would specify wanted caps between plugins for ensuring gstreamer uses the types and formats you’re expecting.

If next plugin only expects video/x-h264, stream-format=byte-stream as SINK caps, this should make no difference.

The first case means BGR (888) video format in standard (CPU) allocated memory.
The second one means NV12 format in NVMM memory, that is contiguous DMA-able memory, suitable for argus, HW encoders/decoders, GPU processing.
nvvidconv plugin allow to copy/convert-format/rescale from NVMM to NVMM/CPU to NVMM/NVMM to CPU (but not CPU to CPU, although its caps don’t tell).

Yes, in most cases, especially if using opencv with a videoWriter, appsrc is a CPU application outputting BGR frames in CPU allocated memory.

No, it’s h264 encoded video, in CPU allocated memory, with byte-stream format suitable for RTP for example. Most containers would use avc format instead.

Hello!

I have a bit similar issue. Please, can you change ... ! nvv4l2decoder ! ... to ... ! nvv4l2decoder enable-frame-type-reporting=true ! ... in your base pipeline. Does it works to you? And what in output?

I have checked now that both:

gst-launch-1.0 -v nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=640, height=480, framerate=30/1' ! nvv4l2h264enc ! h264parse ! nvv4l2decoder enable-frame-type-reporting=true ! videoconvert ! fpsdisplaysink video-sink= fakesink text-overlay=false
#and
gst-launch-1.0 -v nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=640, height=480, framerate=30/1' ! nvv4l2h264enc ! h264parse ! nvv4l2decoder enable-error-check=true ! videoconvert ! fpsdisplaysink video-sink= fakesink text-overlay=false

still stall and make faults in argus upon double Ctrl-C. So it looks your issue is another one.

May be… Also, when i tried to encode and decode in the same pipeline, it only works as:

 nvv4l2h264enc ! h264parse ! tsmpegmux ! queue ! tsdemux ! h264parse ! nvv4l2decoder

Also, isn’t nvvidconv should be between nvarguscamerasrc and caps?

Not sure what you mean here…nvv4l2h264enc may expect input from NVMM memory, as nvarguscamerasrc provides, but adding nvvidconv in between should be harmless. Same, nvv4l2decoder would output into NVMM memory, so you would use nvvidconv after for copying back (may also convert format and resize) to CPU allocated memory.

As i know, nvvidconv (unlike videoconvert), can use NVMM. So yes, it should be harmless.and can fix some video format conflicts. Also it can transfer buffers from CPU allocated memory to NVMM.

@Honey_Patouceul
IN cv2.videoCapture + gstreamer, first the decoded frames pushed into NVVM buffer and then copied into CPU buffer, right? because of opencv only accept CPU buffer, So, In this mode, for one frame decoding we use double memory from 4GB of jetson nano? (NVVM buffer + CPU buffer), I want to know because the deep stream don’t use opencv for decoding, and only use gstreamer for decoding the frames, when I used opencv + gstreamer and decoding the frames with HW, in this case in the same situation, opencv + gstreamer decoder used twice memory usage than deep stream decoder?

and what’s main difference between opencv+gstreamer and deep-stream decoders? CPU/RAM usage.

First, apologies to @benky58un for squatting this topic.

Only in case of a MIPI/CSI camera being accessed through ISP with argus. In most other cases such as with v4l2src not going through ISP, frames would be received in CPU allocated memory.

Yes, an application linked to opencv would only receive frames into CPU allocated cv::Mat using opencv videoio.
However, you can access NVMM buffers with gstreamer plugin nvivafilter. It is intended to perform CUDA operations on NVMM hosted frames, so you can use it with opencv CUDA. You would have to output RGBA frames from this plugin. You may have a look to this example.
Also note that you can directly access from gstreamer buffer.

There may be (from tail to head) a BGR buffer, a BGRx buffer, a NVMM NV12 buffer, and some H264 encoded/depayed/RTP buffers. For a 4K resolution, worst case being BGRx would be 33MB only, so it should be affordable to have a few copies (I have no Nano, but I think that even with Linux and Ubuntu you would have more than 3.5GB available for your application).

Thanks for all your contributions to this community @Honey_Patouceul. You have assisted me personally a few times in the last few months - so a big thank you to you.

I don’t mind the activity on this thread. At least it keeps it active and at the top of the forums leading hopefully to a solution to the nvv4l2decoder issue.

@DaneLLL is there a public issue tracker that we can access to more closely track the progress of the investigation into the nvv4l2decoder issue?

1 Like

Hi,
We are investigating it internally. Will update when there is new finding.

@Honey_Patouceul
You suggested this link, Is it efficient for decoding the rtsp? It use double memory from main RAM? NVVM buffer + CPU buffer

When the decoded frame pushed into NVMM buffer and then copied into CPU buffer, So that frame in NVMM buffer then removed from this buffer?

@Honey_Patouceul
1- In your opinion, Is this correctly work for jetson xavier nx? It’s work for jetson nano.

2- Is it possible to use the decoded of frames from NVMM buffer for processing without copy to CPU buffer? If so, How?

3- If I want to use the decoded of frames for TPU dongle that connected to jetson nano via USB I need to copy the decoded frames from NVMM buffer to CPU buffer? If is possible to use the decoded frames from nvvm buffer directly into USB TPU?

4- If I want to use very efficient mode for decoding the streams, I have to use pure gstreamer? How I can use the buffers of gsteamer in numpy array for processing?

@DaneLLL
I don’t know if it helps, but I have the same issue. And during testing I found out that if I open a second connection to the problematic camera with for instance VLC on a laptop, then decoding with nvv4l2decoder works.
I have attached a log of a working (ok) and non working situation (wrong)
output_v_ok.txt (20.9 KB)
output_v_wrong.txt (17.3 KB)

It seems that the udpsrc caps are detected differently. Might that be the issue?

If there is anything else I can test, please let me know.

Hi,
If you upgrade to JP4.4.1(r32.4.4), you should be able to run with h264parse:

rtspsrc(or udpsrc) ! rtph264depay ! h264parse ! nvv4l2decoder ! ...

Please give it a try.

Ok, sorry, I though the issue was not fixed yet.

I am already on r32.4.4

R32 (release), REVISION: 4.4, GCID: 23942405, BOARD: t186ref, EABI: aarch64, DATE: Fri Oct 16 19:37:08 UTC 2020

So I probably have a different issue then.

Any other ideas how this can happen. I can reproduce it perfectly. Seems to be a race condition of some sort.

Hi,
For UDP streaming, you can refer to

If the issue is still present, please make a new post with steps so that we can replicate it and check.