Avdec_h264 works but nvv4l2decoder does not

decolon13 · February 7, 2024, 9:40pm

Im running on a NVIDIA Orin Nano Developer Kit - Jetpack 5.1.2 [L4T 35.4.1]

I’m trying to decode an h264 video from my connected Razer Kiyo Pro webcam. See below for its output capabilities.

When I use avdec_h264 in my pipeline, it all works, though this uses a good portion of my CPU (which i can not spare once this is integrated into my larger application)

gst-launch-1.0
v4l2src device=/dev/video1 ! video/x-h264, width=1920, height=1080, framerate=30/1 !
avdec_h264 !
nvvidconv !
x264enc tune=zerolatency ! h264parse ! rtph264pay config-interval=1 pt=126 !
udpsink host=192.168.8.179 port=10101

But when I run with nvv4l2decoder, the pipeline freezes with no errors

GST_DEBUG=3 gst-launch-1.0
v4l2src device=/dev/video1 ! video/x-h264, width=1920, height=1080, framerate=30/1 !
nvv4l2decoder ! ‘video/x-raw(memory:NVMM)’ ! nvvidconv ! video/x-raw,format=NV12 !
x264enc tune=zerolatency ! h264parse ! rtph264pay config-interval=1 pt=126 !
udpsink host=192.168.8.179 port=10101

Here is the output from the failing pipeline
GST_DEBUG=3 gst-launch-1.0 v4l2src device=/dev/video1 ! video/x-h264, width=1920, height=1080, framerate=30/1 ! nvv4l2decoder ! ‘video/x-raw(memory:NVMM)’ ! nvvidconv ! video/x-raw,format=NV12 ! x264enc tune=zerolatency ! h264parse ! rtph264pay config-interval=1 pt=126 ! udpsink host=192.168.8.179 port=10101
Setting pipeline to PAUSED …
Opening in BLOCKING MODE
0:00:00.341747144 46095 0xaaaaf0eb4150 WARN v4l2 gstv4l2object.c:4561:gst_v4l2_object_probe_caps:nvv4l2decoder0:src Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1
0:00:00.341808138 46095 0xaaaaf0eb4150 WARN v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xaaaaf0e7a020 Failed to determine interlace mode
0:00:00.341837547 46095 0xaaaaf0eb4150 WARN v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xaaaaf0e7a020 Failed to determine interlace mode
0:00:00.341862891 46095 0xaaaaf0eb4150 WARN v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xaaaaf0e7a020 Failed to determine interlace mode
0:00:00.341887660 46095 0xaaaaf0eb4150 WARN v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xaaaaf0e7a020 Failed to determine interlace mode
Pipeline is live and does not need PREROLL …
Setting pipeline to PLAYING …
New clock: GstSystemClock
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
0:00:00.363484452 46095 0xaaaaf0d77240 WARN v4l2bufferpool gstv4l2bufferpool.c:809:gst_v4l2_buffer_pool_start:v4l2src0:pool:src Uncertain or not enough buffers, enabling copy threshold
0:00:03.753000653 46095 0xaaaaf0d77240 WARN v4l2src gstv4l2src.c:978:gst_v4l2src_create: lost frames detected: count = 1 - ts: 0:00:03.409321483
0:00:06.817980627 46095 0xaaaaf0d77240 WARN v4l2src gstv4l2src.c:978:gst_v4l2src_create: lost frames detected: count = 1 - ts: 0:00:06.474416611
0:00:08.818281770 46095 0xaaaaf0d77240 WARN v4l2src gstv4l2src.c:978:gst_v4l2src_create: lost frames detected: count = 1 - ts: 0:00:08.474479643
0:00:10.818338806 46095 0xaaaaf0d77240 WARN v4l2src gstv4l2src.c:978:gst_v4l2src_create: lost frames detected: count = 1 - ts: 0:00:10.474544643
0:00:12.818435682 46095 0xaaaaf0d77240 WARN v4l2src gstv4l2src.c:978:gst_v4l2src_create: lost frames detected: count = 1 - ts: 0:00:12.474608803
0:00:14.818284131 46095 0xaaaaf0d77240 WARN v4l2src gstv4l2src.c:978:gst_v4l2src_create: lost frames detected: count = 1 - ts: 0:00:14.474678579

Why would the software decoder work but the hardware one not work when they have the same capabilities?

Here are the camera output options
beauceron@ubuntu:~$ v4l2-ctl --device=/dev/video1 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture

[0]: 'YUYV' (YUYV 4:2:2)
	Size: Discrete 1920x1080
		Interval: Discrete 0.017s (60.000 fps)
		Interval: Discrete 0.033s (30.000 fps)
		Interval: Discrete 0.042s (24.000 fps)
		Interval: Discrete 0.050s (20.000 fps)
		Interval: Discrete 0.067s (15.000 fps)
		Interval: Discrete 0.100s (10.000 fps)
		Interval: Discrete 0.200s (5.000 fps)
[1]: 'MJPG' (Motion-JPEG, compressed)
	Size: Discrete 1920x1080
		Interval: Discrete 0.017s (60.000 fps)
		Interval: Discrete 0.033s (30.000 fps)
		Interval: Discrete 0.042s (24.000 fps)
		Interval: Discrete 0.050s (20.000 fps)
		Interval: Discrete 0.067s (15.000 fps)
		Interval: Discrete 0.100s (10.000 fps)
		Interval: Discrete 0.200s (5.000 fps)
[2]: 'H264' (H.264, compressed)
	Size: Discrete 1920x1080
		Interval: Discrete 0.017s (60.000 fps)
		Interval: Discrete 0.033s (30.000 fps)
		Interval: Discrete 0.042s (24.000 fps)
		Interval: Discrete 0.050s (20.000 fps)
		Interval: Discrete 0.067s (15.000 fps)
		Interval: Discrete 0.100s (10.000 fps)
		Interval: Discrete 0.200s (5.000 fps)
[3]: 'NV12' (Y/CbCr 4:2:0)
	Size: Discrete 1920x1080
		Interval: Discrete 0.017s (60.000 fps)
		Interval: Discrete 0.033s (30.000 fps)
		Interval: Discrete 0.042s (24.000 fps)
		Interval: Discrete 0.050s (20.000 fps)
		Interval: Discrete 0.067s (15.000 fps)
		Interval: Discrete 0.100s (10.000 fps)
		Interval: Discrete 0.200s (5.000 fps)

DaneLLL · February 8, 2024, 1:22am

Hi,
Please add h264parse plugin and check if it works:

... ! h264parse ! nvv4l2decoder ! ...

decolon13 · February 8, 2024, 2:18am

Sadly that does not help, the pipeline still does not start. Here is the command and the output

GST_DEBUG=3 gst-launch-1.0 \

v4l2src device=/dev/video1 ! video/x-h264, width=1920, height=1080, framerate=30/1 !
h264parse ! nvv4l2decoder ! ‘video/x-raw(memory:NVMM)’ ! nvvidconv ! video/x-raw,format=NV12 !
x264enc tune=zerolatency ! h264parse ! rtph264pay config-interval=1 pt=126 !
udpsink host=192.168.8.179 port=10101
Setting pipeline to PAUSED …
Opening in BLOCKING MODE
0:00:00.331395162 48455 0xaaaabf121190 WARN v4l2 gstv4l2object.c:4561:gst_v4l2_object_probe_caps:nvv4l2decoder0:src Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1
0:00:00.331459836 48455 0xaaaabf121190 WARN v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xaaaabf0eb0c0 Failed to determine interlace mode
0:00:00.331488317 48455 0xaaaabf121190 WARN v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xaaaabf0eb0c0 Failed to determine interlace mode
0:00:00.331520894 48455 0xaaaabf121190 WARN v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xaaaabf0eb0c0 Failed to determine interlace mode
0:00:00.331540830 48455 0xaaaabf121190 WARN v4l2 gstv4l2object.c:2420:gst_v4l2_object_add_interlace_mode:0xaaaabf0eb0c0 Failed to determine interlace mode
Pipeline is live and does not need PREROLL …
Setting pipeline to PLAYING …
New clock: GstSystemClock
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
0:00:00.353146366 48455 0xaaaabefe0f60 WARN v4l2bufferpool gstv4l2bufferpool.c:809:gst_v4l2_buffer_pool_start:v4l2src0:pool:src Uncertain or not enough buffers, enabling copy threshold

I can get it working if I do this

GST_DEBUG=“3” gst-launch-1.0
v4l2src device=/dev/video1 ! video/x-h264, width=1920, height=1080, framerate=30/1 !
h264parse ! video/x-h264, stream-format=avc ! h264parse ! video/x-h264, stream-format=byte-stream ! nvv4l2decoder enable-max-performance=1 !
nvvidconv !
x264enc tune=zerolatency ! h264parse ! rtph264pay config-interval=1 pt=126 !
udpsink host=192.168.8.179 port=10101

and add the two h264parse elements, but that also eats up most of my CPU, so will not work.

decolon13 · February 8, 2024, 7:17pm

Since the double h264parse works (with a large CPU load) could it be that the camera H264 output is somehow not compatible with the nvv4l2decoder? From the caps it should be compatible, but are there other things which could make it not compatible?

decolon13 · February 9, 2024, 2:19am

It looks like my webcam is outputting H264 in the following format

Format : AVC
Format : AVC
Format/Info : Advanced Video Codec
Format/Url : http://developers.videolan.org/x264.html
Commercial name : AVC
Format profile : Main@L4.2
Format settings : CABAC / 1 Ref Frames
Format settings, CABAC : Yes
Format settings, CABAC : Yes
Format settings, Reference frames : 1
Format settings, Reference frames : 1 frame
Format settings, GOP : M=1, N=30

Is that compatible with the hardware decoder?

decolon13 · February 10, 2024, 5:36am

I tried with a different h264 webcam with no luck. This time with an Elgato Facecam Pro.

@DaneLLL Any thoughts?

DaneLLL · February 15, 2024, 2:17am

Hi,
Please run the command like this:
Gstreamer pipeline for capturing H264 from USB - #5 by DaneLLL
And attach the h264 stream. So that we can check further.

You may try the commands like:
Gstreamer pipeline for capturing H264 from USB - #7 by DaneLLL
And see whether it can be decoded.

decolon13 · February 20, 2024, 4:19am

Hi @DaneLLL Thanks for the response.

This issue has changed a bit from the last messages. Before, the pipeline would just freeze when using nvv4l2decoder. Then I updated to Jetpack 6.0 DP [L4T 36.2.0] and the pipeline started running, but outputs a jittery stream with lots of warnings.

I ran this command

gst-launch-1.0 v4l2src num-buffers=166 device=/dev/video0 ! video/x-h264, width=1920, height=1080, framerate=30/1 ! filesink location=dump.264

and saved this file
dump.264.zip (3.0 MB)

I also ran this command

GST_DEBUG=3 gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-h264, width=1920, height=1080, framerate=30/1 ! \
h264parse ! nvv4l2decoder enable-max-performance=1 ! nvvidconv ! video/x-raw,format=NV12 ! \
x264enc tune=zerolatency ! h264parse ! rtph264pay config-interval=1 pt=126 ! \
udpsink host=192.168.8.179 port=10101

based on your example items, and got the below output with the buffer not free warnings happening continuously until the pipeline is stopped.

GST_DEBUG=3 gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-h264, width=1920, height=1080, framerate=30/1 ! \
h264parse ! nvv4l2decoder enable-max-performance=1 ! nvvidconv ! video/x-raw,format=NV12 ! \
x264enc tune=zerolatency ! h264parse ! rtph264pay config-interval=1 pt=126 ! \
udpsink host=192.168.8.179 port=10101
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE
0:00:00.369101436  3919 0xaaaafe56cb00 WARN                    v4l2 gstv4l2object.c:4634:gst_v4l2_object_probe_caps:<nvv4l2decoder0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
NvMMLiteOpen : Block : BlockType = 261
NvMMLiteBlockCreate : Block : BlockType = 261
0:00:00.391896773  3919 0xaaaafe566cc0 WARN          v4l2bufferpool gstv4l2bufferpool.c:842:gst_v4l2_buffer_pool_start:<v4l2src0:pool0:src> Uncertain or not enough buffers, enabling copy threshold
0:00:00.738322364  3919 0xaaaafe566cc0 WARN          v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 0 is not free
0:00:00.839684491  3919 0xaaaafe566cc0 WARN                    v4l2 gstv4l2object.c:4634:gst_v4l2_object_probe_caps:<nvv4l2decoder0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1
Redistribute latency...
0:00:00.848922169  3919 0xaaaafe566cc0 WARN            v4l2videodec gstv4l2videodec.c:2252:gst_v4l2_video_dec_decide_allocation:<nvv4l2decoder0> Duration invalid, not setting latency
0:00:00.849450346  3919 0xaaaafe566cc0 WARN          v4l2bufferpool gstv4l2bufferpool.c:1114:gst_v4l2_buffer_pool_start:<nvv4l2decoder0:pool:src> Uncertain or not enough buffers, enabling copy threshold
0:00:00.850696371  3919 0xaaaafe566cc0 WARN          v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 1 is not free
0:00:00.851180803  3919 0xaaaafe566cc0 WARN          v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 2 is not free
0:00:00.851522126  3919 0xaaaafe566cc0 WARN          v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 3 is not free
0:00:00.851821080  3919 0xaaaafe566cc0 WARN          v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 0 is not free
0:00:00.863037894  3919 0xaaaafe566d80 WARN          v4l2bufferpool gstv4l2bufferpool.c:1565:gst_v4l2_buffer_pool_dqbuf:<nvv4l2decoder0:pool:src> Driver should never set v4l2_buffer.field to ANY
0:00:00.882202617  3919 0xaaaafe566cc0 WARN          v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 1 is not free
Redistribute latency...
0:00:00.917220913  3919 0xaaaafe566cc0 WARN          v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 2 is not free
0:00:00.947206916  3919 0xaaaafe566cc0 WARN          v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 3 is not free
0:00:00.982210812  3919 0xaaaafe566cc0 WARN          v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 0 is not free
0:00:01.017168242  3919 0xaaaafe566cc0 WARN          v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 1 is not free
0:00:01.047053283  3919 0xaaaafe566cc0 WARN          v4l2bufferpool gstv4l2bufferpool.c:477:gst_v4l2_buffer_pool_alloc_buffer:<v4l2src0:pool0:src> newly allocated buffer 2 is not free

DaneLLL · February 20, 2024, 4:38am

Hi,
We can decode the steam in gstreamer command and jetson_multimedia_api sample:

$ DISPLAY=:0 gst-launch-1.0 filesrc location= ~/dump.264 ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw ! xvimagesink

00_video_decode$ DISPLAY=:0 ./video_decode H264 ~/dump.264

Could you try the command:

$ DISPLAY=:0 gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-h264, width=1920, height=1080, framerate=30/1 ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw ! xvimagesink

We would expect it working by replacing filesrc with v4l2src. It is supposed to work if the dump stream can be well decoded.

decolon13 · February 22, 2024, 1:39am

Is there a way to do this with the Orin running in headless mode?

DaneLLL · February 29, 2024, 8:16am

Hi,
Please try

$ gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-h264, width=1920, height=1080, framerate=30/1 ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw,format=I420 ! multifilesink location=dump%05d.yuv

The decode frames will be saved to storage one by one. You can check the frames through YUV viewer.

system · March 27, 2024, 5:31am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
H.264 decoding failure Jetson Nano gstreamer	12	3677	October 18, 2021
Jetson Nano Video Decoder Not Working at 1080p60 Jetson Nano gstreamer , video	5	395	March 7, 2024
Jetson 4k Encoding -> Decoding Pipeline and latency Jetson Nano camera , gstreamer	20	6882	October 15, 2021
TX2 H264 RTSP Stream decoding issues Jetson TX2	27	11553	October 18, 2021
GStreamer - stream video to IP Jetson Nano gstreamer	12	5484	October 15, 2021
Making a 720p live video monitoring Jetson TK1	56	13136	October 18, 2021
"NvMediaParserParse Unsupported Codec" error when use deepstream-app DeepStream SDK	11	1885	October 12, 2021
Nvv4l2decoder does not work with certain rtsp video stream DeepStream SDK	13	2499	June 13, 2022
RTSP stream not work "Stream format not found, dropping the frame" DeepStream SDK rtsp , gstreamer , nvbugs	24	1560	January 3, 2024
Problems with streaming ( and receiving ) a simple usb-camera with jetson nano and gstreamer Jetson Nano camera , gstreamer	9	4305	October 15, 2021

Avdec_h264 works but nvv4l2decoder does not

Related topics