AGX Xavier - nvv4l2decoder not working

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

Hey,

i am using the jetson agx xavier with jetpack 4.9.140-tegra, deepstream 5.0 and tensorrt 7.1.0.16

I try to do inference using a v4l2 logitech webcam which provides an h264 stream.

My pipeline looks as following:

 gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-h264, width=1280, height=720, framerate=30/1 ! \
    h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1280 height=720 ! \
    nvinfer config-file-path=config_infer_primary_ssd.txt ! \
    nvvideoconvert ! nvdsosd ! nvegltransform !  nveglglessink sync=0

My output:

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-h264, width=1280, height=720, framerate=30/1 ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1280 height=720 ! nvinfer config-file-path= config_infer_primary_ssd.txt ! nvvideoconvert ! nvdsosd ! nvegltransform ! nveglglessink sync=0
Warn: ‘threshold’ parameter has been deprecated. Use ‘pre-cluster-threshold’ instead.
Setting pipeline to PAUSED …
Using winsys: x11
Opening in BLOCKING MODE
0:00:02.949437056 17622 0x557c1d1f60 INFO nvinfer gstnvinfer.cpp:602:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1577> [UID = 1]: deserialized trt engine from :/home/developer/deepstream/aravis/mobilenetv2/build-debayer_on_board_ssd-Desktop-Debug/ssd_mobilenetv2.uff_b1_gpu0_fp32.engine
INFO: [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT Input 3x300x300
1 OUTPUT kFLOAT NMS 1x100x7
2 OUTPUT kFLOAT NMS_1 1x1x1
0:00:02.949644608 17622 0x557c1d1f60 INFO nvinfer gstnvinfer.cpp:602:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1681> [UID = 1]: Use deserialized engine model: /home/developer/deepstream/aravis/mobilenetv2/build-debayer_on_board_ssd-Desktop-Debug/ssd_mobilenetv2.uff_b1_gpu0_fp32.engine
0:00:02.955842240 17622 0x557c1d1f60 INFO nvinfer gstnvinfer_impl.cpp:311:notifyLoadModelStatus: [UID 1]: Load new model:config_infer_primary_ssd.txt sucessfully
Pipeline is live and does not need PREROLL …
Got context from element ‘eglglessink0’: gst.egl.EGLDisplay=context, display=(GstEGLDisplay)NULL;
Setting pipeline to PLAYING …
New clock: GstSystemClock
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261

And then nothing happens. I need to CTRL + C two time to end the process.

I tried it on dGPU and it works without a problem.

If i change nvv4l2decoder to avdec_h264 the pipeline works but since the avdec_h264 element is not accelerated i have no use case for it.

Here the last lines of the Gstreamer Debug log:

0:00:04.347477280 19552 0x55ae1e88f0 DEBUG v4l2videodec gstv4l2videodec.c:648:gst_v4l2_video_dec_set_format: Setting format: video/x-h264, width=(int)1280, height=(int)720, framerate=(fraction)30/1, stream-format=(string)byte-stream, alignment=(string)au, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, profile=(string)constrained-baseline, level=(string)4
0:00:04.347493440 19552 0x55ae1e88f0 DEBUG GST_PADS gstpad.c:6258:gst_pad_get_task_state:>nvv4l2decoder0:src< pad has no task
0:00:04.347506400 19552 0x55ae1e88f0 DEBUG v4l2 gstv4l2object.c:4369:gst_v4l2_object_stop:>nvv4l2decoder0:sink< stopping
0:00:04.347517792 19552 0x55ae1e88f0 DEBUG v4l2 gstv4l2object.c:4377:gst_v4l2_object_stop:>nvv4l2decoder0:sink< deactivating pool
0:00:04.347531328 19552 0x55ae1e88f0 DEBUG bufferpool gstbufferpool.c:552:gst_buffer_pool_set_active:>nvv4l2decoder0:pool:sink< pool was in the right state
0:00:04.347561280 19552 0x55ae1e88f0 DEBUG bufferpool gstbufferpool.c:197:gst_buffer_pool_finalize:>nvv4l2decoder0:pool:sink< 0x7f10016450 finalize
0:00:04.347572192 19552 0x55ae1e88f0 DEBUG bufferpool gstbufferpool.c:552:gst_buffer_pool_set_active:>nvv4l2decoder0:pool:sink< pool was in the right state
0:00:04.347584224 19552 0x55ae1e88f0 DEBUG GST_POLL gstpoll.c:772:gst_poll_free: 0x7f1001f990: freeing
0:00:04.347657952 19552 0x55ae1e88f0 DEBUG v4l2 gstv4l2object.c:4369:gst_v4l2_object_stop:>nvv4l2decoder0:src< stopping

If someone needs the whole log, i will upload it.

PS:
My paths may look a bit confusing, just ignore that. I am looking forward to run this within an cpp application and thats just the folder where all needed files as the trt-engine etc are.

Thanks in advance

2

+1 - seeing this problem when using nvv4l2decoder with an rtspsrc. Works fine with avdec_h264. I’m on an NVidia Jetson Nano.

0:00:01.930659944 23983   0x7f3c0068a0 DEBUG               GST_PADS gstpad.c:4072:gst_pad_query:<nvv4l2decoder0:sink> sent query 0x7f2c00a6d0 (caps), result 1
0:00:01.930704997 23983   0x7f3c0068a0 DEBUG               GST_CAPS gstutils.c:3065:gst_pad_query_caps:<nvv4l2decoder0:sink> query returned video/x-h264, width=(int)960, height=(int)720, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, stream-format=(string)byte-stream, alignment=(string)au, parsed=(boolean)true, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, profile=(string)constrained-baseline, level=(string)3.1
0:00:01.930745519 23983   0x7f3c0068a0 DEBUG               GST_PADS gstpad.c:4072:gst_pad_query:<nvv4l2decoder0:sink> sent query 0x7f2c025ad0 (accept-caps), result 1
0:00:01.930773280 23983   0x7f3c0068a0 DEBUG                default gstutils.c:3146:gst_pad_query_accept_caps:<nvv4l2decoder0:sink> query returned 1
0:00:01.930802499 23983   0x7f3c0068a0 DEBUG           videodecoder gstvideodecoder.c:1384:gst_video_decoder_sink_event:<nvv4l2decoder0> received event 12814, caps
0:00:01.930848125 23983   0x7f3c0068a0 DEBUG           videodecoder gstvideodecoder.c:683:gst_video_decoder_setcaps:<nvv4l2decoder0> setcaps video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, pixel-aspect-ratio=(fraction)1/1, width=(int)960, height=(int)720, framerate=(fraction)30/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, profile=(string)constrained-baseline, level=(string)3.1
0:00:01.930897762 23983   0x7f3c0068a0 DEBUG           videodecoder gstvideodecoder.c:690:gst_video_decoder_setcaps:<nvv4l2decoder0> Checking if caps changed old video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, parsed=(boolean)true new video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, pixel-aspect-ratio=(fraction)1/1, width=(int)960, height=(int)720, framerate=(fraction)30/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, profile=(string)constrained-baseline, level=(string)3.1
0:00:01.930965784 23983   0x7f3c0068a0 DEBUG             video-info video-info.c:364:gst_video_info_from_caps: parsing caps video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, pixel-aspect-ratio=(fraction)1/1, width=(int)960, height=(int)720, framerate=(fraction)30/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, profile=(string)constrained-baseline, level=(string)3.1
0:00:01.931000525 23983   0x7f3c0068a0 DEBUG             video-info video-info.c:475:gst_video_info_from_caps: no colorimetry, using default
0:00:01.931048651 23983   0x7f3c0068a0 DEBUG           v4l2videodec gstv4l2videodec.c:648:gst_v4l2_video_dec_set_format:<nvv4l2decoder0> Setting format: video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, pixel-aspect-ratio=(fraction)1/1, width=(int)960, height=(int)720, framerate=(fraction)30/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, profile=(string)constrained-baseline, level=(string)3.1
0:00:01.931074016 23983   0x7f3c0068a0 DEBUG               GST_PADS gstpad.c:6258:gst_pad_get_task_state:<nvv4l2decoder0:src> pad has no task
0:00:01.931097611 23983   0x7f3c0068a0 DEBUG                   v4l2 gstv4l2object.c:4369:gst_v4l2_object_stop:<nvv4l2decoder0:sink> stopping
0:00:01.931119538 23983   0x7f3c0068a0 DEBUG                   v4l2 gstv4l2object.c:4377:gst_v4l2_object_stop:<nvv4l2decoder0:sink> deactivating pool
0:00:01.931144903 23983   0x7f3c0068a0 DEBUG             bufferpool gstbufferpool.c:552:gst_buffer_pool_set_active:<nvv4l2decoder0:pool:sink> pool was in the right state
0:00:01.931189644 23983   0x7f3c0068a0 DEBUG             bufferpool gstbufferpool.c:197:gst_buffer_pool_finalize:<nvv4l2decoder0:pool:sink> 0x7f7001ced0 finalize
0:00:01.931215738 23983   0x7f3c0068a0 DEBUG             bufferpool gstbufferpool.c:552:gst_buffer_pool_set_active:<nvv4l2decoder0:pool:sink> pool was in the right state
0:00:01.931241364 23983   0x7f3c0068a0 DEBUG               GST_POLL gstpoll.c:772:gst_poll_free: 0x7f2c009ed0: freeing
0:00:01.931314855 23983   0x7f3c0068a0 DEBUG                   v4l2 gstv4l2object.c:4369:gst_v4l2_object_stop:<nvv4l2decoder0:src> stopping
4

Hi,
For USB cameras, we can follow the steps in

to launch it. Usually we see YUV422(such as YUYV, UYVY) and MJPEG are listed in v4l2-ctl. Could you check if your camera has H264 listed? Would be great if you can also share model ID of the Logitech webcam.

5

Thanks for the reply.

The command
v4l2-ctl -d /dev/video0 --list-formats-ext
says: -‘YUYV’ (YUYV 4:2:2)
- ‘H264’ (H.264, compressed)
- ‘MJPG’ (Motion-JPEG, compressed)

I can use the YUY2 format as used in your pipeline but the throughput is not very high which results in low fps.
The camera i am using is Logitech Support + Download

6

For what it’s worth, for my issue, using decodebin right after rtspsrc instead of rtph264depay, h264parse, and nvv4l2decoder got things working properly. decodebin decides to use nvv4l2decoder behind the scenes anyway.

Dumping the pipeline, I see that decodebin added a few more caps filters here and there… it’s not clear to me what exactly I got wrong in my pipeline, but it appears that decodebin knows more than I do 😅

7

Hi,
Please set

$ export GST_DEBUG=*FACTORY*:4

and run the pipeline. It lists all plugins selected by decodebin.

9

Hi,

using decodebin leads to the same error.

Pipeline:

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-h264, width=1280, height=720, framerate=30/1 ! decodebin ! m.sink_0 nvstreammux name=m batch-size=1 width=1280 height=720 ! nvinfer config-file-path=config_infer_primary_ssd.txt ! nvvideoconvert ! nvdsosd ! nvegltransform ! nveglglessink sync=0

the output with export GST_DEBUG=FACTORY:4 set:

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-h264, width=1280, height=720, framerate=30/1 ! decodebin ! m.sink_0 nvstreammux name=m batch-size=1 width=1280 height=720 ! nvinfer config-file-path=config_infer_primary_ssd.txt ! nvvideoconvert ! nvdsosd ! nvegltransform ! nveglglessink sync=0
0:00:00.044479552 26525 0x55a72e5c40 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “v4l2src”
0:00:00.048636192 26525 0x55a72e5c40 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “decodebin”
0:00:00.049795936 26525 0x55a72e5c40 INFO GST_ELEMENT_FACTORY gstelementfactory.c:359:gst_element_factory_create: creating element “typefind” named “typefind”
0:00:00.080914976 26525 0x55a72e5c40 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “nvstreammux”
0:00:00.128260800 26525 0x55a72e5c40 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “nvinfer”
Warn: ‘threshold’ parameter has been deprecated. Use ‘pre-cluster-threshold’ instead.
0:00:00.130552992 26525 0x55a72e5c40 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “nvvideoconvert”
0:00:00.135535360 26525 0x55a72e5c40 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “nvdsosd”
0:00:00.137041472 26525 0x55a72e5c40 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “nvegltransform”
0:00:00.138273792 26525 0x55a72e5c40 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “nveglglessink”
0:00:00.138796736 26525 0x55a72e5c40 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “pipeline”
0:00:00.139049568 26525 0x55a72e5c40 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “capsfilter”
Setting pipeline to PAUSED …
Using winsys: x11
Deserialize yoloLayerV3 plugin: yolo_83
Deserialize yoloLayerV3 plugin: yolo_95
Deserialize yoloLayerV3 plugin: yolo_107
0:00:02.955728928 26525 0x55a72e5c40 INFO nvinfer gstnvinfer.cpp:602:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1577> [UID = 1]: deserialized trt engine from :/home/developer/deepstream/aravis/build-debayer_on_board-Desktop-Debug/model_b1_gpu0_int8.engine
INFO: [Implicit Engine Info]: layers num: 4
0 INPUT kFLOAT data 3x608x608
1 OUTPUT kFLOAT yolo_83 255x19x19
2 OUTPUT kFLOAT yolo_95 255x38x38
3 OUTPUT kFLOAT yolo_107 255x76x76
0:00:02.955909664 26525 0x55a72e5c40 INFO nvinfer gstnvinfer.cpp:602:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1681> [UID = 1]: Use deserialized engine model: /home/developer/deepstream/aravis/build-debayer_on_board-Desktop-Debug/model_b1_gpu0_int8.engine
0:00:02.959829120 26525 0x55a72e5c40 INFO nvinfer gstnvinfer_impl.cpp:311:notifyLoadModelStatus: [UID 1]: Load new model:config_infer_primary_yoloV3.txt sucessfully
Pipeline is live and does not need PREROLL …
Got context from element ‘eglglessink0’: gst.egl.EGLDisplay=context, display=(GstEGLDisplay)NULL;
Setting pipeline to PLAYING …
New clock: GstSystemClock
0:00:02.971584448 26525 0x55a764d720 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “h264parse”
0:00:02.972566976 26525 0x55a764d720 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “capsfilter”
0:00:02.976988480 26525 0x55a764d720 INFO GST_ELEMENT_FACTORY gstelementfactory.c:361:gst_element_factory_create: creating element “nvv4l2decoder”
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261

11

Hi,
This looks same as

Please try the pipeline without h264parse:

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-h264, width=1280, height=720, framerate=30/1 ! nvv4l2decoder ! nvoverlaysink
12

Without h264parse it is working. Thank you!

14

It’s not working without h264parse in AGX

15

Hi ag3hbk,

Please help to open a new topic. Thanks

16