AGX Xavier - nvv4l2decoder not working

Hey,

i am using the jetson agx xavier with jetpack 4.9.140-tegra, deepstream 5.0 and tensorrt 7.1.0.16

I try to do inference using a v4l2 logitech webcam which provides an h264 stream.

My pipeline looks as following:

 gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-h264, width=1280, height=720, framerate=30/1 ! \
    h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1280 height=720 ! \
    nvinfer config-file-path=config_infer_primary_ssd.txt ! \
    nvvideoconvert ! nvdsosd ! nvegltransform !  nveglglessink sync=0

My output:

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-h264, width=1280, height=720, framerate=30/1 ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1280 height=720 ! nvinfer config-file-path= config_infer_primary_ssd.txt ! nvvideoconvert ! nvdsosd ! nvegltransform ! nveglglessink sync=0
Warn: ‘threshold’ parameter has been deprecated. Use ‘pre-cluster-threshold’ instead.
Setting pipeline to PAUSED …
Using winsys: x11
Opening in BLOCKING MODE
0:00:02.949437056 17622 0x557c1d1f60 INFO nvinfer gstnvinfer.cpp:602:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1577> [UID = 1]: deserialized trt engine from :/home/developer/deepstream/aravis/mobilenetv2/build-debayer_on_board_ssd-Desktop-Debug/ssd_mobilenetv2.uff_b1_gpu0_fp32.engine
INFO: [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT Input 3x300x300
1 OUTPUT kFLOAT NMS 1x100x7
2 OUTPUT kFLOAT NMS_1 1x1x1
0:00:02.949644608 17622 0x557c1d1f60 INFO nvinfer gstnvinfer.cpp:602:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1681> [UID = 1]: Use deserialized engine model: /home/developer/deepstream/aravis/mobilenetv2/build-debayer_on_board_ssd-Desktop-Debug/ssd_mobilenetv2.uff_b1_gpu0_fp32.engine
0:00:02.955842240 17622 0x557c1d1f60 INFO nvinfer gstnvinfer_impl.cpp:311:notifyLoadModelStatus: [UID 1]: Load new model:config_infer_primary_ssd.txt sucessfully
Pipeline is live and does not need PREROLL …
Got context from element ‘eglglessink0’: gst.egl.EGLDisplay=context, display=(GstEGLDisplay)NULL;
Setting pipeline to PLAYING …
New clock: GstSystemClock
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261

And then nothing happens. I need to CTRL + C two time to end the process.

I tried it on dGPU and it works without a problem.

If i change nvv4l2decoder to avdec_h264 the pipeline works but since the avdec_h264 element is not accelerated i have no use case for it.

Here the last lines of the Gstreamer Debug log:

0:00:04.347477280 19552 0x55ae1e88f0 DEBUG v4l2videodec gstv4l2videodec.c:648:gst_v4l2_video_dec_set_format: Setting format: video/x-h264, width=(int)1280, height=(int)720, framerate=(fraction)30/1, stream-format=(string)byte-stream, alignment=(string)au, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, profile=(string)constrained-baseline, level=(string)4
0:00:04.347493440 19552 0x55ae1e88f0 DEBUG GST_PADS gstpad.c:6258:gst_pad_get_task_state:>nvv4l2decoder0:src< pad has no task
0:00:04.347506400 19552 0x55ae1e88f0 DEBUG v4l2 gstv4l2object.c:4369:gst_v4l2_object_stop:>nvv4l2decoder0:sink< stopping
0:00:04.347517792 19552 0x55ae1e88f0 DEBUG v4l2 gstv4l2object.c:4377:gst_v4l2_object_stop:>nvv4l2decoder0:sink< deactivating pool
0:00:04.347531328 19552 0x55ae1e88f0 DEBUG bufferpool gstbufferpool.c:552:gst_buffer_pool_set_active:>nvv4l2decoder0:pool:sink< pool was in the right state
0:00:04.347561280 19552 0x55ae1e88f0 DEBUG bufferpool gstbufferpool.c:197:gst_buffer_pool_finalize:>nvv4l2decoder0:pool:sink< 0x7f10016450 finalize
0:00:04.347572192 19552 0x55ae1e88f0 DEBUG bufferpool gstbufferpool.c:552:gst_buffer_pool_set_active:>nvv4l2decoder0:pool:sink< pool was in the right state
0:00:04.347584224 19552 0x55ae1e88f0 DEBUG GST_POLL gstpoll.c:772:gst_poll_free: 0x7f1001f990: freeing
0:00:04.347657952 19552 0x55ae1e88f0 DEBUG v4l2 gstv4l2object.c:4369:gst_v4l2_object_stop:>nvv4l2decoder0:src< stopping

If someone needs the whole log, i will upload it.

PS:
My paths may look a bit confusing, just ignore that. I am looking forward to run this within an cpp application and thats just the folder where all needed files as the trt-engine etc are.

Thanks in advance

+1 - seeing this problem when using nvv4l2decoder with an rtspsrc. Works fine with avdec_h264. I’m on an NVidia Jetson Nano.

0:00:01.930659944 23983   0x7f3c0068a0 DEBUG               GST_PADS gstpad.c:4072:gst_pad_query:<nvv4l2decoder0:sink> sent query 0x7f2c00a6d0 (caps), result 1
0:00:01.930704997 23983   0x7f3c0068a0 DEBUG               GST_CAPS gstutils.c:3065:gst_pad_query_caps:<nvv4l2decoder0:sink> query returned video/x-h264, width=(int)960, height=(int)720, framerate=(fraction)30/1, pixel-aspect-ratio=(fraction)1/1, stream-format=(string)byte-stream, alignment=(string)au, parsed=(boolean)true, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, profile=(string)constrained-baseline, level=(string)3.1
0:00:01.930745519 23983   0x7f3c0068a0 DEBUG               GST_PADS gstpad.c:4072:gst_pad_query:<nvv4l2decoder0:sink> sent query 0x7f2c025ad0 (accept-caps), result 1
0:00:01.930773280 23983   0x7f3c0068a0 DEBUG                default gstutils.c:3146:gst_pad_query_accept_caps:<nvv4l2decoder0:sink> query returned 1
0:00:01.930802499 23983   0x7f3c0068a0 DEBUG           videodecoder gstvideodecoder.c:1384:gst_video_decoder_sink_event:<nvv4l2decoder0> received event 12814, caps
0:00:01.930848125 23983   0x7f3c0068a0 DEBUG           videodecoder gstvideodecoder.c:683:gst_video_decoder_setcaps:<nvv4l2decoder0> setcaps video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, pixel-aspect-ratio=(fraction)1/1, width=(int)960, height=(int)720, framerate=(fraction)30/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, profile=(string)constrained-baseline, level=(string)3.1
0:00:01.930897762 23983   0x7f3c0068a0 DEBUG           videodecoder gstvideodecoder.c:690:gst_video_decoder_setcaps:<nvv4l2decoder0> Checking if caps changed old video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, parsed=(boolean)true new video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, pixel-aspect-ratio=(fraction)1/1, width=(int)960, height=(int)720, framerate=(fraction)30/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, profile=(string)constrained-baseline, level=(string)3.1
0:00:01.930965784 23983   0x7f3c0068a0 DEBUG             video-info video-info.c:364:gst_video_info_from_caps: parsing caps video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, pixel-aspect-ratio=(fraction)1/1, width=(int)960, height=(int)720, framerate=(fraction)30/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, profile=(string)constrained-baseline, level=(string)3.1
0:00:01.931000525 23983   0x7f3c0068a0 DEBUG             video-info video-info.c:475:gst_video_info_from_caps: no colorimetry, using default
0:00:01.931048651 23983   0x7f3c0068a0 DEBUG           v4l2videodec gstv4l2videodec.c:648:gst_v4l2_video_dec_set_format:<nvv4l2decoder0> Setting format: video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, pixel-aspect-ratio=(fraction)1/1, width=(int)960, height=(int)720, framerate=(fraction)30/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, profile=(string)constrained-baseline, level=(string)3.1
0:00:01.931074016 23983   0x7f3c0068a0 DEBUG               GST_PADS gstpad.c:6258:gst_pad_get_task_state:<nvv4l2decoder0:src> pad has no task
0:00:01.931097611 23983   0x7f3c0068a0 DEBUG                   v4l2 gstv4l2object.c:4369:gst_v4l2_object_stop:<nvv4l2decoder0:sink> stopping
0:00:01.931119538 23983   0x7f3c0068a0 DEBUG                   v4l2 gstv4l2object.c:4377:gst_v4l2_object_stop:<nvv4l2decoder0:sink> deactivating pool
0:00:01.931144903 23983   0x7f3c0068a0 DEBUG             bufferpool gstbufferpool.c:552:gst_buffer_pool_set_active:<nvv4l2decoder0:pool:sink> pool was in the right state
0:00:01.931189644 23983   0x7f3c0068a0 DEBUG             bufferpool gstbufferpool.c:197:gst_buffer_pool_finalize:<nvv4l2decoder0:pool:sink> 0x7f7001ced0 finalize
0:00:01.931215738 23983   0x7f3c0068a0 DEBUG             bufferpool gstbufferpool.c:552:gst_buffer_pool_set_active:<nvv4l2decoder0:pool:sink> pool was in the right state
0:00:01.931241364 23983   0x7f3c0068a0 DEBUG               GST_POLL gstpoll.c:772:gst_poll_free: 0x7f2c009ed0: freeing
0:00:01.931314855 23983   0x7f3c0068a0 DEBUG                   v4l2 gstv4l2object.c:4369:gst_v4l2_object_stop:<nvv4l2decoder0:src> stopping

Hi,
For USB cameras, we can follow the steps in


to launch it. Usually we see YUV422(such as YUYV, UYVY) and MJPEG are listed in v4l2-ctl. Could you check if your camera has H264 listed? Would be great if you can also share model ID of the Logitech webcam.