Nvv4l2decoder got stuck in Deepstream6.1.1

user167252 · September 14, 2022, 8:25am

• Hardware Platform ( GPU): A30X
• DeepStream Version: 6.1.1
• TensorRT Version: 8.4.1.5
**• NVIDIA GPU Driver Version: 515.65.01 **
• Issue Type: questions

I started the DeepStream docker container (nvcr.io/nvidia/deepstream: 6.1.1-devel) and ran the following pipeline:

gst-launch-1.0 -e filesrc location=./xxx.mp4 !
qtdemux !
h264parse !
nvv4l2decoder !
fakesink dump=True

However, nvv4l2decoder got stuck in Deepstream6.1.1.
Do you know why nvv4l2decoder stucks?
Is there a problem with the execution environment?

The environment is:
Hardware Platform(GPU):A30X
docker image:nvcr.io/nvidia/deepstream:6.1.1-devel
OS:ubuntu20.04
cuda:11.7.1
GPU Driver:515.65.01
TensorRT Version:8.4.1.5

yingliu · September 15, 2022, 1:25pm

Is there any output after executing the command? You can add "GST_DEBUG=3 " before gst-launch-1.0 to enable more log to find more clues.

user167252 · September 16, 2022, 8:02am

The following log was output:.
Do you know why nvv4l2decoder got stuck?

GST_DEBUG=3 gst-launch-1.0 -e qtdemux Setting pipeline to PAUSED …
0:00:00.133263795 0:00:00.133291428 0:00:00.133296217 0:00:00.133300716 0:00:00.133312999 0:00:00.133317348 0:00:00.133324431 0:00:00.133328679 0:00:00.133337637 0:00:00.133343007 0:00:00.133347335 0:00:00.133352405 0:00:00.133360180 0:00:00.133364899 0:00:00.133369127 0:00:00.133374157 0:00:00.133383034 0:00:00.133389286 0:00:00.133395007 0:00:00.133401088 0:00:00.133412911 0:00:00.133418632 0:00:00.133422349 0:00:00.133427188 0:00:00.133436095 0:00:00.133441987 0:00:00.133447417 0:00:00.133452827 0:00:00.133459931 0:00:00.133465532 0:00:00.133470582 0:00:00.133475591 0:00:00.133485590 0:00:00.133492403 0:00:00.133498064 0:00:00.133502713 0:00:00.133554352 0:00:00.133560544 0:00:00.133566025 0:00:00.133571926 0:00:00.133934732 0:00:00.133942226 0:00:00.133947446 0:00:00.133952366 0:00:00.133960461 0:00:00.134204580 Pipeline is PREROLLING …
0:00:00.134566313 0:00:00.134684620 filesrc location=/opt/nvidia/deepstream/deepstream-6.1/samples/streams/sample_1080p_h264.mp4 !
! h264parse ! nvv4l2decoder ! fakesink dump=True
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat MJPG
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat MJPG
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat AV10
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat AV10
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat DVX5
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat DVX5
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat DVX4
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat DVX4
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat MPG4
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat MPG4
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat MPG2
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat MPG2
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat H265
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat H265
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat VP90
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat VP90
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat VP80
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat VP80
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe minimum capture size for pixelformat H264
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:sink Unable to try format: Unknown error -1
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:sink Could not probe maximum capture size for pixelformat H264
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:2942:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe minimum capture size for pixelformat NM12
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:3057:gst_v4l2_object_get_nearest_size:nvv4l2decoder0:src Unable to try format: Unknown error -1
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:2948:gst_v4l2_object_probe_caps_for_format:nvv4l2decoder0:src Could not probe maximum capture size for pixelformat NM12
70 0x55a64cc0f8c0 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0x55a64cc0eaf0 Failed to determine interlace mode
70 0x55a64cc0f8c0 WARN basesrc gstbasesrc.c:3600:gst_base_src_start_complete: pad not activated yet
70 0x55a64cbe06a0 WARN qtdemux qtdemux.c:3250:qtdemux_parse_trex: failed to find fragment defaults for stream 1
70 0x55a64cbe06a0 WARN qtdemux qtdemux.c:3250:qtdemux_parse_trex: failed to find fragment defaults for stream 2

tvai · September 16, 2022, 9:26am

Hi,

I have exactly the same problem with 2 A30 GPUs on the same server.
The kind of deadlock happens with driver 515.65.01 and also 515.48.07.
In my case I tried it with a h265 stream, but I guess it would happen with h264 too.
We are running DeepStream 6.0.1, but the same happens with the docker image of 6.1 and 5.1-21.02.

Using CPU decoder make my pipeline works, and of course it’s not an option.

We have another server with the same A30 GPU that is working fine with driver 510.47.03.

We haven’t tested yet to downgrade the driver version on the server running with 515.

gst-launch output gives me:

0:00:01.728621898   145 0x7f2a3c0084a0 DEBUG           v4l2videodec gstv4l2videodec.c:1740:gst_v4l2_video_dec_sink_getcaps:<nvv4l2decoder0> Returning sink caps video/x-h265, width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ], framerate=(fraction)[ 0/1, 2147483647/1 ], stream-format=(string)byte-stream, alignment=(string)au, parsed=(boolean)true
0:00:01.728800158   145 0x7f2a3c0084a0 DEBUG           v4l2videodec gstv4l2videodec.c:1740:gst_v4l2_video_dec_sink_getcaps:<nvv4l2decoder0> Returning sink caps video/x-h265, width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ], framerate=(fraction)[ 0/1, 2147483647/1 ], stream-format=(string)byte-stream, alignment=(string)au, parsed=(boolean)true
0:00:01.733844059   145 0x7f2a3c0084a0 DEBUG           v4l2videodec gstv4l2videodec.c:1740:gst_v4l2_video_dec_sink_getcaps:<nvv4l2decoder0> Returning sink caps video/x-h265, width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ], framerate=(fraction)[ 0/1, 2147483647/1 ], stream-format=(string)byte-stream, alignment=(string)au, parsed=(boolean)true
0:00:01.734314872   145 0x7f2a3c0084a0 DEBUG           v4l2videodec gstv4l2videodec.c:1740:gst_v4l2_video_dec_sink_getcaps:<nvv4l2decoder0> Returning sink caps video/x-h265, width=(int)2592, height=(int)1944, framerate=(fraction)25/1, stream-format=(string)byte-stream, alignment=(string)au, parsed=(boolean)true, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, profile=(string)main, tier=(string)main, level=(string)5
0:00:01.734414035   145 0x7f2a3c0084a0 DEBUG           v4l2videodec gstv4l2videodec.c:704:gst_v4l2_video_dec_set_format:<nvv4l2decoder0> Setting format: video/x-h265, stream-format=(string)byte-stream, alignment=(string)au, width=(int)2592, height=(int)1944, framerate=(fraction)25/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, profile=(string)main, tier=(string)main, level=(string)5
0:00:01.737964417   145 0x7f2a3c0084a0 DEBUG           v4l2videodec gstv4l2videodec.c:1349:gst_v4l2_video_dec_handle_frame:<nvv4l2decoder0> Handling frame 0
0:00:01.737994678   145 0x7f2a3c0084a0 DEBUG           v4l2videodec gstv4l2videodec.c:1389:gst_v4l2_video_dec_handle_frame:<nvv4l2decoder0> Sending header

Then nothing happens.

tvai · September 16, 2022, 11:02am

Downgrading to 510.85.02 made my pipeline work!
Something is bad with 515 and A30. (I have 515 with A100 and other GPUs working fine).

Amycao · September 30, 2022, 3:17am

The issue specific to A30 card. internal bug filed to track this issue.

tvai · October 3, 2022, 3:08pm

Thanks @Amycao, will you post here news when it’s fixed? just to know when we can update safely the drivers.

Amycao · October 8, 2022, 7:14am

Yes.

system · October 22, 2022, 7:15am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.