Nvv4l2decoder driver bug error on Tesla T4

Hi, I’m currently having issues with DeepStream 5 when used on a server equipped with two T4 cards. When running a simple pipeline (The real pipeline is much more complex!):

DISPLAY=:0 GST_DEBUG=3 gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvideoconvert ! fakesink

This generates the following error:

  0:00:00.270630316 27402 0x55f5372cf370 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

I’ve tested different setups, and I’ve meet all the requirements for DeepStream, but still my pipeline doesn’t work as expected. Tested multiple installations using CUDA 10.2 and newer drivers but none works. At this point, DeepStream 5 seems UNUSABLE with Testa T4 GPUs when running the real pipeline we’re working the pipe stops playing after a couple of seconds (this changes randomly)

Setup:
• Hardware Platform (Jetson / GPU)
2x Tesla T4 GPUs
• DeepStream Version
5.0_5.0.0-1
• JetPack Version (valid for Jetson only)
NA
• TensorRT Version
TensorRT 7.0.0.11-ga-20191216
• NVIDIA GPU Driver Version (valid for GPU only)
440.33.01
• Operative System
Ubuntu Server x64 18.04.03 4.15.0-109-generic (Tested using also the desktop version with the 5+ kernel)
• CUDA VERSION
cuda_10.2.89_440.33.01_linux.run (Used this to install the 440+ driver required by DeepStream)
All CUDA samples work correctly.
• cuDDN VERSION
7.6.5.32 (Test passed)

UPDATE:
The process was reproduced using.

  • OS: Ubuntu 18.04.03 Kernel 4.15.0-109-generic (Minimal install with third party packages option enabled on install)
  • Driver: nvidia-driver-440/bionic-updates,bionic-security,now 440.100-0ubuntu0.18.04.1 amd64 [installed] (Which was the one recommended by the OS, when installed it automatically installed the 4.15 kernel. We switched grub to use this kernel and proceeded with upgrading the system after installing and rebooting with the driver)
  • GCC: gcc (Ubuntu 7.5.0-3 ubuntu1~18.04) 7.5.0

After this point we continued installing the samne packages mentioned above.
driver -> cuda-toolkit-10-2 -> cudnn (Tests passed) -> tensorrt (nv-tensorrt-repo-ubuntu1804-cuda10.2-trt7.0.0.11-ga-20191216_1-1_amd64.deb) -> gstreamer tools using apt install (Same as on the deepstream 5 guide) -> deepstream-5.0_5.0.0-1_amd64.deb

GST_DEBUG=3 gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvideoconvert ! fakesink

Still fails. Even after deleting gstreamer’s cache.

companalysis@dell-comp-analysis-2:~/repo$ GST_DEBUG=3 gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvideoconvert ! fakesink
0:00:00.017073517  7793 0x555b18339f00 WARN      GST_PLUGIN_LOADING gstplugin.c:792:_priv_gst_plugin_load_file_for_registry: module_open failed: libtrtserver.so: cannot open shared object file: No such file or directory

(gst-plugin-scanner:7793): GStreamer-WARNING **: 14:47:50.826: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so': libtrtserver.so: cannot open shared object file: No such file or directory
0:00:00.090947618  7793 0x555b18339f00 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x120000: 'AVR (Audio Visual Research)' is not mapped
0:00:00.090962215  7793 0x555b18339f00 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x180000: 'CAF (Apple Core Audio File)' is not mapped
0:00:00.090965814  7793 0x555b18339f00 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x100000: 'HTK (HMM Tool Kit)' is not mapped
0:00:00.090969433  7793 0x555b18339f00 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0xc0000: 'MAT4 (GNU Octave 2.0 / Matlab 4.2)' is not mapped
0:00:00.090972531  7793 0x555b18339f00 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0xd0000: 'MAT5 (GNU Octave 2.1 / Matlab 5.0)' is not mapped
0:00:00.090975651  7793 0x555b18339f00 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x210000: 'MPC (Akai MPC 2k)' is not mapped
0:00:00.091000616  7793 0x555b18339f00 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0xe0000: 'PVF (Portable Voice Format)' is not mapped
0:00:00.091004367  7793 0x555b18339f00 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x160000: 'SD2 (Sound Designer II)' is not mapped
0:00:00.091009146  7793 0x555b18339f00 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x190000: 'WVE (Psion Series 3)' is not mapped
0:00:00.141968739  7793 0x555b18339f00 WARN                  ladspa gstladspa.c:507:plugin_init:<plugin65> no LADSPA plugins found, check LADSPA_PATH
Setting pipeline to PAUSED ...
0:00:00.346729268  7792 0x55a94289f890 WARN                    v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
0:00:00.346747690  7792 0x55a94289f890 WARN                    v4l2 gstv4l2object.c:2921:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe minimum capture size for pixelformat MJPG
0:00:00.346753540  7792 0x55a94289f890 WARN                    v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
0:00:00.346759217  7792 0x55a94289f890 WARN                    v4l2 gstv4l2object.c:2927:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe maximum capture size for pixelformat MJPG
0:00:00.346772817  7792 0x55a94289f890 WARN                    v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
0:00:00.346777919  7792 0x55a94289f890 WARN                    v4l2 gstv4l2object.c:2921:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe minimum capture size for pixelformat MPG4
0:00:00.346782372  7792 0x55a94289f890 WARN                    v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
0:00:00.346787386  7792 0x55a94289f890 WARN                    v4l2 gstv4l2object.c:2927:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe maximum capture size for pixelformat MPG4
0:00:00.346797958  7792 0x55a94289f890 WARN                    v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
0:00:00.346802988  7792 0x55a94289f890 WARN                    v4l2 gstv4l2object.c:2921:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe minimum capture size for pixelformat H265
0:00:00.346807739  7792 0x55a94289f890 WARN                    v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
0:00:00.346812573  7792 0x55a94289f890 WARN                    v4l2 gstv4l2object.c:2927:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe maximum capture size for pixelformat H265
0:00:00.346822657  7792 0x55a94289f890 WARN                    v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
0:00:00.346827203  7792 0x55a94289f890 WARN                    v4l2 gstv4l2object.c:2921:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe minimum capture size for pixelformat H264
0:00:00.346831813  7792 0x55a94289f890 WARN                    v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:sink> Unable to try format: Unknown error -1
0:00:00.346836944  7792 0x55a94289f890 WARN                    v4l2 gstv4l2object.c:2927:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:sink> Could not probe maximum capture size for pixelformat H264
0:00:00.347078271  7792 0x55a94289f890 WARN                    v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:src> Unable to try format: Unknown error -1
0:00:00.347084228  7792 0x55a94289f890 WARN                    v4l2 gstv4l2object.c:2921:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:src> Could not probe minimum capture size for pixelformat NM12
0:00:00.347090634  7792 0x55a94289f890 WARN                    v4l2 gstv4l2object.c:3035:gst_v4l2_object_get_nearest_size:<nvv4l2decoder0:src> Unable to try format: Unknown error -1
0:00:00.347095433  7792 0x55a94289f890 WARN                    v4l2 gstv4l2object.c:2927:gst_v4l2_object_probe_caps_for_format:<nvv4l2decoder0:src> Could not probe maximum capture size for pixelformat NM12
0:00:00.347102334  7792 0x55a94289f890 WARN                    v4l2 gstv4l2object.c:2372:gst_v4l2_object_add_interlace_mode:0x55a9425364b0 Failed to determine interlace mode
0:00:00.347315299  7792 0x55a94289f890 WARN                 basesrc gstbasesrc.c:3583:gst_base_src_start_complete:<filesrc0> pad not activated yet
Pipeline is PREROLLING ...
0:00:00.347728447  7792 0x55a942968770 WARN                 qtdemux qtdemux_types.c:233:qtdemux_type_get: unknown QuickTime node type sgpd
0:00:00.347757371  7792 0x55a942968770 WARN                 qtdemux qtdemux_types.c:233:qtdemux_type_get: unknown QuickTime node type sbgp
0:00:00.347809751  7792 0x55a942968770 WARN                 qtdemux qtdemux.c:3031:qtdemux_parse_trex:<qtdemux0> failed to find fragment defaults for stream 1
0:00:00.347949292  7792 0x55a942968770 WARN                 qtdemux qtdemux.c:3031:qtdemux_parse_trex:<qtdemux0> failed to find fragment defaults for stream 2
0:00:00.545891203  7792 0x55a942968770 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:00.545930365  7792 0x55a942968770 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:00.545983281  7792 0x55a942968770 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:00.545996905  7792 0x55a942968770 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:00.546043430  7792 0x55a942968770 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:00.546056160  7792 0x55a942968770 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:00.546094370  7792 0x55a942968770 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:00.546105874  7792 0x55a942968770 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:00.546144162  7792 0x55a942968770 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:00.546156414  7792 0x55a942968770 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:00.546194256  7792 0x55a942968770 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:00.546206188  7792 0x55a942968770 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

0:00:00.546865622  7792 0x55a942968770 WARN            v4l2videodec gstv4l2videodec.c:1609:gst_v4l2_video_dec_decide_allocation:<nvv4l2decoder0> Duration invalid, not setting latency
0:00:00.546908901  7792 0x55a942968770 WARN          v4l2bufferpool gstv4l2bufferpool.c:1057:gst_v4l2_buffer_pool_start:<nvv4l2decoder0:pool:src> Uncertain or not enough buffers, enabling copy threshold
0:00:00.547846461  7792 0x7f7850013c00 WARN          v4l2bufferpool gstv4l2bufferpool.c:1535:gst_v4l2_buffer_pool_dqbuf:<nvv4l2decoder0:pool:src> Driver should never set v4l2_buffer.field to ANY
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:01.310476956
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

UPDATE 2:

The pipeline does run. When viewed using an Xorg server on an x11vnc instance we are able to visualize the video despite the error. On our pipelines we use the sync property on the nveglglessink sync=false to supress the timestamp warning.

We repeated the test using this property on the simple pipeline:

$ DISPLAY=:0 GST_DEBUG=1 gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvideoconvert ! nveglglessink sync=false

0:00:00.280056718  7821 0x558104cb1ca0 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:01.344413609
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

The video does not fully play when not in sync.

When using sync=true:

DISPLAY=:0 GST_DEBUG=1 gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvideoconvert ! nveglglessink sync=true

0:00:00.284750949  7843 0x557a2a0248a0 ERROR                   v4l2 gstv4l2object.c:2074:gst_v4l2_object_get_interlace_mode: Driver bug detected - check driver with v4l2-compliance from http://git.linuxtv.org/v4l-utils.git

Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:48.066704572
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

Our pipelines were built and tested using a Jetson Nano developer kit. On this device the sync=true | sync=false property still allows for the pipe to work and play until the video feed ends. Also, the driver bug error doesn’t appear on the Jetson Nano.

Jetson Nano is quite different to T4 cards. The HW is different. The error does impact the pipeline and it is under investigation now.

Thank you. Correct, without the sync=false flag on the nveglglessink it works as expected. Both platforms are different but there should exists a smoother transition process between development boards and production.

The fix will be delivered with next version Deepstream SDK.

I always get the driver bug warnings on GTX1660, using the latest stable drivers.