Gstreamer pipeline inside docker no longer working with L4T 32.6.1

Hello guys, i am running gstreamer in nvcr.io/nvidia/l4t-base:r32.6.1 with --nvidia

Pipeline I am using worked like a charm in l4t-r32.4

Now it works on 2fps (instead of 16fps) , here is report from GST_DEBUG 3 :

lzzii@jtsnx:~$ docker run -d -v /vcr:/vcr  --network host 156837977579.dkr.ecr.eu-north-1.amazonaws.com/vcr:pcrdev-466
797b738e803b0d929d3007b058b81671f7c800e64cd3dbfa91871bb5603ecb0a
lzzii@jtsnx:~$ docker logs -f 797b738e803b0d929d3007b058b81671f7c800e64cd3dbfa91871bb5603ecb0a
I0816 06:36:44.242964 547575914512 lpr_docker.py:47] Connection Redis<ConnectionPool<Connection<host=localhost,port=6379,db=0>>>
I0816 06:36:44.266714 547575914512 lpr_docker.py:57] Warmup 150 frames
I0816 06:36:46.025620 547575914512 lpr_docker.py:36] Writing video to /vcr/basler_2021-08-16_06-36-46_025.mkv
0:00:00.158670228    10   0x558795d470 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x120000: 'AVR (Audio Visual Research)' is not mapped
0:00:00.158728667    10   0x558795d470 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x180000: 'CAF (Apple Core Audio File)' is not mapped
0:00:00.158746845    10   0x558795d470 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x100000: 'HTK (HMM Tool Kit)' is not mapped
0:00:00.158765752    10   0x558795d470 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0xc0000: 'MAT4 (GNU Octave 2.0 / Matlab 4.2)' is not mapped
0:00:00.158780961    10   0x558795d470 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0xd0000: 'MAT5 (GNU Octave 2.1 / Matlab 5.0)' is not mapped
0:00:00.158795492    10   0x558795d470 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x210000: 'MPC (Akai MPC 2k)' is not mapped
0:00:00.158812784    10   0x558795d470 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0xe0000: 'PVF (Portable Voice Format)' is not mapped
0:00:00.158829764    10   0x558795d470 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x160000: 'SD2 (Sound Designer II)' is not mapped
0:00:00.158852317    10   0x558795d470 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x190000: 'WVE (Psion Series 3)' is not mapped
0:00:00.229333916    10   0x558795d470 WARN                  ladspa gstladspa.c:507:plugin_init:<plugin104> no LADSPA plugins found, check LADSPA_PATH
(Argus) Error FileOperationFailed: Connecting to nvargus-daemon failed: No such file or directory (in src/rpc/socket/client/SocketClientDispatch.cpp, function openSocketConnection(), line 205)
(Argus) Error FileOperationFailed: Cannot create camera provider (in src/rpc/socket/client/SocketClientDispatch.cpp, function createCameraProvider(), line 106)
0:00:00.849165752    10   0x558795d470 WARN                     omx gstomx.c:2826:plugin_init: Failed to load configuration file: Valid key file could not be found in search dirs (searched in: /root/.config:/etc/xdg as per GST_OMX_CONFIG_DIR environment variable, the xdg user config directory (or XDG_CONFIG_HOME) and the system config directory (or XDG_CONFIG_DIRS)
Setting pipeline to PAUSED ...
0:00:01.288611767     9   0x55bbe67290 WARN                    v4l2 gstv4l2object.c:2388:gst_v4l2_object_add_interlace_mode:0x55bbf64200 Failed to determine interlace mode
0:00:01.288700155     9   0x55bbe67290 WARN                    v4l2 gstv4l2object.c:2388:gst_v4l2_object_add_interlace_mode:0x55bbf64200 Failed to determine interlace mode
0:00:01.288742396     9   0x55bbe67290 WARN                    v4l2 gstv4l2object.c:2388:gst_v4l2_object_add_interlace_mode:0x55bbf64200 Failed to determine interlace mode
0:00:01.288784064     9   0x55bbe67290 WARN                    v4l2 gstv4l2object.c:2388:gst_v4l2_object_add_interlace_mode:0x55bbf64200 Failed to determine interlace mode
0:00:01.288911360     9   0x55bbe67290 WARN                    v4l2 gstv4l2object.c:4447:gst_v4l2_object_probe_caps:<nvv4l2h265enc0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1
Opening in BLOCKING MODE 
Pipeline is PREROLLING ...
I0816 06:36:47.345232 547575914512 lpr_docker.py:111] Framerate: 0.8 fps
Redistribute latency...
NvMMLiteOpen : Block : BlockType = 8 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 8 
0:00:01.313378661     9   0x55bc028d40 WARN          v4l2bufferpool gstv4l2bufferpool.c:1065:gst_v4l2_buffer_pool_start:<nvv4l2h265enc0:pool:src> Uncertain or not enough buffers, enabling copy threshold
NVMEDIA: H265 : Profile : 1 
NVMEDIA_ENC: bBlitMode is set to TRUE 
I0816 06:36:47.379323 547575914512 lpr_docker.py:111] Framerate: 1.5 fps
I0816 06:36:47.406226 547575914512 lpr_docker.py:111] Framerate: 2.2 fps
I0816 06:36:47.430921 547575914512 lpr_docker.py:111] Framerate: 2.9 fps
I0816 06:36:57.576690 547575914512 lpr_docker.py:111] Framerate: 0.4 fps

and pipeline looks like this

    args = shlex.split(
        "gst-launch-1.0 fdsrc do-timestamp=true \
        ! rawvideoparse format=gray8 width={width} height={height} \
        ! nvvidconv \
        ! nvv4l2h265enc bitrate={bitrate} \
        ! h265parse \
        ! matroskamux streamable=true \
        ! filesink location={filename}".format(
            height=img.shape[0],
            width=img.shape[1],
            bitrate=(FLAGS.mbps * 1000000),
            filename=get_filename(FLAGS.dir),
        )
    )

what is different?

also this error is strange, since i am not using nvargus :

(Argus) Error FileOperationFailed: Connecting to nvargus-daemon failed: No such file or directory (in src/rpc/socket/client/SocketClientDispatch.cpp, function openSocketConnection(), line 205)
(Argus) Error FileOperationFailed: Cannot create camera provider (in src/rpc/socket/client/SocketClientDispatch.cpp, function createCameraProvider(), line 106)

but can be solved by mounting argus socket to container

-v /tmp/argus_socket:/tmp/argus_socket 

Hi,
If it works by adding argus_socket, we would suggest add it. If you don’t need Argus, please remove libv4l2_nvargus.so from l4t.csv and try again:

lib, /usr/lib/aarch64-linux-gnu/tegra/libv4l2_nvargus.so

hey, so i found out that actually problem is with my custom rootfs.
when i do use jetpack 4.6 sdcard, it works.
however this is strange, since all ma AI workloads are running okay

i have installed following nvidia-jetpack components to my custom rootfs (based on ubuntu 18.04)

nvidia-cuda
nvidia-cudnn8
nvidia-tensorrt
nvidia-container
nvidia-visionworks

can please please someone point me to packages which are required to have in rootfs in order to run gstreamer in nvidia-docker?

i am using nvcr.io/nvidia/l4t-base:r32.6.1 image as base for my gstreamer projects.

And those gstreamer docker images of mine works on jetpack4.6 sdcard image, but unfortunatelly not on my custom rootfs based on ubuntu18.04, which i have created according to l4t documentation (apply_binaries.sh and so on to rootfs)

i have been trying to make it work unfortunatelly without any luck
also i do have following csv files in /etc/nvidia-container-runtime/host-files-for-container.d/

lzzii@jtsnx:/etc/nvidia-container-runtime/host-files-for-container.d$ ls
cuda.csv  cudnn.csv  l4t.csv  tensorrt.csv

or at least please point me to binaries/libraries which are mounted via containerd to l4t-base image, so i can check if i have them all and they are not 0 size

Hi,
All gstreamer libraries are listed in gstreamer user guide. Please refer to it.

thanks for response,
i am fighting with this, but still no luck. I have even moved back to basics, to investigate issue. what i am doing rn is :

i am running docker image using this command

docker run -it --runtime nvidia nvcr.io/nvidia/l4t-base:r32.6.1 bash

and inside container executing test pipeline

root@e070cb3393b4:/# gst-launch-1.0 videotestsrc ! \
>   'video/x-raw, format=(string)I420, width=(int)640, \
>   height=(int)480' ! omxh265enc ! filesink location=test.h265 -e
(Argus) Error FileOperationFailed: Connecting to nvargus-daemon failed: No such file or directory (in src/rpc/socket/client/SocketClientDispatch.cpp, function openSocketConnection(), line 205)
(Argus) Error FileOperationFailed: Cannot create camera provider (in src/rpc/socket/client/SocketClientDispatch.cpp, function createCameraProvider(), line 106)
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Framerate set to : 30 at NvxVideoEncoderSetParameterNvMMLiteOpen : Block : BlockType = 8 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 8 
NVMEDIA: H265 : Profile : 1 
NVMEDIA_ENC: bBlitMode is set to TRUE 

and no matter if i do use pipeline with omxh265enc or nvv4l2h265enc
it always get stuck in prerolling, probably waiting on hvenc to do its work?, which will never happen, and resulting file is 0 size

i’ve checked all “l4t.csv” inside nvidia runtime, even permission on /dev/nvhost* devices on jetson nano machine, and i am unable to locate an issue.

where should i look?

GST_DEBUG=2 is not showing anything interesting

root@38c9a073159e:/# gst-launch-1.0 videotestsrc ! \
>   'video/x-raw, format=(string)I420, width=(int)640, \
> ^Cheight=(int)480' ! omxh265enc ! filesink location=test.h265 -e
root@38c9a073159e:/# GST_DEBUG=2 gst-launch-1.0 videotestsrc ! \
>   'video/x-raw, format=(string)I420, width=(int)640, \
>   height=(int)480' ! omxh265enc ! filesink location=test.h265 -e
0:00:00.044470461    11   0x5584740290 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x120000: 'AVR (Audio Visual Research)' is not mapped
0:00:00.044540358    11   0x5584740290 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x180000: 'CAF (Apple Core Audio File)' is not mapped
0:00:00.044561765    11   0x5584740290 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x100000: 'HTK (HMM Tool Kit)' is not mapped
0:00:00.044586140    11   0x5584740290 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0xc0000: 'MAT4 (GNU Octave 2.0 / Matlab 4.2)' is not mapped
0:00:00.044612391    11   0x5584740290 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0xd0000: 'MAT5 (GNU Octave 2.1 / Matlab 5.0)' is not mapped
0:00:00.044651610    11   0x5584740290 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x210000: 'MPC (Akai MPC 2k)' is not mapped
0:00:00.044686351    11   0x5584740290 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0xe0000: 'PVF (Portable Voice Format)' is not mapped
0:00:00.044715830    11   0x5584740290 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x160000: 'SD2 (Sound Designer II)' is not mapped
0:00:00.044742654    11   0x5584740290 WARN                 default gstsf.c:98:gst_sf_create_audio_template_caps: format 0x190000: 'WVE (Psion Series 3)' is not mapped
0:00:00.421750490    11   0x5584740290 WARN                  ladspa gstladspa.c:507:plugin_init:<plugin180> no LADSPA plugins found, check LADSPA_PATH
(Argus) Error FileOperationFailed: Connecting to nvargus-daemon failed: No such file or directory (in src/rpc/socket/client/SocketClientDispatch.cpp, function openSocketConnection(), line 205)
(Argus) Error FileOperationFailed: Cannot create camera provider (in src/rpc/socket/client/SocketClientDispatch.cpp, function createCameraProvider(), line 106)
0:00:00.788269226    11   0x5584740290 WARN                     omx gstomx.c:2826:plugin_init: Failed to load configuration file: Valid key file could not be found in search dirs (searched in: /root/.config:/etc/xdg as per GST_OMX_CONFIG_DIR environment variable, the xdg user config directory (or XDG_CONFIG_HOME) and the system config directory (or XDG_CONFIG_DIRS)
0:00:01.166977563    10   0x55c6d30600 WARN                     omx gstomx.c:2826:plugin_init: Failed to load configuration file: Valid key file could not be found in search dirs (searched in: /root/.config:/etc/xdg as per GST_OMX_CONFIG_DIR environment variable, the xdg user config directory (or XDG_CONFIG_HOME) and the system config directory (or XDG_CONFIG_DIRS)
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Framerate set to : 30 at NvxVideoEncoderSetParameterNvMMLiteOpen : Block : BlockType = 8 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 8 
0:00:01.190170650    10   0x55c6f14c00 WARN             omxvideoenc gstomxvideoenc.c:1860:gst_omx_video_enc_set_format:<omxh265enc-omxh265enc0> Error setting temporal_tradeoff 0 : Vendor specific error (0x00000001)
NVMEDIA: H265 : Profile : 1 
NVMEDIA_ENC: bBlitMode is set to TRUE 

trying simple pipeline like

gst-launch-1.0 videotestsrc ! nvvidconv ! nvoverlaysink

works, so it must be somehow related to hw encoder?

root@38c9a073159e:/# GST_DEBUG=2 gst-launch-1.0 videotestsrc ! nvvidconv ! nvoverlaysink              
0:00:00.085134437    26   0x556a120210 WARN                     omx gstomx.c:2826:plugin_init: Failed to load configuration file: Valid key file could not be found in search dirs (searched in: /root/.config:/etc/xdg as per GST_OMX_CONFIG_DIR environment variable, the xdg user config directory (or XDG_CONFIG_HOME) and the system config directory (or XDG_CONFIG_DIRS)
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock

inside container, i do have all devices present

root@38c9a073159e:/# ls -l /dev/nvhost-*
crw-rw---- 1 root video 506,  1 Aug 20 07:50 /dev/nvhost-as-gpu
crw-rw---- 1 root video 242,  0 Aug 20 07:50 /dev/nvhost-ctrl
crw-rw---- 1 root video 506,  2 Aug 20 07:50 /dev/nvhost-ctrl-gpu
crw-rw---- 1 root   994 506,  3 Aug 20 07:50 /dev/nvhost-dbg-gpu
crw-rw---- 1 root video 506,  0 Aug 20 07:50 /dev/nvhost-gpu
crw-rw---- 1 root video 242, 17 Aug 20 07:50 /dev/nvhost-msenc
crw-rw---- 1 root video 242,  9 Aug 20 07:50 /dev/nvhost-nvdec
crw-rw---- 1 root video 242, 21 Aug 20 07:50 /dev/nvhost-nvjpg
crw-rw---- 1 root   994 506,  4 Aug 20 07:50 /dev/nvhost-prof-gpu
crw-rw---- 1 root video 242, 13 Aug 20 07:50 /dev/nvhost-vic

so this really is very strange to me

and of course, my rootfs has nvidia-l4t-gstreamer package installed which apparently provides nvidia gstreamer plugins, and yes they are mounted via l4t.csv to mentioned docker image

lzzii@jtsnx:~$ dpkg -l |grep nvidia-l4t
ii  nvidia-l4t-3d-core                   32.6.1-20210726122000               arm64        NVIDIA GL EGL Package
ii  nvidia-l4t-apt-source                32.6.1-20210726122000               arm64        NVIDIA L4T apt source list debian package
ii  nvidia-l4t-bootloader                32.6.1-20210726122000               arm64        NVIDIA Bootloader Package
ii  nvidia-l4t-camera                    32.6.1-20210726122000               arm64        NVIDIA Camera Package
ii  nvidia-l4t-configs                   32.6.1-20210726122000               arm64        NVIDIA configs debian package
ii  nvidia-l4t-core                      32.6.1-20210726122000               arm64        NVIDIA Core Package
ii  nvidia-l4t-cuda                      32.6.1-20210726122000               arm64        NVIDIA CUDA Package
ii  nvidia-l4t-firmware                  32.6.1-20210726122000               arm64        NVIDIA Firmware Package
ii  nvidia-l4t-gputools                  32.6.1-20210726122000               arm64        NVIDIA dgpu helper Package
ii  nvidia-l4t-graphics-demos            32.6.1-20210726122000               arm64        NVIDIA graphics demo applications
ii  nvidia-l4t-gstreamer                 32.6.1-20210726122000               arm64        NVIDIA GST Application files
ii  nvidia-l4t-init                      32.6.1-20210726122000               arm64        NVIDIA Init debian package
ii  nvidia-l4t-initrd                    32.6.1-20210726122000               arm64        NVIDIA initrd debian package
ii  nvidia-l4t-jetson-io                 32.6.1-20210726122000               arm64        NVIDIA Jetson.IO debian package
ii  nvidia-l4t-kernel                    4.9.253-tegra-32.6.1-20210726122000 arm64        NVIDIA Kernel Package
ii  nvidia-l4t-kernel-dtbs               4.9.253-tegra-32.6.1-20210726122000 arm64        NVIDIA Kernel DTB Package
ii  nvidia-l4t-kernel-headers            4.9.253-tegra-32.6.1-20210726122000 arm64        NVIDIA Linux Tegra Kernel Headers Package
ii  nvidia-l4t-libvulkan                 32.6.1-20210726122000               arm64        NVIDIA Vulkan Loader Package
ii  nvidia-l4t-multimedia                32.6.1-20210726122000               arm64        NVIDIA Multimedia Package
ii  nvidia-l4t-multimedia-utils          32.6.1-20210726122000               arm64        NVIDIA Multimedia Package
ii  nvidia-l4t-oem-config                32.6.1-20210726122000               arm64        NVIDIA OEM-Config Package
ii  nvidia-l4t-tools                     32.6.1-20210726122000               arm64        NVIDIA Public Test Tools Package
ii  nvidia-l4t-wayland                   32.6.1-20210726122000               arm64        NVIDIA Wayland Package
ii  nvidia-l4t-weston                    32.6.1-20210726122000               arm64        NVIDIA Weston Package
ii  nvidia-l4t-x11                       32.6.1-20210726122000               arm64        NVIDIA X11 Package
ii  nvidia-l4t-xusb-firmware             32.6.1-20210726122000               arm64        NVIDIA USB Firmware Package

an update, that

adding is-live to videotestsrc will move it to playing state, however video file is still zero size

root@fbe8626f8a47:/# gst-launch-1.0 videotestsrc is-live=true ! video/x-raw, format=I420,width=1920, height=1080 ! omxh264enc ! qtmux ! filesink location=$FILE -e
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
Framerate set to : 30 at NvxVideoEncoderSetParameterNvMMLiteOpen : Block : BlockType = 4 
New clock: GstSystemClock
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
H264: Profile = 66, Level = 40 
NVMEDIA_ENC: bBlitMode is set to TRUE 
^Chandling interrupt.
Interrupt: Stopping pipeline ...
EOS on shutdown enabled -- Forcing EOS on the pipeline
Waiting for EOS...
Got EOS from element "pipeline0".
EOS received - stopping pipeline...
Execution ended after 0:01:42.328426976
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

I had to install all gstreamer packages on my custom rootfs :

sudo apt-get install gstreamer1.0-tools gstreamer1.0-alsa \
  gstreamer1.0-plugins-base gstreamer1.0-plugins-good \
  gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly \
  gstreamer1.0-libav

and it solved issue - gst pipelines now works from container.

but why? are these packages needed on host os in order to have it working?

i understand , that nvidia-l4t-gstreamer is needed, as it gets mounted to l4t-base:r32.6.1 container via nvidia runtime, but why gstreamer full stack must be present on jetson host os ?

Hi,
Could you try to install the packages inside docker? Some of the packages are basic. If you don’t have it in host OS, you would need to install these inside docker. Should at least need

gstreamer1.0-tools gstreamer1.0-alsa gstreamer1.0-plugins-base gstreamer1.0-plugins-good