Depstream:7.0-triton-arm-sbsa can not use nvvidconv

Please provide complete information as applicable to your setup.

• Hardware Platform (GPU)
• DeepStream Version: 7.0
• TensorRT Version
• NVIDIA GPU Driver Version: 535.183.06
• Issue Type: questions

I am using a Kunpeng 920 CPU (ARM64) with the Kylin Server OS(Base on RHEL 8) installed.

Part of my program code is:

nvvidconv = Gst.ElementFactory.make("nvvideoconvert", "convertor")
if not nvvidconv:
    sys.stderr.write(" Unable to create nvvidconv \n")

It runs well in the nvcr.io/nvidia/deepstream:7.0-samples-multiarch image (on x86 ,ubuntu2204),
but in the nvcr.io/nvidia/deepstream:7.0-triton-arm-sbsa image, it prints Unable to create nvvidconv.


My test

to enter the container(nvcr.io/nvidia/deepstream:7.0-triton-arm-sbsa

sudo docker run -it --rm --net=host --gpus all -e DISPLAY=$DISPLAY --device /dev/snd -v /tmp/.X11-unix/:/tmp/.X11-unix nvcr.io/nvidia/deepstream:7.0-triton-arm-sbsa bash

I ran:

gst-inspect-1.0 nvvideoconvert

then get error:

(gst-inspect-1.0:100): GStreamer-WARNING **: 08:47:39.420: External plugin loader failed. This most likely means that the plugin loader helper binary was not found or could not be run. You might need to set the GST_PLUGIN_SCANNER environment variable if your setup is unusual. This should normally not be required though.
GLib (gthread-posix.c): Unexpected error from C library during 'pthread_setspecific': Invalid argument.  Aborting.
Aborted (core dumped)

My question:

  1. Does the nvcr.io/nvidia/deepstream:7.0-triton-arm-sbsa image support ARM64 and RHEL 8 system as the host?
  2. If it does support, what is the cause of the above error, and how can it be resolved?

Since this is for a project, I look forward to your response. Thank you.

i add --privileged in docker command,then in container run gst-inspect-1.0 nvvideoconvert , the msg :

GLib (gthread-posix.c): Unexpected error from C library during 'pthread_setspecific': Invalid argument.  Aborting.
GLib (gthread-posix.c): Unexpected error from C library during 'pthread_setspecific': Invalid argument.  Aborting.

(gst-plugin-scanner:105): GStreamer-WARNING **: 01:55:57.625: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_udp.so': librivermax.so.0: cannot open shared object file: No such file or directory
EGL failed to initialize!  Exiting...
/bin/dash: 1: lsmod: not found
/bin/dash: 1: modprobe: not found

(gst-plugin-scanner:105): GStreamer-WARNING **: 01:55:57.661: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstchromaprint.so': libavcodec.so.58: cannot open shared object file: No such file or directory

(gst-plugin-scanner:105): GStreamer-WARNING **: 01:55:57.703: adding type GstEvent multiple times

(gst-plugin-scanner:105): GStreamer-WARNING **: 01:55:57.716: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstmpeg2dec.so': libmpeg2.so.0: cannot open shared object file: No such file or directory

(gst-plugin-scanner:105): GStreamer-WARNING **: 01:55:57.717: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstmpeg2enc.so': libmpeg2encpp-2.1.so.0: cannot open shared object file: No such file or directory

(gst-plugin-scanner:105): GStreamer-WARNING **: 01:55:57.719: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstmpg123.so': libmpg123.so.0: cannot open shared object file: No such file or directory

(gst-plugin-scanner:105): GStreamer-WARNING **: 01:55:57.729: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstopenmpt.so': libmpg123.so.0: cannot open shared object file: No such file or directory
No such element or plugin 'nvvideoconvert'

We haven’t tested this device, but I think it should work.

In addition to the GPU driver, nvidia-container-toolkit must also be installed correctly.

https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/index.html

Then try the following command line to start docker.

sudo docker run -it --rm --runtime=nvidia --network=host -e NVIDIA_DRIVER_CAPABILITIES=compute,utility,video,graphics --gpus all --privileged -e DISPLAY=:0 -v /tmp/.X11-unix:/tmp/.X11-unix -v /etc/X11:/etc/X11 nvcr.io/nvidia/deepstream:7.0-triton-arm-sbsa

Thank you for your reply

  1. nvidia-container-toolkit already installed
  2. when i run:
sudo docker run -it --rm --runtime=nvidia --network=host -e NVIDIA_DRIVER_CAPABILITIES=compute,utility,video,graphics --gpus all --privileged -e DISPLAY=:0 -v /tmp/.X11-unix:/tmp/.X11-unix -v /etc/X11:/etc/X11 nvcr.io/nvidia/deepstream:7.0-triton-arm-sbsa

the log show run success. but my app has a rtsp server, when i use VLC to play the rtsp url,the VLC screen is black, but the VLC time is increasing. ( It runs well in the nvcr.io/nvidia/deepstream:7.0-samples-multiarch image (on x86 ,ubuntu2204))

  1. Which GPU model do you use ?
  2. If you run the sampe apps such as deepstream_test1_app and deepstream3_test3_app, Are they work normally ?
  3. sbsa is more similar to jetson. Some plug-ins have some differences between x86 and arm.
    Can you share full log after running this command GST_DEBUG=3 ./your_app > log.log 2>&1 ?
  1. GPU model: T4
  2. the log:
    log.log (38.6 KB)
  1. I started the Docker container with the following command:
export DISPLAY=:0
xhost +
sudo docker run -it --runtime=nvidia --network=host -e NVIDIA_DRIVER_CAPABILITIES=compute,utility,video,graphics --gpus all --privileged -e DISPLAY=:0 -v /tmp/.X11-unix:/tmp/.X11-unix -v /etc/X11:/etc/X11 -v /home/cmt/deploy/bzds7-tunnel-fire-det/source:/root/apps/myapp nvcr.io/nvidia/deepstream:7.0-triton-arm-sbsa

Then I tested running the deepstream-test1-app program, and the log is as follows:

GLib (gthread-posix.c): Unexpected error from C library during 'pthread_setspecific': Invalid argument.  Aborting.
GLib (gthread-posix.c): Unexpected error from C library during 'pthread_setspecific': Invalid argument.  Aborting.

(gst-plugin-scanner:148): GStreamer-WARNING **: 02:02:16.718: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_udp.so': librivermax.so.0: cannot open shared object file: No such file or directory
libEGL warning: DRI3: Screen seems not DRI3 capable
libEGL warning: DRI2: failed to authenticate
Cannot load EGL 
/bin/dash: 1: lsmod: not found
/bin/dash: 1: modprobe: not found

(gst-plugin-scanner:148): GStreamer-WARNING **: 02:02:16.793: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstchromaprint.so': libavcodec.so.58: cannot open shared object file: No such file or directory

(gst-plugin-scanner:148): GStreamer-WARNING **: 02:02:16.841: adding type GstEvent multiple times

(gst-plugin-scanner:148): GStreamer-WARNING **: 02:02:16.855: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstmpeg2dec.so': libmpeg2.so.0: cannot open shared object file: No such file or directory

(gst-plugin-scanner:148): GStreamer-WARNING **: 02:02:16.855: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstmpeg2enc.so': libmpeg2encpp-2.1.so.0: cannot open shared object file: No such file or directory

(gst-plugin-scanner:148): GStreamer-WARNING **: 02:02:16.858: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstmpg123.so': libmpg123.so.0: cannot open shared object file: No such file or directory

(gst-plugin-scanner:148): GStreamer-WARNING **: 02:02:16.871: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstopenmpt.so': libmpg123.so.0: cannot open shared object file: No such file or directory
/bin/dash: 1: lsmod: not found
/bin/dash: 1: modprobe: not found
One element could not be created. Exiting.

please help me

1.T4 GPU is used for computational purposes, there is without display outs.

Therefore deepstream-test1-app must be modified and recompiled to run properly.
Other programs that use nv3dsink element are similar.

if(prop.integrated) {
    sink = gst_element_factory_make("nv3dsink", "nv3d-sink");
  } else {
#ifdef __aarch64__
    // sink = gst_element_factory_make ("nv3dsink", "nvvideo-renderer");
  sink = gst_element_factory_make ("fakesink", "nvvideo-renderer");
#else
    sink = gst_element_factory_make ("nveglglessink", "nvvideo-renderer");
#endif
  }
cd /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1
make CUDA_VER=12.2
./deepstream-test1-app /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264

2.Try the following command line, make sure the rtsp sink feature can run properly.

/opt/nvidia/deepstream/deepstream/user_additional_install.sh

gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264 ! h264parse ! nvv4l2decoder ! nvrtspoutsinkbin

3.From the log you uploaded, the rtsp server does not work properly, which may be caused by incorrect pipeline. You can try adding queue element

rtspserver rtsp-server.c:1285:gst_rtsp_server_io_func:<GstRTSPServer@0xaaadbd8be750>e[00m Could not accept client on socket 0xaaadbd8c0d10: Generic error


Not enough buffering available for  the processing deadline of 0:00:00.020000000, add enough queues to buffer  0:00:00.020000000 additional data. Shortening processing latency to 0:00:00.000000000.)
0:02:41.724575541 e[34m 3141e[00m 0xfffd8c002da0 e[33;01mWARN   e[00m e[00m            basesink gstbasesink.c:1249:gst_base_sink_query_latency:<multiudpsink0>e[00m warning: Pipeline construction is invalid, please add queues.
0:02:41.724613911 e[34m 3141e[00m 0xfffd8c002da0 e[33;01mWARN   e[00m e[00m            basesink gstbasesink.c:1249:gst_base_sink_query_latency:<multiudpsink0>e[00m warning: Not enough buffering available for  the processing deadline of 0:00:00.020000000, add enough queues to buffer  0:00:00.020000000 additional data. Shortening processing latency to 0:00:00.000000000.
0:02:41.724674552 e[34m 3141e[00m 0xfffd8c002da0 e[33;01mWARN   e[00m e[00m           rtspmedia rtsp-media.c:3281:default_handle_message:e[00m 0xfffd180033b0: got warning Pipeline construction is invalid, please add queues. (../libs/gst/base/gstbasesink.c(1249): gst_base_sink_query_latency (): /GstPipeline:media-pipeline/GstMultiUDPSink:multiudpsink0:

I find this error:

0:00:00.653828494   122 0xaaaafaa00210 ERROR     GST_PLUGIN_LOADING gstpluginloader.c:279:plugin_loader_replay_pending: Plugin file /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_udp.so failed to load. Blacklisting

the ldd:

root@localhost:~/apps/myapp# ldd /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_udp.so
        linux-vdso.so.1 (0x0000fffe41f80000)
        librivermax.so.0 => not found
        libnvdsbufferpool.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvdsbufferpool.so.1.0.0 (0x0000fffe41e40000)
        libgstbase-1.0.so.0 => /usr/lib/aarch64-linux-gnu/libgstbase-1.0.so.0 (0x0000fffe41db0000)
        libgstreamer-1.0.so.0 => /usr/lib/aarch64-linux-gnu/libgstreamer-1.0.so.0 (0x0000fffe41c40000)
        libgio-2.0.so.0 => /usr/lib/aarch64-linux-gnu/libgio-2.0.so.0 (0x0000fffe41a10000)
        libglib-2.0.so.0 => /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0 (0x0000fffe418b0000)
        libgobject-2.0.so.0 => /usr/lib/aarch64-linux-gnu/libgobject-2.0.so.0 (0x0000fffe41830000)
        libgstsdp-1.0.so.0 => /usr/lib/aarch64-linux-gnu/libgstsdp-1.0.so.0 (0x0000fffe41800000)
        libcuda.so.1 => /usr/lib/aarch64-linux-gnu/libcuda.so.1 (0x0000fffe3fbc0000)
        libm.so.6 => /usr/lib/aarch64-linux-gnu/libm.so.6 (0x0000fffe3fb20000)
        libc.so.6 => /usr/lib/aarch64-linux-gnu/libc.so.6 (0x0000fffe3f970000)
        /lib/ld-linux-aarch64.so.1 (0x0000fffe41f90000)
        libgstvideo-1.0.so.0 => /usr/lib/aarch64-linux-gnu/libgstvideo-1.0.so.0 (0x0000fffe3f8a0000)
        libnvbufsurface.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurface.so.1.0.0 (0x0000fffe3f7d0000)
        libgmodule-2.0.so.0 => /usr/lib/aarch64-linux-gnu/libgmodule-2.0.so.0 (0x0000fffe3f7b0000)
        libunwind.so.8 => /usr/lib/aarch64-linux-gnu/libunwind.so.8 (0x0000fffe3f770000)
        libdw.so.1 => /usr/lib/aarch64-linux-gnu/libdw.so.1 (0x0000fffe3f6b0000)
        libz.so.1 => /usr/lib/aarch64-linux-gnu/libz.so.1 (0x0000fffe3f680000)
        libmount.so.1 => /usr/lib/aarch64-linux-gnu/libmount.so.1 (0x0000fffe3f620000)
        libselinux.so.1 => /usr/lib/aarch64-linux-gnu/libselinux.so.1 (0x0000fffe3f5e0000)
        libpcre2-8.so.0 => /usr/lib/aarch64-linux-gnu/libpcre2-8.so.0 (0x0000fffe3f540000)
        libffi.so.8 => /usr/lib/aarch64-linux-gnu/libffi.so.8 (0x0000fffe3f520000)
        libgstrtp-1.0.so.0 => /usr/lib/aarch64-linux-gnu/libgstrtp-1.0.so.0 (0x0000fffe3f4e0000)
        libgstpbutils-1.0.so.0 => /usr/lib/aarch64-linux-gnu/libgstpbutils-1.0.so.0 (0x0000fffe3f480000)
        libdl.so.2 => /usr/lib/aarch64-linux-gnu/libdl.so.2 (0x0000fffe3f460000)
        librt.so.1 => /usr/lib/aarch64-linux-gnu/librt.so.1 (0x0000fffe3f440000)
        libpthread.so.0 => /usr/lib/aarch64-linux-gnu/libpthread.so.0 (0x0000fffe3f420000)
        liborc-0.4.so.0 => /usr/lib/aarch64-linux-gnu/liborc-0.4.so.0 (0x0000fffe3f380000)
        libnvrm_mem.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_mem.so (0x0000fffe3f360000)
        libnvrm_surface.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_surface.so (0x0000fffe3f340000)
        libnvrm_chip.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_chip.so (0x0000fffe3f320000)
        libEGL.so.1 => /usr/lib/aarch64-linux-gnu/libEGL.so.1 (0x0000fffe3f2f0000)
        libnvos.so => /usr/lib/aarch64-linux-gnu/tegra/libnvos.so (0x0000fffe3f2d0000)
        libnvbuf_fdmap.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_fdmap.so.1.0.0 (0x0000fffe3f2b0000)
        liblzma.so.5 => /usr/lib/aarch64-linux-gnu/liblzma.so.5 (0x0000fffe3f270000)
        libelf.so.1 => /usr/lib/aarch64-linux-gnu/libelf.so.1 (0x0000fffe3f240000)
        libbz2.so.1.0 => /usr/lib/aarch64-linux-gnu/libbz2.so.1.0 (0x0000fffe3f210000)
        libblkid.so.1 => /usr/lib/aarch64-linux-gnu/libblkid.so.1 (0x0000fffe3f1c0000)
        libgstaudio-1.0.so.0 => /usr/lib/aarch64-linux-gnu/libgstaudio-1.0.so.0 (0x0000fffe3f130000)
        libgsttag-1.0.so.0 => /usr/lib/aarch64-linux-gnu/libgsttag-1.0.so.0 (0x0000fffe3f0e0000)
        libGLdispatch.so.0 => /usr/lib/aarch64-linux-gnu/libGLdispatch.so.0 (0x0000fffe3ef50000)
        libstdc++.so.6 => /usr/lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000fffe3ed20000)
        libgcc_s.so.1 => /usr/lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000fffe3ecf0000)

The absence of the rivermax library in the system could indeed be causing the black screen issue. How can this be resolved?

This is not an error, It’s used for nvidia bluefield nic only.

Does this command line work properly?

root@localhost:/opt/nvidia/deepstream/deepstream-7.0# gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264 ! h264parse ! nvv4l2decoder ! nvrtspoutsinkbin
GLib (gthread-posix.c): Unexpected error from C library during 'pthread_setspecific': Invalid argument.  Aborting.
GLib (gthread-posix.c): Unexpected error from C library during 'pthread_setspecific': Invalid argument.  Aborting.

(gst-plugin-scanner:818): GStreamer-WARNING **: 05:24:43.145: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_udp.so': librivermax.so.0: cannot open shared object file: No such file or directory
libEGL warning: DRI3: Screen seems not DRI3 capable
libEGL warning: DRI2: failed to authenticate
Cannot load EGL 
/bin/dash: 1: lsmod: not found
/bin/dash: 1: modprobe: not found
/bin/dash: 1: lsmod: not found
/bin/dash: 1: modprobe: not found
Unable to create video pipelineWARNING: erroneous pipeline: could not link nvv4l2decoder0 to dsnvrtspoutbin0

It seems that there is a problem with the driver installation. Try reinstalling the aarch64 version of the driver.

If reinstalling does not solve the problem, it is recommended to consult Huawei.

root@localhost:~/apps/myapp# gst-launch-1.0 filesrc location=./sample_720p.h264 ! h264parse ! nvv4l2decoder ! nvrtspoutsinkbin
/bin/bash: line 1: lsmod: command not found
/bin/bash: line 1: modprobe: command not found

 *** dsnvrtspoutbin0: Launched RTSP Streaming at rtsp://localhost:8554/ds-test ***

Setting pipeline to PAUSED ...
Redistribute latency...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Redistribute latency...
Redistribute latency...
0:00:02.0 / 0:00:40.1 (5.0 %)
(gst-launch-1.0:339): GLib-GObject-CRITICAL **: 06:33:30.414: g_object_get_is_valid_property: object class 'GstUDPSrc' has no property named 'pt'

Execution was successful, but the screen is green in VLC.

Try use the software encoder.

 gst-launch-1.0 filesrc location=./sample_720p.h264 ! h264parse ! nvv4l2decoder ! nvrtspoutsinkbin enc-type=1

If ok, maybe some issues in the encoder driver or the system.

Can you share full log of the above command line after running touch /tmp/cuvidv4l2_logs.

touch /tmp/cuvidv4l2_logs can open the hardware encoder driver log.

gst-launch-1.0 filesrc location=./sample_720p.h264 ! h264parse ! nvv4l2decoder ! nvrtspoutsinkbin enc-type=1

green screen again,

the touch /tmp/cuvidv4l2_logs log:

cuvidv4l2_logs.log (113.5 KB)

This error log caused by your system.It doesn’t seem to be a deepstream problem, so let’s solve it first.

Setting pipeline to PAUSED ...
0:00:00.083199870  1084 0xaaac7cbd3a50 ERROR             rtspserver rtsp-server.c:1004:gst_rtsp_server_create_socket:<GstRTSPServer@0xaaac7cc05570> failed to create socket
0:00:00.083219080  1084 0xaaac7cbd3a50 ERROR             rtspserver rtsp-server.c:1374:gst_rtsp_server_create_source:<GstRTSPServer@0xaaac7cc05570> failed to create socket
0:00:00.083233341  1084 0xaaac7cbd3a50 ERROR             rtspserver rtsp-server.c:1420:gst_rtsp_server_attach:<GstRTSPServer@0xaaac7cc05570> failed to create watch: Error binding to address 0.0.0.0:8554: Address already in use

I have resolved the port issue, but the vlc screen is still green

cuvidv4l2_logs_0731.log (97.4 KB)

When I run the built-in plugins of GStreamer, everything works fine. However, once I use the plugins of DeepStream, the screen turns green.

gst-launch-1.0 filesrc location=./sample_720p.h264 ! h264parse  ! flvmux ! rtmpsink location="rtmp://192.168.31.10:1935/live/officecam01?sign=41db35390ddad33f83944f44b8b75ded"

It runs ok.

gst-launch-1.0 filesrc location=./sample_720p.h264 ! h264parse ! nvv4l2decoder ! nvrtspoutsinkbin enc-type=1

the vlc screen is green

please help me, thank you

This is a problem with the encoder/decoder driver.

Deepstream should be able to run on aarch64 architecture, but it has not been tested on redhat-based systems.

https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_Installation.html#dgpu-setup-for-redhat-enterprise-linux-rhel

Can you try to reinstall Ubuntu-22.04 OS?

i try the ubuntu 2204, screen is still green