nVidia A100 with Deepstream 7 App run failure

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Ahoi everyone,

I hope you can help me out on my AI endeavour.
At work I have a DELL R7525 server with a nVidia A100 running on ESXi 8 with passthrough enabled, assigned to a Ubuntu Server VM running:

PRETTY_NAME=“Ubuntu 22.04.4 LTS”
NAME=“Ubuntu”
VERSION_ID=“22.04”
VERSION=“22.04.4 LTS (Jammy Jellyfish)”
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

The Ubuntu Server VM does not have a GUI installed - Terminal only

I followed this Deepstream SDK 7.0 setup for dGPU / Ubuntu: Quickstart Guide — DeepStream documentation 6.4 documentation (nvidia.com)

as well as followed Ubuntu 22.04 LTS (x86_64) — DeepStream documentation 6.4 documentation (nvidia.com) because of the A100 GPU.

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
nVidia A100 80GB PCIe
• DeepStream Version
Deepstream 7.0
• JetPack Version (valid for Jetson only)
nA
• TensorRT Version
dpkg-query -W “nvinfer”

libnvinfer-bin 10.1.0.27-1+cuda12.4
libnvinfer-dev 8.6.1.6-1+cuda12.0
libnvinfer-dev-cross-amd64
libnvinfer-dispatch-dev 8.6.1.6-1+cuda12.0
libnvinfer-dispatch-dev-cross-amd64
libnvinfer-dispatch10 10.1.0.27-1+cuda12.4
libnvinfer-dispatch8 8.6.1.6-1+cuda12.0
libnvinfer-doc
libnvinfer-headers-dev 8.6.1.6-1+cuda12.0
libnvinfer-headers-plugin-dev 8.6.1.6-1+cuda12.0
libnvinfer-lean-dev 8.6.1.6-1+cuda12.0
libnvinfer-lean-dev-cross-amd64
libnvinfer-lean10 10.1.0.27-1+cuda12.4
libnvinfer-lean8 8.6.1.6-1+cuda12.0
libnvinfer-plugin-dev 8.6.1.6-1+cuda12.0
libnvinfer-plugin-dev-cross-amd64
libnvinfer-plugin10 10.1.0.27-1+cuda12.4
libnvinfer-plugin8 8.6.1.6-1+cuda12.0
libnvinfer-samples 8.6.1.6-1+cuda12.0
libnvinfer-vc-plugin-dev 8.6.1.6-1+cuda12.0
libnvinfer-vc-plugin-dev-cross-amd64
libnvinfer-vc-plugin10 10.1.0.27-1+cuda12.4
libnvinfer-vc-plugin8 8.6.1.6-1+cuda12.0
libnvinfer10 10.1.0.27-1+cuda12.4
libnvinfer8 8.6.1.6-1+cuda12.0
python3-libnvinfer-dispatch 10.1.0.27-1+cuda12.4
python3-libnvinfer-lean 10.1.0.27-1+cuda12.4

• NVIDIA GPU Driver Version (valid for GPU only)
nvidia-smi

image
image900×576 15.6 KB

• Issue Type( questions, new requirements, bugs)
I’m connected to my Ubuntu Server 22.04 VM via SSH and configured the sample file as followed:

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl
 
[tiled-display]
enable=0
rows=2
columns=2
width=1280
height=720
gpu-id=0
#(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
#(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory, applicable for Tesla
#(2): nvbuf-mem-cuda-device - Allocate Device cuda memory, applicable for Tesla
#(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory, applicable for Tesla
#(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
nvbuf-memory-type=0
 
[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=file://../../streams/sample_1080p_h264.mp4
num-sources=4
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0
 
[sink0]
enable=0
#Type - 1=FakeSink 2=EglSink/nv3dsink (Jetson only) 3=File
type=1
sync=1
source-id=0
gpu-id=0
nvbuf-memory-type=0
 
[sink1]
enable=0
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=0
#1=mp4 2=mkv
container=1
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=0
#iframeinterval=10
bitrate=2000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
# set profile only for hw encoder, sw encoder selects profile based on sw-preset
profile=0
output-file=out.mp4
source-id=0
 
[sink2]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
#sw-preset=1 #for SW enc=(0)None (1)ultrafast (2)superfast (3)veryfast (4)faster
#(5)fast (6)medium (7)slow (8)slower (9)veryslow (10)placebo
sync=0
#iframeinterval=10
bitrate=400000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
# set profile only for hw encoder, sw encoder selects profile based on sw-preset
profile=0
# set below properties in case of RTSPStreaming
rtsp-port=8554
udp-port=5400
 
[osd]
enable=1
gpu-id=0
border-width=1
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0
 
[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=0
buffer-pool-size=4
batch-size=4
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
## Set muxer output width and height
width=1920
height=1080
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0
## If set to TRUE, system timestamp will be attached as ntp timestamp
## If set to FALSE, ntp timestamp from rtspsrc, if available, will be attached
# attach-sys-ts-as-ntp=1
 
# config-file property is mandatory for any gie section.
# Other properties are optional and if set will override the properties set in
# the infer config file.
[primary-gie]
enable=1
gpu-id=0
model-engine-file=../../models/Primary_Detector/resnet18_trafficcamnet.etlt_b4_gpu0_int8.engine
batch-size=4
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=0
gie-unique-id=1
nvbuf-memory-type=0
config-file=config_infer_primary.txt
 
[tracker]
enable=1
# For NvDCF and NvDeepSORT tracker, tracker-width and tracker-height must be a multiple of 32, respectively
tracker-width=960
tracker-height=544
ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
# ll-config-file required to set different tracker types
# ll-config-file=config_tracker_IOU.yml
# ll-config-file=config_tracker_NvSORT.yml
ll-config-file=config_tracker_NvDCF_perf.yml
# ll-config-file=config_tracker_NvDCF_accuracy.yml
# ll-config-file=config_tracker_NvDeepSORT.yml
gpu-id=0
display-tracking-id=1
 
[secondary-gie0]
enable=1
model-engine-file=../../models/Secondary_VehicleTypes/resnet18_vehicletypenet.etlt_b16_gpu0_int8.engine
gpu-id=0
batch-size=16
gie-unique-id=4
operate-on-gie-id=1
operate-on-class-ids=0;
config-file=config_infer_secondary_vehicletypes.txt
 
[secondary-gie1]
enable=1
model-engine-file=../../models/Secondary_VehicleMake/resnet18_vehiclemakenet.etlt_b16_gpu0_int8.engine
batch-size=16
gpu-id=0
gie-unique-id=5
operate-on-gie-id=1
operate-on-class-ids=0;
config-file=config_infer_secondary_vehiclemake.txt
 
[tests]
file-loop=0

When running a Deepstream Sample App i get the following error

ubuntu@nvidia-iot-edge:/opt/nvidia/deepstream/deepstream-7.0/samples/configs/deepstream-app$ sudo deepstream-app --gst-debug-level=2 -c source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt
 
(deepstream-app:2106): GLib-GObject-CRITICAL **: 13:59:14.913: g_object_set_is_valid_property: object class 'nvv4l2h264enc' has no property named 'preset-level'
 
(deepstream-app:2106): GLib-GObject-CRITICAL **: 13:59:14.913: g_object_set_is_valid_property: object class 'nvv4l2h264enc' has no property named 'insert-sps-pps'
 
*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8554/ds-test ***
 
Error: Could not get cuda device count (cudaErrorInitializationError)
Failed to parse group property
** ERROR: <gst_nvinfer_parse_config_file:1392>: failed
Error: Could not get cuda device count (cudaErrorInitializationError)
Failed to parse group property
** ERROR: <gst_nvinfer_parse_config_file:1392>: failed
Error: Could not get cuda device count (cudaErrorInitializationError)
Failed to parse group property
** ERROR: <gst_nvinfer_parse_config_file:1392>: failed
0:00:03.492472331  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:3100:gst_v4l2_object_get_nearest_size:<sink_sub_bin_encoder1:sink> Unable to try format: Unknown error -1
0:00:03.492500334  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:2985:gst_v4l2_object_probe_caps_for_format:<sink_sub_bin_encoder1:sink> Could not probe minimum capture size for pixelformat YM12
0:00:03.492509601  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:3100:gst_v4l2_object_get_nearest_size:<sink_sub_bin_encoder1:sink> Unable to try format: Unknown error -1
0:00:03.492516674  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:2991:gst_v4l2_object_probe_caps_for_format:<sink_sub_bin_encoder1:sink> Could not probe maximum capture size for pixelformat YM12
0:00:03.492525561  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:3100:gst_v4l2_object_get_nearest_size:<sink_sub_bin_encoder1:sink> Unable to try format: Unknown error -1
0:00:03.492532324  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:2985:gst_v4l2_object_probe_caps_for_format:<sink_sub_bin_encoder1:sink> Could not probe minimum capture size for pixelformat Y444
0:00:03.492536642  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:3100:gst_v4l2_object_get_nearest_size:<sink_sub_bin_encoder1:sink> Unable to try format: Unknown error -1
0:00:03.492542944  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:2991:gst_v4l2_object_probe_caps_for_format:<sink_sub_bin_encoder1:sink> Could not probe maximum capture size for pixelformat Y444
0:00:03.492553945  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:3100:gst_v4l2_object_get_nearest_size:<sink_sub_bin_encoder1:sink> Unable to try format: Unknown error -1
0:00:03.492558974  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:2985:gst_v4l2_object_probe_caps_for_format:<sink_sub_bin_encoder1:sink> Could not probe minimum capture size for pixelformat P410
0:00:03.492562170  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:3100:gst_v4l2_object_get_nearest_size:<sink_sub_bin_encoder1:sink> Unable to try format: Unknown error -1
0:00:03.492565977  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:2991:gst_v4l2_object_probe_caps_for_format:<sink_sub_bin_encoder1:sink> Could not probe maximum capture size for pixelformat P410
0:00:03.492574603  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:3100:gst_v4l2_object_get_nearest_size:<sink_sub_bin_encoder1:sink> Unable to try format: Unknown error -1
0:00:03.492580895  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:2985:gst_v4l2_object_probe_caps_for_format:<sink_sub_bin_encoder1:sink> Could not probe minimum capture size for pixelformat PM10
0:00:03.492584191  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:3100:gst_v4l2_object_get_nearest_size:<sink_sub_bin_encoder1:sink> Unable to try format: Unknown error -1
0:00:03.492587918  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:2991:gst_v4l2_object_probe_caps_for_format:<sink_sub_bin_encoder1:sink> Could not probe maximum capture size for pixelformat PM10
0:00:03.492593659  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:3100:gst_v4l2_object_get_nearest_size:<sink_sub_bin_encoder1:sink> Unable to try format: Unknown error -1
0:00:03.492599720  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:2985:gst_v4l2_object_probe_caps_for_format:<sink_sub_bin_encoder1:sink> Could not probe minimum capture size for pixelformat NM12
0:00:03.492602946  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:3100:gst_v4l2_object_get_nearest_size:<sink_sub_bin_encoder1:sink> Unable to try format: Unknown error -1
0:00:03.492607255  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:2991:gst_v4l2_object_probe_caps_for_format:<sink_sub_bin_encoder1:sink> Could not probe maximum capture size for pixelformat NM12
0:00:03.492638313  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:3100:gst_v4l2_object_get_nearest_size:<sink_sub_bin_encoder1:src> Unable to try format: Unknown error -1
0:00:03.492645266  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:2985:gst_v4l2_object_probe_caps_for_format:<sink_sub_bin_encoder1:src> Could not probe minimum capture size for pixelformat H264
0:00:03.492648793  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:3100:gst_v4l2_object_get_nearest_size:<sink_sub_bin_encoder1:src> Unable to try format: Unknown error -1
0:00:03.492652750  2106 0x55d13d5955e0 WARN                    v4l2 gstv4l2object.c:2991:gst_v4l2_object_probe_caps_for_format:<sink_sub_bin_encoder1:src> Could not probe maximum capture size for pixelformat H264
Unable to set device in gst_nvstreammux_change_state
Unable to set device in gst_nvstreammux_change_state
0:00:03.492799656  2106 0x55d13d5955e0 WARN                     bin gstbin.c:2808:reset_state:<src_bin_muxer> Failed to switch back down to NULL
Unable to set device in gst_nvstreammux_change_state
0:00:03.897058652  2106 0x55d13d5955e0 WARN                     bin gstbin.c:2808:reset_state:<multi_src_bin> Failed to switch back down to NULL
** ERROR: <main:706>: Failed to set pipeline to PAUSED
Quitting
Unable to set device in gst_nvstreammux_change_state
App run failed

• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

Running:

sudo deepstream-app --gst-debug-level=2 -c source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt

• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Cheers Maximilian

3

There is no hardware encoder in A100 Video Encode and Decode GPU Support Matrix | NVIDIA Developer. Please set the encoder type to software encoder.

4

Ahoi and thank you very much for the fast response.

I adapted the sind as followed:

[sink1]
enable=0
...

[sink2]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=1
sw-preset=1 #for SW enc=(0)None (1)ultrafast (2)superfast (3)veryfast (4)faster
#(5)fast (6)medium (7)slow (8)slower (9)veryslow (10)placebo
sync=0
#iframeinterval=10
bitrate=400000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
# set profile only for hw encoder, sw encoder selects profile based on sw-preset
profile=0
# set below properties in case of RTSPStreaming
rtsp-port=8554
udp-port=5400

as well as:

[sink1]
enable=0
...

[sink2]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=1
#sw-preset=1 #for SW enc=(0)None (1)ultrafast (2)superfast (3)veryfast (4)faster
#(5)fast (6)medium (7)slow (8)slower (9)veryslow (10)placebo
sync=0
#iframeinterval=10
bitrate=400000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
# set profile only for hw encoder, sw encoder selects profile based on sw-preset
profile=0
# set below properties in case of RTSPStreaming
rtsp-port=8554
udp-port=5400

With both settings i still get the following error:

ubuntu@nvidia-iot-edge:/opt/nvidia/deepstream/deepstream-7.0/samples/configs/deepstream-app$ sudo deepstream-app --gst-debug-level=2 -c source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt
Error: Could not get cuda device count (cudaErrorInitializationError)
Failed to parse group property
** ERROR: <gst_nvinfer_parse_config_file:1392>: failed
Error: Could not get cuda device count (cudaErrorInitializationError)
Failed to parse group property
** ERROR: <gst_nvinfer_parse_config_file:1392>: failed
Error: Could not get cuda device count (cudaErrorInitializationError)
Failed to parse group property
** ERROR: <gst_nvinfer_parse_config_file:1392>: failed
Unable to set device in gst_nvstreammux_change_state
Unable to set device in gst_nvstreammux_change_state
0:00:01.956517162  3557 0x5581224bd730 WARN                     bin gstbin.c:2808:reset_state:<src_bin_muxer> Failed to switch back down to NULL
Unable to set device in gst_nvstreammux_change_state
0:00:01.956630075  3557 0x5581224bd730 WARN                     bin gstbin.c:2808:reset_state:<multi_src_bin> Failed to switch back down to NULL
** ERROR: <main:706>: Failed to set pipeline to PAUSED
Quitting
Unable to set device in gst_nvstreammux_change_state
App run failed

Kind regards and many thanks for your help and support.

BR Maximilian

5

From the log you post, the low level drivers are not correct.
Do you install DeepStream according to the instruction in Installation — DeepStream documentation 6.4 documentation?
Or you are running with DeepStream docker?

6

Ahoi and thank you very much for the response.

I use the A100 GPU via passthrough on ESXi 8.
I followed the Installation of Deepstream according to the official documentation via Installation — DeepStream documentation 6.4 documentation (nvidia.com)

every step of the installation succeeded and I installed the Graph Composer and Graph Composer Runtime Ubuntu 22.04 LTS (x86_64) — DeepStream documentation 6.4 documentation (nvidia.com)

No I do not run Deepstream Docker.

Important to note is that this installation manual Installation — DeepStream documentation 6.4 documentation (nvidia.com) install Deepstream 7.0 see here: Installation — DeepStream documentation 6.4 documentation (nvidia.com)

image
image1116×526 51.5 KB

Really appreciate your help

BR Maximilian

7

I do not reproduce the same issue. can you consult the virtual machine vendor to identify the difference?

8

Ahoi @Fiona.Chen

I use a Ubuntu 18.04 VM on a ESXi 8 hypervisor with a nVidia A100 assigned:

image
image1920×642 55.3 KB

The nVidia A100 is assigned via Passthrough and all drivers are installed correct.

saddly it still does not work to execute the test aaplications with this setup…

BR Maximilian

9

How did you run the docker? Has the correct GPU driver and CUDA version been installed on the host machine before you start to use the docker? Quickstart Guide — DeepStream 5.1 Release documentation (nvidia.com)

10

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.