Unable to run samples/configs/deepstream-app/source2_1080p_dec_infer-resnet_demux_int_8.txt" on EC2 instance

[vakkati]

Please provide complete information as applicable to your setup.
• AWS p2.xl EC2 instance
• AMI: Deep Learning Base GPU AMI (Ubuntu 20.04) 20230911
• DeepStream 6.3 dGPU Debian
• CUDA 12.1
• GStreamer 1.16.3
• TensorRT 8.5.3.1
• NVIDIA driver 525.125.06
• questions
• Issue arises when running the command “deepstream-app -c /opt/nvidia/deepstream/deepstream-6.3/samples/configs/deepstream-app/source2_108deepstream-app” with the deployment given below
**• Issue reproducable by connecting to EC2 through SSH and running the following commands

Install Dependencies

sudo apt install \

libssl1.1 \

libgstreamer1.0-0 \

gstreamer1.0-tools \

gstreamer1.0-plugins-good \

gstreamer1.0-plugins-bad \

gstreamer1.0-plugins-ugly \

gstreamer1.0-libav \

libgstreamer-plugins-base1.0-dev \

libgstrtspserver-1.0-0 \

libjansson4 \

libyaml-cpp-dev \

libjsoncpp-dev \

protobuf-compiler \

gcc \

make \

git \

python3

Install CUDA Toolkit 12.1

sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub

sudo add-apt-repository “deb Index of /compute/cuda/repos/ubuntu2004/x86_64 /”

sudo apt-get update

sudo apt-get install cuda-toolkit-12-1

Install NVIDIA driver 525.125.06 (only for data center GPUs)

Download and install using NVIDIA driver 525.125.06 from NVIDIA Unix drivers page at: Data Center Driver for Linux x64 | 525.125.06 | CBL Mariner, Linux 64-bit | NVIDIA

Already downloaded and saved here: s3://…/NVIDIA-Linux-x86_64-525.125.06.run

aws s3 cp s3://…/NVIDIA-Linux-x86_64-525.125.06.run .

Run the following commands:

chmod 755 NVIDIA-Linux-x86_64-525.125.06.run

sudo ./NVIDIA-Linux-x86_64-525.125.06.run --no-cc-version-check

(driver install failed, current driver is 535, try to proceed)

Install cuda-compat-12-1 (only for data center GPUs)

sudo apt-get install -y cuda-compat-12-1

export LD_LIBRARY_PATH=/usr/local/cuda/compat:$LD_LIBRARY_PATH

Install TensorRT 8.5.3.1

sudo apt-get install libnvinfer8=8.5.3-1+cuda11.8 libnvinfer-plugin8=8.5.3-1+cuda11.8 libnvparsers8=8.5.3-1+cuda11.8 \

libnvonnxparsers8=8.5.3-1+cuda11.8 libnvinfer-bin=8.5.3-1+cuda11.8 libnvinfer-dev=8.5.3-1+cuda11.8 \

libnvinfer-plugin-dev=8.5.3-1+cuda11.8 libnvparsers-dev=8.5.3-1+cuda11.8 libnvonnxparsers-dev=8.5.3-1+cuda11.8 \

libnvinfer-samples=8.5.3-1+cuda11.8 libcudnn8=8.7.0.84-1+cuda11.8 libcudnn8-dev=8.7.0.84-1+cuda11.8 \

python3-libnvinfer=8.5.3-1+cuda11.8 python3-libnvinfer-dev=8.5.3-1+cuda11.8

Install librdkafka (to enable Kafka protocol adaptor for message broker)

git clone GitHub - confluentinc/librdkafka: The Apache Kafka C/C++ library

cd librdkafka

git reset --hard 7101c2310341ab3f4675fc565f64f0967e135a6a

./configure

make

sudo make install

(Make error, try to proceed)

sudo mkdir -p /opt/nvidia/deepstream/deepstream-6.3/lib

sudo cp -R /usr/local/lib/librdkafka* /opt/nvidia/deepstream/deepstream-6.3/lib

Install the DeepStream SDK

Download the DeepStream 6.3 dGPU Debian package deepstream-6.3_6.3.0-1_amd64.deb

aws s3 cp s3://…/deepstream-6.3_6.3.0-1_amd64.deb .

sudo apt-get install ./deepstream-6.3_6.3.0-1_amd64.deb

Run the deepstream-app (the reference application)

deepstream-app -c <path_to_config_file>

You can find sample configuration files under /opt/nvidia/deepstream/deepstream-6.3/samples directory
**

[junshengy]

@vakkati

“deepstream-app -c /opt/nvidia/deepstream/deepstream-6.3/samples/configs/deepstream-app/source2_108deepstream-app”

The correct command line should be like this

deepstream-app -c /opt/nvidia/deepstream/deepstream-6.3/samples/configs/deepstream-app/source2_1080p_dec_infer-resnet_demux_int8.txt 

So what is your question? Can you share your error log?

[vakkati]

Apologies, typo on the command. Running the command gives me the following error:

Not sure if I did something wrong with the deployment but would love to fix this error

[junshengy]

Run this script to clear gstreamer’s cache

/opt/nvidia/deepstream/deepstream/install.sh

Then run the following command line to see if there is any output

gst-inspect-1.0 nvstreammux

[vakkati]

Running those commands resulted in the following output, followed by more “nvbufsurftransform:cuInit failed : 100” not in the picture:

While installing the NVIDIA Accelerated Graphic Driver for Linux I had the following popup. This might be due to the EC2 instance not having a /usr/lib directory:

[vakkati]

[junshengy]

@vakkati

followed by more “nvbufsurftransform:cuInit failed : 100” n

I think your GPU driver or CUDA is not installed correctly.

https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__TYPES.html#group__CUDA__TYPES_1gc6c391505e117393cc2558fff6bfc2e9

CUDA_ERROR_NO_DEVICE = 100
This indicates that no CUDA-capable devices were detected by the installed CUDA driver.

[vakkati]

I agree as well, it seems that the NVIDIA GPU driver is not properly installed. During the installation process I get the following popup message, however my EC2 instance does not have a ‘/usr/lib’, directory, which is where it tries to fetch the gstreamer-1.0 from (‘usr/lib/x86_64-linux-gnu/gstreamer-1.0/…’). The popup suggests installing the pkg-config utility and the X.org SDK/development package but I’m unsure how to configure the location of gstreamer-1.0 and fix this problem. Any guidance in fixing the problem would be greatly appreciated

[junshengy]

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

The driver is related to the kernel. You may need to refer to the AWS gpu driver installation documentation.

Install NVIDIA drivers on Linux instances - Amazon Elastic Compute Cloud