Deepstream 6.2 - NVIDIA driver is incompatible with required CUDA

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 6.2
• JetPack Version (valid for Jetson only)
• TensorRT Version 8.5.1.7
• NVIDIA GPU Driver Version (valid for GPU only) 525.60.13
• Issue Type( questions, new requirements, bugs)

Hello! I have the next problem. I am following this tutorial for installing Deepstream 6.2 - Quickstart Guide — DeepStream 6.2 Release documentation . I am following the exact instructions listed here, which means I have installed CUDA 11.8 and afterwards I have installed NVIDIA Driver 525.60.13.

Since I want to create a custom plugin for Deepstream which will use my neural network, I decided to convert it from onnx to TensorRT and I have this error

builder = trt.Builder(logger)
TypeError: pybind11::init(): factory function returned nullptr

A quick search has shown me that it is important to have versions consistency for TensorRT to run and that I need to look at CUDA versions. And that’s where I found inconsistency. Driver 525.60.13 that I was instructed to install outputs this when I run

nvidia-smi

NVIDIA-SMI 525.60.13 Driver Version: 525.60.13 CUDA Version: 12.0

Also this command

cat /usr/local/cuda/version.json

shows me this information:

"cuda" : {
      "name" : "CUDA SDK",
      "version" : "11.8.0"
   }
.....
"nvidia_driver" : {
      "name" : "NVIDIA Linux Driver",
      "version" : "520.61.05"
   }

So basically what I have - I have NVIDIA driver 525.60.13 that is incompatible with CUDA 11.8. Because CUDA expects an older driver and NVIDIA driver expects a younger CUDA. I also verified CUDA compatibility here - CUDA Compatibility :: NVIDIA Data Center GPU Driver Documentation. And it does seem that CUDA 11.8 can not work with 525.60.13.

I am a bit confused. Please help me with these questions:
1)nvidia-smi outputs installed CUDA or expected CUDA? If installed CUDA, does it mean I have two cudas installed on my system?
2) Is it possible to force 525.60.13 to cooperate with CUDA 11.8?
3) Is it possible that there is some inconsistency in DeepStream 6.2 installation guide? If so, how to solve it?

Thank you in advance!

3

DeepStream does support ONNX model, you don’t need to convert onnx to TensorRT engine file by yourself.

4

For the usage of TensorRT python interfaces, please raise topic in Latest Deep Learning (Training & Inference)/TensorRT topics - NVIDIA Developer Forums

5

Hi @Fiona.Chen ! Thank you for your reply and letting me know I can use ONNX models efficiently without any prior preparations.

Could you please help me with other questions as well? This TensorRT story was not bugging me that much anyway, but other questions regarding driver and CUDA compatibility… I am worried if I have the right setup and there is no mistake in the DeepStream 6.2 Guide

6

Update:

While I am waiting for your assistance, I decided to verify if my installation of CUDA Toolkit 11.8 was correct. Important to note, I had 0 errors when I installed it, and I see libcudart.so in lib64 folder.

I ran nvcc -V after the installation and it shows this:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

I have found cuda-samples-11.8, compiled them. And when I try to run any sample, I see this:

./deviceQuery
./deviceQuery Starting…

CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 100
→ no CUDA-capable device is detected
Result = FAIL

So I am starting to suspect that it is important for me to use NVIDIA driver and CUDA Toolkit that are compatible with each other. Please let me know if there is a gap in my understanding. Thank you!

7

Which GPU are you working on?

How did you install the NVIDIA driver and CUDA?

Have you following the steps here?Quickstart Guide — DeepStream 6.2 Release documentation

8

GPU - GeForce RTX 4090
NVIDIA driver and CUDA was installed as was described in Quickstart Guide — DeepStream 6.2 Release documentation precisely. The only thing I didn’t do yet, is these two commands for libdrkafka

$ sudo mkdir -p /opt/nvidia/deepstream/deepstream-6.2/lib
$ sudo cp /usr/local/lib/librdkafka* /opt/nvidia/deepstream/deepstream-6.2/lib

But I do not think this is relevant to the problem

9

What is the result with the ls -l /usr/local/cuda* command?

10

Hello @Fiona.Chen!

UPDATE: I fixed an issue.

Yesterday I noticed that my nvidia-smi outputs ERR! in the field of GPU Unit name. Basically, the driver can not analyze/detect my GPU card properly. So although it looked like everything works, there were some missing bits because of the driver.

So what I did - I downloaded NVIDIA driver 535.146.02, because this driver was recommended for my video card which is GeForce RTX 4090.

After I reinstalled the driver, nvidia-smi was able to give full information with 0 errors. After that I reinstalled CUDA 11.8 (just in case), and it started working.

One interesting observation: Deepstream 6.2 also works! I just successfully built and ran deepstream-3d-action-recognition using my current config and everything is on point, although in Deepstream 6.2 guide the older driver was recommended.

My current quesiton is - given that I am using a newer NVIDIA driver that is compatible with my hardware, should I expect problems with Deepstream 6.2 considering that sample application is running? Thx!

Just to recap for everyone who stumble upon this thread, my config:

• Hardware Platform (Jetson / GPU) GeForce RTX 4090
• DeepStream Version 6.2
• TensorRT Version 8.5.1.7
• NVIDIA GPU Driver Version (valid for GPU only) 535.146.02
• CUDA Toolkit version 11.8

11

The DeepStream 6.2 was not verified on driver 535.xxx, we do not guarantee there is no any problem even theoretically drivers support backward compatibility.

12

Got it. Well, I ran another sample application from a so-called TAO package - GitHub - NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream

And it also runs with no issue. I will update this thread as I move forward and let others know whether or not DeepStream 6.2 is usable with 535 driver. Thank you for your help, I appreciate it

13

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.