nvbufsurftransform:cuInit failed : 999

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

• NVIDIA GeForce RTX 3090 Ti
• DeepStream 6.2

I am using Deepstream to analyze my rtsp stream. It is working ok, but sometimes I have a error:

nvbufsurftransform:cuInit failed : 999

If I have this error, I need to reboot my computer. Sometimes after this error I need to reinstall nvidia drivers. What can I do to fix it?

3

This is a low level CUDA error. There are many possible problems which can trigger such error. Can you reproduce the error stablely? Can you describe how to reproduce the error?

4

The error is intermittent. For instance, when I initially launch my program with Deepstream, it functions properly. However, after a certain period of time, the error reoccurs. This can happen after a few hours or a few days. To resolve it, I simply reboot the system, and it functions properly again.

5

According to your description, can you monitor the memory usage and other system resource usage when running your DeepStream application?

6

I will monitor system resources. Will return with new information

7

Can you also debug more to identify the root cause? Such as using a simple DeepStream pipeline to reproduce the issue to exclude the impact from other non-DeepStream functions and modules. It is better to reproduce the issue with our original sample code or just with very limited changes to our sample code.

8

There are no unusual resource consumption. This is stable with RAM, CPU and GPU. I have 2 docker containers with deepstream app. At one point both get an error: nvbufsurftransform:cuInit failed : 999. After this moment I can’t use CUDA anymore. I need reboot.

>>> torch.cuda.is_available()
/home/nvidia/.local/lib/python3.8/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at  ../c10/cuda/CUDAFunctions.cpp:115.)
  return torch._C._cuda_getDeviceCount() > 0
False

It looks like problem is not with deepstream. Problems are in my Nvidia drivers