• NVIDIA GeForce RTX 3090 Ti
• DeepStream 6.2
I am using Deepstream to analyze my rtsp stream. It is working ok, but sometimes I have a error:
nvbufsurftransform:cuInit failed : 999
If I have this error, I need to reboot my computer. Sometimes after this error I need to reinstall nvidia drivers. What can I do to fix it?
This is a low level CUDA error. There are many possible problems which can trigger such error. Can you reproduce the error stablely? Can you describe how to reproduce the error?
The error is intermittent. For instance, when I initially launch my program with Deepstream, it functions properly. However, after a certain period of time, the error reoccurs. This can happen after a few hours or a few days. To resolve it, I simply reboot the system, and it functions properly again.
According to your description, can you monitor the memory usage and other system resource usage when running your DeepStream application?
I will monitor system resources. Will return with new information
Can you also debug more to identify the root cause? Such as using a simple DeepStream pipeline to reproduce the issue to exclude the impact from other non-DeepStream functions and modules. It is better to reproduce the issue with our original sample code or just with very limited changes to our sample code.
There are no unusual resource consumption. This is stable with RAM, CPU and GPU. I have 2 docker containers with deepstream app. At one point both get an error:
nvbufsurftransform:cuInit failed : 999. After this moment I can’t use CUDA anymore. I need reboot.
/home/nvidia/.local/lib/python3.8/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:115.)
return torch._C._cuda_getDeviceCount() > 0
It looks like problem is not with deepstream. Problems are in my Nvidia drivers