Why Nvidia-Cuda drivers never worked in my Ubuntu?

thairup · July 17, 2023, 4:04am

jag@Aigen:~

lspci | grep -i nvidia

02:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
02:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)

$ cat /proc/driver/nvidia/version

NVRM version: NVIDIA UNIX x86_64 Kernel Module 535.54.03 Tue Jun 6 22:20:39 UTC 2023
GCC version: gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)

jag@Aigen:~$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Jun_13_19:16:58_PDT_2023
Cuda compilation tools, release 12.2, V12.2.91
Build cuda_12.2.r12.2/compiler.32965470_0

$ nvidia-smi

$ uname -m && cat /etc/*release
x86_64
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION=“Ubuntu 20.04.6 LTS”
NAME=“Ubuntu”
VERSION=“20.04.6 LTS (Focal Fossa)”
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME=“Ubuntu 20.04.6 LTS”
VERSION_ID=“20.04”
HOME_URL=“https://www.ubuntu.com/”
SUPPORT_URL=“https://help.ubuntu.com/”
BUG_REPORT_URL=“https://bugs.launchpad.net/ubuntu/”
PRIVACY_POLICY_URL=“Data privacy | Ubuntu”
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

$ gcc --version
gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0

$ uname -r
5.4.0-152-generic

env) jag@Aigen:/media/jag/NEU/3PAX/ubuntu-webui$ dmesg |grep nvidia
[ 2.652693] nvidia: loading out-of-tree module taints kernel.
[ 2.652710] nvidia: module license ‘NVIDIA’ taints kernel.
[ 2.735774] nvidia: module verification failed: signature and/or required key missing - tainting kernel
[ 2.754398] nvidia-nvlink: Nvlink Core is being initialized, major device number 239
[ 2.757983] nvidia 0000:02:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
[ 2.879795] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 530.30.02 Wed Feb 22 03:45:40 UTC 2023
[ 2.881410] [drm] [nvidia-drm] [GPU ID 0x00000200] Loading driver
[ 2.881414] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:02:00.0 on minor 0
[ 5.018738] nvidia-uvm: Loaded the UVM driver, major device number 237.
[ 45.924249] audit: type=1400 audit(1688018510.932:41): apparmor=“DENIED” operation=“open” profile=“snap.nvtop.nvtop” name=“/proc/driver/nvidia/capabilities/mig/config” pid=1985 comm=“nvtop” requested_mask=“r” denied_mask=“r” fsuid=1000 ouid=0
[ 45.925223] audit: type=1400 audit(1688018510.932:42): apparmor=“DENIED” operation=“open” profile=“snap.nvtop.nvtop” name=“/proc/driver/nvidia/capabilities/mig/config” pid=1985 comm=“nvtop” requested_mask=“r” denied_mask=“r” fsuid=1000 ouid=0
[ 45.925239] audit: type=1400 audit(1688018510.932:43): apparmor=“DENIED” operation=“open” profile=“snap.nvtop.nvtop” name=“/proc/driver/nvidia/capabilities/mig/config” pid=1985 comm=“nvtop” requested_mask=“r” denied_mask=“r” fsuid=1000 ouid=0
[ 45.925248] audit: type=1400 audit(1688018510.932:44): apparmor=“DENIED” operation=“open” profile=“snap.nvtop.nvtop” name=“/proc/driver/nvidia/capabilities/mig/monitor” pid=1985 comm=“nvtop” requested_mask=“r” denied_mask=“r” fsuid=1000 ouid=0
[ 45.925331] audit: type=1400 audit(1688018510.932:45): apparmor=“DENIED” operation=“open” profile=“snap.nvtop.nvtop” name=“/proc/driver/nvidia/capabilities/mig/monitor” pid=1985 comm=“nvtop” requested_mask=“r” denied_mask=“r” fsuid=1000 ouid=0
[ 45.925340] audit: type=1400 audit(1688018510.932:46): apparmor=“DENIED” operation=“open” profile=“snap.nvtop.nvtop” name=“/proc/driver/nvidia/capabilities/mig/monitor” pid=1985 comm=“nvtop” requested_mask=“r” denied_mask=“r” fsuid=1000 ouid=0
(env) jag@Aigen:/media/jag/NEU/3PAX/ubuntu-webui$

ERRORS

Normal launch

jag@Aigen:/media/jag/NEU/3PAX/ubuntu-webui$ ./webui.sh
Persistence mode is already Enabled for GPU 00000000:02:00.0.
All done.
/usr/local/cuda-12.1/bin:/home/jag/.yarn/bin:/home/jag/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
/usr/local/cuda-12.1/lib :/usr/local/cuda-12.1/lib64
./webui-user.sh: строка 15: export: «/usr=/usr»: это недопустимый идентификатор
./webui-user.sh: строка 64: export: «=»: это недопустимый идентификатор
./webui-user.sh: строка 64: export: «libtcmalloc.so.4»: это недопустимый идентификатор

################################################################
Install script for stable-diffusion + Web UI
Tested on Debian 11 (Bullseye)
################################################################

################################################################
Running on jag user
################################################################

################################################################
Repo already cloned, using it as install directory
################################################################

################################################################
Create and activate python venv
################################################################

################################################################
Accelerating launch.py…
################################################################
The following values were not passed to accelerate launch and had defaults used instead:
--num_processes was set to a value of 1
--num_machines was set to a value of 1
--mixed_precision was set to a value of 'no'
--dynamo_backend was set to a value of 'no'
To avoid this warning pass in values for each of the problematic parameters or run accelerate config.
Python 3.11.4+ (main, Jun 24 2023, 08:51:39) [GCC 9.4.0]
Version: v1.3.2
Commit hash: baf6946e06249c5af9851c60171692c44ef633e0
Traceback (most recent call last):
File “/media/jag/NEU/3PAX/ubuntu-webui/launch.py”, line 38, in
main()
File “/media/jag/NEU/3PAX/ubuntu-webui/launch.py”, line 29, in main
prepare_environment()
File “/media/jag/NEU/3PAX/ubuntu-webui/modules/launch_utils.py”, line 257, in prepare_environment
raise RuntimeError(
RuntimeError: Torch is not able to use GPU; add --skip-torch-cuda-test to COMMANDLINE_ARGS variable to disable this check
Traceback (most recent call last):
File “/media/jag/NEU/3PAX/ubuntu-webui/env/bin/accelerate”, line 8, in
sys.exit(main())
^^^^^^
File “/media/jag/NEU/3PAX/ubuntu-webui/env/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py”, line 45, in main
args.func(args)
File “/media/jag/NEU/3PAX/ubuntu-webui/env/lib/python3.11/site-packages/accelerate/commands/launch.py”, line 923, in launch_command
simple_launcher(args)
File “/media/jag/NEU/3PAX/ubuntu-webui/env/lib/python3.11/site-packages/accelerate/commands/launch.py”, line 579, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command ‘[’/media/jag/NEU/3PAX/ubuntu-webui/env/bin/python’, ‘launch.py’]’ returned non-zero exit status 1.

2/ Short Launch:

jag@Aigen:/media/jag/NEU/3PAX/ubuntu-webui$ source env/bin/activate
(env) jag@Aigen:/media/jag/NEU/3PAX/ubuntu-webui$ CUDA_VISIBLE_DEVICES=0 python launch.py
Python 3.11.4+ (main, Jun 24 2023, 08:51:39) [GCC 9.4.0]
Version: v1.3.2
Commit hash: baf6946e06249c5af9851c60171692c44ef633e0
Traceback (most recent call last):
File “/media/jag/NEU/3PAX/ubuntu-webui/launch.py”, line 38, in
main()
File “/media/jag/NEU/3PAX/ubuntu-webui/launch.py”, line 29, in main
prepare_environment()
File “/media/jag/NEU/3PAX/ubuntu-webui/modules/launch_utils.py”, line 257, in prepare_environment
raise RuntimeError(
RuntimeError: Torch is not able to use GPU; add --skip-torch-cuda-test to COMMANDLINE_ARGS variable to disable this check
(env) jag@Aigen:/media/jag/NEU/3PAX/ubuntu-webui$

3/ MAIN ERR: RuntimeError: Torch is not able to use GPU; add --skip-torch-cuda-test to COMMANDLINE_ARGS variable to disable this check

3/ tests

jag@Aigen:~/codes/cuda-samples-master/bin/x86_64/linux/release$ ./bandwidthTest
[CUDA Bandwidth Test] - Starting…
Running on…

cudaGetDeviceProperties returned 802
→ system not yet initialized
CUDA error at bandwidthTest.cu:256 code=802(cudaErrorSystemNotReady) “cudaSetDevice(currentDevice)”

test3

(env) jag@Aigen:/media/jag/NEU/3PAX/ubuntu-webui$ python test3.py
Python VERSION: 3.8.10 (default, May 26 2023, 14:05:08)
[GCC 9.4.0]
__pyTorch VERSION: <module ‘torch.version’ from ‘/media/jag/NEU/3PAX/ubuntu-webui/env/lib/python3.8/site-packages/torch/version.py’>
__CUDA VERSION
__CUDNN VERSION: 8500
__Number CUDA Devices: 1
__Devices
index, name, driver_version, memory.total [MiB], memory.used [MiB], memory.free [MiB]
0, NVIDIA GeForce GTX 1070, 530.30.02, 8192 MiB, 143 MiB, 7966 MiB
Traceback (most recent call last):
File “test3.py”, line 11, in
print(‘Active CUDA Device: GPU’, torch.cuda.current_device())
File “/media/jag/NEU/3PAX/ubuntu-webui/env/lib/python3.8/site-packages/torch/cuda/__init.py”, line 674, in current_device
_lazy_init()
File “/media/jag/NEU/3PAX/ubuntu-webui/env/lib/python3.8/site-packages/torch/cuda/init.py”, line 247, in _lazy_init
torch._C._cuda_init()
RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 802: system not yet initialized

jag@Aigen:~/codes/cuda-samples-master/bin/x86_64/linux/release$ ./bandwidthTest
[CUDA Bandwidth Test] - Starting…
Running on…

cudaGetDeviceProperties returned 802
→ system not yet initialized
CUDA error at bandwidthTest.cu:256 code=802(cudaErrorSystemNotReady) “cudaSetDevice(currentDevice)”

thairup · July 17, 2023, 4:06am

If Ndivia cannot solve this problem - just give me a fix (or patch) for your 535 driver to fix this bug:
cudaGetDeviceProperties returned 802
“CUDA error at bandwidthTest.cu:256 code=802(cudaErrorSystemNotReady) “cudaSetDevice(currentDevice)””