Why Nvidia-Cuda drivers never worked in my Ubuntu?

thairup · July 17, 2023, 4:04am

jag@Aigen:~

lspci | grep -i nvidia

02:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
02:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)

$ cat /proc/driver/nvidia/version

NVRM version: NVIDIA UNIX x86_64 Kernel Module 535.54.03 Tue Jun 6 22:20:39 UTC 2023
GCC version: gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)

jag@Aigen:~$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Jun_13_19:16:58_PDT_2023
Cuda compilation tools, release 12.2, V12.2.91
Build cuda_12.2.r12.2/compiler.32965470_0

$ nvidia-smi

$ uname -m && cat /etc/*release
x86_64
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION=“Ubuntu 20.04.6 LTS”
NAME=“Ubuntu”
VERSION=“20.04.6 LTS (Focal Fossa)”
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME=“Ubuntu 20.04.6 LTS”
VERSION_ID=“20.04”
HOME_URL=“https://www.ubuntu.com/”
SUPPORT_URL=“https://help.ubuntu.com/”
BUG_REPORT_URL=“https://bugs.launchpad.net/ubuntu/”
PRIVACY_POLICY_URL=“Data privacy | Ubuntu”
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

$ gcc --version
gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0

$ uname -r
5.4.0-152-generic

env) jag@Aigen:/media/jag/NEU/3PAX/ubuntu-webui$ dmesg |grep nvidia
[ 2.652693] nvidia: loading out-of-tree module taints kernel.
[ 2.652710] nvidia: module license ‘NVIDIA’ taints kernel.
[ 2.735774] nvidia: module verification failed: signature and/or required key missing - tainting kernel
[ 2.754398] nvidia-nvlink: Nvlink Core is being initialized, major device number 239
[ 2.757983] nvidia 0000:02:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
[ 2.879795] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 530.30.02 Wed Feb 22 03:45:40 UTC 2023
[ 2.881410] [drm] [nvidia-drm] [GPU ID 0x00000200] Loading driver
[ 2.881414] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:02:00.0 on minor 0
[ 5.018738] nvidia-uvm: Loaded the UVM driver, major device number 237.
[ 45.924249] audit: type=1400 audit(1688018510.932:41): apparmor=“DENIED” operation=“open” profile=“snap.nvtop.nvtop” name=“/proc/driver/nvidia/capabilities/mig/config” pid=1985 comm=“nvtop” requested_mask=“r” denied_mask=“r” fsuid=1000 ouid=0
[ 45.925223] audit: type=1400 audit(1688018510.932:42): apparmor=“DENIED” operation=“open” profile=“snap.nvtop.nvtop” name=“/proc/driver/nvidia/capabilities/mig/config” pid=1985 comm=“nvtop” requested_mask=“r” denied_mask=“r” fsuid=1000 ouid=0
[ 45.925239] audit: type=1400 audit(1688018510.932:43): apparmor=“DENIED” operation=“open” profile=“snap.nvtop.nvtop” name=“/proc/driver/nvidia/capabilities/mig/config” pid=1985 comm=“nvtop” requested_mask=“r” denied_mask=“r” fsuid=1000 ouid=0
[ 45.925248] audit: type=1400 audit(1688018510.932:44): apparmor=“DENIED” operation=“open” profile=“snap.nvtop.nvtop” name=“/proc/driver/nvidia/capabilities/mig/monitor” pid=1985 comm=“nvtop” requested_mask=“r” denied_mask=“r” fsuid=1000 ouid=0
[ 45.925331] audit: type=1400 audit(1688018510.932:45): apparmor=“DENIED” operation=“open” profile=“snap.nvtop.nvtop” name=“/proc/driver/nvidia/capabilities/mig/monitor” pid=1985 comm=“nvtop” requested_mask=“r” denied_mask=“r” fsuid=1000 ouid=0
[ 45.925340] audit: type=1400 audit(1688018510.932:46): apparmor=“DENIED” operation=“open” profile=“snap.nvtop.nvtop” name=“/proc/driver/nvidia/capabilities/mig/monitor” pid=1985 comm=“nvtop” requested_mask=“r” denied_mask=“r” fsuid=1000 ouid=0
(env) jag@Aigen:/media/jag/NEU/3PAX/ubuntu-webui$

ERRORS

Normal launch

jag@Aigen:/media/jag/NEU/3PAX/ubuntu-webui$ ./webui.sh
Persistence mode is already Enabled for GPU 00000000:02:00.0.
All done.
/usr/local/cuda-12.1/bin:/home/jag/.yarn/bin:/home/jag/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
/usr/local/cuda-12.1/lib :/usr/local/cuda-12.1/lib64
./webui-user.sh: строка 15: export: «/usr=/usr»: это недопустимый идентификатор
./webui-user.sh: строка 64: export: «=»: это недопустимый идентификатор
./webui-user.sh: строка 64: export: «libtcmalloc.so.4»: это недопустимый идентификатор

################################################################
Install script for stable-diffusion + Web UI
Tested on Debian 11 (Bullseye)
################################################################

################################################################
Running on jag user
################################################################

################################################################
Repo already cloned, using it as install directory
################################################################

################################################################
Create and activate python venv
################################################################

################################################################
Accelerating launch.py…
################################################################
The following values were not passed to accelerate launch and had defaults used instead:
--num_processes was set to a value of 1
--num_machines was set to a value of 1
--mixed_precision was set to a value of 'no'
--dynamo_backend was set to a value of 'no'
To avoid this warning pass in values for each of the problematic parameters or run accelerate config.
Python 3.11.4+ (main, Jun 24 2023, 08:51:39) [GCC 9.4.0]
Version: v1.3.2
Commit hash: baf6946e06249c5af9851c60171692c44ef633e0
Traceback (most recent call last):
File “/media/jag/NEU/3PAX/ubuntu-webui/launch.py”, line 38, in
main()
File “/media/jag/NEU/3PAX/ubuntu-webui/launch.py”, line 29, in main
prepare_environment()
File “/media/jag/NEU/3PAX/ubuntu-webui/modules/launch_utils.py”, line 257, in prepare_environment
raise RuntimeError(
RuntimeError: Torch is not able to use GPU; add --skip-torch-cuda-test to COMMANDLINE_ARGS variable to disable this check
Traceback (most recent call last):
File “/media/jag/NEU/3PAX/ubuntu-webui/env/bin/accelerate”, line 8, in
sys.exit(main())
^^^^^^
File “/media/jag/NEU/3PAX/ubuntu-webui/env/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py”, line 45, in main
args.func(args)
File “/media/jag/NEU/3PAX/ubuntu-webui/env/lib/python3.11/site-packages/accelerate/commands/launch.py”, line 923, in launch_command
simple_launcher(args)
File “/media/jag/NEU/3PAX/ubuntu-webui/env/lib/python3.11/site-packages/accelerate/commands/launch.py”, line 579, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command ‘[’/media/jag/NEU/3PAX/ubuntu-webui/env/bin/python’, ‘launch.py’]’ returned non-zero exit status 1.

2/ Short Launch:

jag@Aigen:/media/jag/NEU/3PAX/ubuntu-webui$ source env/bin/activate
(env) jag@Aigen:/media/jag/NEU/3PAX/ubuntu-webui$ CUDA_VISIBLE_DEVICES=0 python launch.py
Python 3.11.4+ (main, Jun 24 2023, 08:51:39) [GCC 9.4.0]
Version: v1.3.2
Commit hash: baf6946e06249c5af9851c60171692c44ef633e0
Traceback (most recent call last):
File “/media/jag/NEU/3PAX/ubuntu-webui/launch.py”, line 38, in
main()
File “/media/jag/NEU/3PAX/ubuntu-webui/launch.py”, line 29, in main
prepare_environment()
File “/media/jag/NEU/3PAX/ubuntu-webui/modules/launch_utils.py”, line 257, in prepare_environment
raise RuntimeError(
RuntimeError: Torch is not able to use GPU; add --skip-torch-cuda-test to COMMANDLINE_ARGS variable to disable this check
(env) jag@Aigen:/media/jag/NEU/3PAX/ubuntu-webui$

3/ MAIN ERR: RuntimeError: Torch is not able to use GPU; add --skip-torch-cuda-test to COMMANDLINE_ARGS variable to disable this check

3/ tests

jag@Aigen:~/codes/cuda-samples-master/bin/x86_64/linux/release$ ./bandwidthTest
[CUDA Bandwidth Test] - Starting…
Running on…

cudaGetDeviceProperties returned 802
→ system not yet initialized
CUDA error at bandwidthTest.cu:256 code=802(cudaErrorSystemNotReady) “cudaSetDevice(currentDevice)”

test3

(env) jag@Aigen:/media/jag/NEU/3PAX/ubuntu-webui$ python test3.py
Python VERSION: 3.8.10 (default, May 26 2023, 14:05:08)
[GCC 9.4.0]
__pyTorch VERSION: <module ‘torch.version’ from ‘/media/jag/NEU/3PAX/ubuntu-webui/env/lib/python3.8/site-packages/torch/version.py’>
__CUDA VERSION
__CUDNN VERSION: 8500
__Number CUDA Devices: 1
__Devices
index, name, driver_version, memory.total [MiB], memory.used [MiB], memory.free [MiB]
0, NVIDIA GeForce GTX 1070, 530.30.02, 8192 MiB, 143 MiB, 7966 MiB
Traceback (most recent call last):
File “test3.py”, line 11, in
print(‘Active CUDA Device: GPU’, torch.cuda.current_device())
File “/media/jag/NEU/3PAX/ubuntu-webui/env/lib/python3.8/site-packages/torch/cuda/__init.py”, line 674, in current_device
_lazy_init()
File “/media/jag/NEU/3PAX/ubuntu-webui/env/lib/python3.8/site-packages/torch/cuda/init.py”, line 247, in _lazy_init
torch._C._cuda_init()
RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 802: system not yet initialized

jag@Aigen:~/codes/cuda-samples-master/bin/x86_64/linux/release$ ./bandwidthTest
[CUDA Bandwidth Test] - Starting…
Running on…

cudaGetDeviceProperties returned 802
→ system not yet initialized
CUDA error at bandwidthTest.cu:256 code=802(cudaErrorSystemNotReady) “cudaSetDevice(currentDevice)”

thairup · July 17, 2023, 4:06am

If Ndivia cannot solve this problem - just give me a fix (or patch) for your 535 driver to fix this bug:
cudaGetDeviceProperties returned 802
“CUDA error at bandwidthTest.cu:256 code=802(cudaErrorSystemNotReady) “cudaSetDevice(currentDevice)””

Topic		Replies	Views
Cannot install cuda CUDA Setup and Installation	9	4763	June 27, 2024
bandwidthTest example throws cudaErrorCallRequiresNewerDriver error when launched via nv-nsight-cu-cli Nsight Compute linux , driver	17	1323	February 9, 2024
Error installing CUDA 11.8 on Ubuntu 22.04 CUDA Setup and Installation cuda , kernel , ubuntu	2	1415	June 14, 2024
cudaGetDeviceCount returned 100 -> no CUDA-capable device is connected CUDA Setup and Installation	5	3068	October 31, 2022
CUDA 10 installation problems on Ubuntu 18.04 CUDA Setup and Installation	24	94587	December 11, 2020
Can't install CUDA 11.7 on Ubuntu 22.04 LTS CUDA Setup and Installation cuda , ubuntu	2	5256	August 27, 2023
run on K40 Linux	83	4611	June 29, 2018
CUDA Samples not working; possible installation mistakes - help please CUDA Setup and Installation	1	2146	July 8, 2022
Ubuntu 22.04.3 LTS Server, Tesla P100, Driver Version: 470.199.02, CUDA Version: 11.4 CUDA Setup and Installation	3	3377	August 19, 2023
Ubuntu 10.04 and CUDA CUDA Programming and Performance	24	155578	April 6, 2011

Why Nvidia-Cuda drivers never worked in my Ubuntu?

Related topics