Quad (4x) A6000 WSL2 CUDA Init Errors

WyattBartel · December 28, 2022, 12:07am

Hey there,

I’m running into some issues with WSL2 on a 4x A6000 Machine. I’ve tried both CUDA 11.7 and 12.0 samples and get “Error: only 0 Devices available, 1 requested. Exiting.” after successfully building and attempting to run the nbody Cuda sample.

$ echo $PATH returns the following successful cuda path:
/usr/local/cuda-12.0/bin

$ echo $CUDA_HOME returns the following:
/usr/local/cuda-12.0/

$ ldconfig -p returns the following list of cuda libraries having been loaded:

Nvidia-SMI returns the following:

…what exactly am I missing from this seemingly appropriate WSL2 setup? I did not install Nvidia display drivers, I am on a brand new WSL2 Ubuntu image running on the latest kernel from Microsoft. I am on Windows 11 22H2.

Drivers as stated in Nvidia-SMI output.

Trying to load Pytorch torch.cuda.is_available() returns the following errors:

“UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error?”
Error 2: out of memory (Triggered internally at /opt/pytorch/pytorch/c10/cuda/“CUDAFunctions.cpp:109”)

Any help here would be wildly appreciated. Thanks!

WyattBartel · January 2, 2023, 3:09am

Update:

If I do a quick:
$export CUDA_VISIBLE_DEVICES=3 (or 1, or 2)

everything works fine. Moving up to the final GPU with $export CUDA_VISIBLE_DEVICES=4 causes a failure.

KnowsNothing · May 3, 2023, 8:35am

The same issue happens with WSL2 on my 4x A6000 machine, but to me any option of CUDA_VISIBLE_DEVICES including GPU 1 causes failures while 0, 2, 3 work fine. Still looking for solutions. :(

boy.2it4 · May 18, 2023, 8:54am

Thanks, finally make things work out by limited to 3 visible devices. I’m using 4x 2080Tis and also facing the same problem.

WyattBartel · May 18, 2023, 9:02am

Given that this is now mildly repro’ed by 3 people, does any Nvidia team member have thoughts on whether this is a Microsoft / WSL or an Nvidia challenge? Or correctable user error perhaps?

rcroman · July 8, 2023, 4:29am

Any news on this? I’m seeing a similar issue with 4x A6000’s. Works fine with three. Nvidia-smi shows all four available, but no joy unless limited by CUDA_VISIBLE_DEVICES or physically unplugging the fourth GPU. This is with driver version 536.25 and CUDA 12.2.

rcroman · July 8, 2023, 4:34am

Possible clue: running dmesg -w while attempting to start up a container with four GPUS produces a whole pile of these errors:

misc dxg: dxgk: dxgkio_reserve_gpu_va: Ioctl failed: -75

cochinnmoon · August 12, 2023, 3:51pm

I resolved this by setting SLI Configuration to “Activate All Displays” in the Nvidia control panel. Given that nvlink/sli was not working before this change, this doesn’t seem to have any downside as far as I can tell.

warren.schultz · August 30, 2023, 3:47pm

Same issue as the others.
Running latest on WSL2, Ubuntu 22.04, CUDA 12.2. 4x RTX 6000 Ada.
Using export CUDA_VISIBLE_DEVICES=0,1,2,3
Any combination of 1 and 3 fail.
0,1,2,3 fail
0,1,2 pass
0,1,3 fail
0,2,3 pass
1,2,3 fail
0,1 pass
0,2 pass
0,3 pass
1,2 pass
1,3 fail
2,3 pass

See related posts here:

github.com/microsoft/WSL

CUDA Initialization Issue and Proposed Workaround with PyTorch in wsl2

opened 02:45PM - 07 Jul 23 UTC

wzsxb233

GPU

### Windows Version Windows11 ### WSL Version 1.2.5.0 ### Are you using WSL …1 or WSL 2? - [X] WSL 2 - [ ] WSL 1 ### Kernel Version 5.15.90.1 ### Distro Version Ubuntu -default ### Other Software I am writing to report an unexpected behavior I’ve encountered when working with PyTorch and CUDA on a wsl2 on Windows 11 system equipped with multiple NVIDIA RTX 3090 GPUs. Environment Details: Operating System: Windows 11 CUDA Version: 12.2 WSL Version: 2 GPUs: 4x NVIDIA RTX 3090 PyTorch Version: 2.01 (CUDA 11.8) Problem Statement: When I set the CUDA_VISIBLE_DEVICES environment variable to enable all the GPUs (0,1,2,3) on the system and then run a PyTorch script that calls torch.cuda.is_available(), I encounter an “Out of Memory” error. Notably, this error does not occur if I only enable GPU 1 or a combination of GPU 0,2,3. Furthermore, this error can be circumvented if I call torch.cuda.device_count() before torch.cuda.is_available(). ### Repro Steps Steps to Reproduce: Set the environment variable: export CUDA_VISIBLE_DEVICES=0,1,2,3 Run a Python script that imports PyTorch and calls torch.cuda.is_available() ### Expected Behavior Expected Behavior: The torch.cuda.is_available() function should return True if GPUs are available and accessible. ### Actual Behavior Observed Behavior: An “Out of Memory” error is triggered internally at ../c10/cuda/CUDAFunctions.cpp:109. The torch.cuda.is_available() function returns False. Workaround: I found that calling torch.cuda.device_count() before torch.cuda.is_available() circumvents the error. However, this workaround requires modifying each script to include this extra call. While the workaround is effective, it may be beneficial to investigate and address the root cause of this issue. I wanted to bring this to your attention and look forward to any insights or potential solutions you might provide. ### Diagnostic Logs _No response_

github.com/microsoft/WSL

Running CUDA samples with multiple GPUs is failed

opened 06:33AM - 21 Dec 21 UTC

yes89929

### Version Microsoft Windows [Version 10.0.22000.376] ### WSL Version …- [X] WSL 2 - [ ] WSL 1 ### Kernel Version 5.10.60.1 ### Distro Version Ubuntu 20.04 and Ubuntu 18.04 ### Other Software CPU: Intel(R) Core(TM) i9-9900X GPU: Nvidia Titan RTX * 4 (driver 510.06) RAM: 128GB ### Repro Steps Install CUDA on WSL ``` sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub sudo sh -c 'echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 /" > /etc/apt/sources.list.d/cuda.list' sudo apt-get update sudo apt-get install -y cuda-toolkit-11-0 ``` Run samples ``` cd /usr/local/cuda-11.0/samples/4_Finance/BlackScholes sudo make ./BlackScholes ``` ``` cd /usr/local/cuda-11.0/samples/1_Utilities/deviceQuery sudo make ./deviceQuery ``` ### Expected Behavior Return success ### Actual Behavior ``` [./BlackScholes] - Starting... CUDA error at ../../common/inc/helper_cuda.h:777 code=2(cudaErrorMemoryAllocation) "cudaGetDeviceCount(&device_count)" ``` ``` ./deviceQuery Starting... CUDA Device Query (Runtime API) version (CUDART static linking) cudaGetDeviceCount returned 2 -> out of memory Result = FAIL ``` ### Diagnostic Logs _No response_

I’ve also found an odd partial workaround, which indicates that it is an initialization issue of some sort at the driver level. It doesn’t necessarily work reliably, but it allows you to get the system GPU configurations.

Toggling GPU performance counters to unrestricted, applying the setting, and then switching back allows the GPU check functions to complete.