Error running CUDA container WSL2 (undefined symbol: devicesetgpcclkvfoffset)

I am following the CUDA on WSL2 steps here : CUDA on WSL :: CUDA Toolkit Documentation

machine specifications

  1. uname -a: 5.10.16.3-microsoft-standard-WSL2
  2. OS build: 21382.1
  3. Driver Version: 470.14
  4. CUDA Version: 11.3

build 21382:

image

kernel : 5.10.16

uname -a
Linux tscience-dev3 5.10.16.3-microsoft-standard-WSL2 #1 SMP Fri Apr 2 22:23:49 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

ls -lh /dev/dxg
crw-rw-rw- 1 root root 10, 63 May 17 12:19 /dev/dxg

I am able to run the BlackScholes fine. It detects the GPU.

Running

[./BlackScholes] - Starting...
GPU Device 0: "Turing" with compute capability 7.5

Initializing data...
...allocating CPU memory for options.
...allocating GPU memory for options.
...generating input data in CPU mem.
...copying input data to GPU mem.
Data init done.

Executing Black-Scholes GPU kernel (512 iterations)...
Options count             : 8000000
BlackScholesGPU() time    : 0.152631 msec
Effective memory bandwidth: 524.140381 GB/s
Gigaoptions per second    : 52.414038

BlackScholes, Throughput = 52.4140 GOptions/s, Time = 0.00015 s, Size = 8000000 options, NumDevsUsed = 1, Workgroup = 128

Reading back GPU results...
Checking the results...
...running CPU calculations.

Comparing the results...
L1 norm: 1.741792E-07
Max absolute error: 1.192093E-05

Shutting down...
...releasing GPU memory.
...releasing CPU memory.
Shutdown done.

[BlackScholes] - Test Summary

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

Test passed

But when i try running CUDA container in WSL2 i get this error:

sudo docker run --gpus all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmark
docker: Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: driver error: failed to process request: unknown.
ERRO[0002] error waiting for container: context canceled

here is the output of nvidia-container-cli:

sudo nvidia-container-cli -k -d /dev/tty info

-- WARNING, the following logs are for debugging purposes only --

I0517 19:48:08.863329 11781 nvc.c:372] initializing library context (version=1.4.0, build=704a698b7a0ceec07a48e56c37365c741718c2df)
I0517 19:48:08.863392 11781 nvc.c:346] using root /
I0517 19:48:08.863399 11781 nvc.c:347] using ldcache /etc/ld.so.cache
I0517 19:48:08.863448 11781 nvc.c:348] using unprivileged user 65534:65534
I0517 19:48:08.863470 11781 nvc.c:389] attempting to load dxcore to see if we are running under Windows Subsystem for Linux (WSL)
I0517 19:48:08.968891 11781 dxcore.c:226] Creating a new WDDM Adapter for hAdapter:40000000 luid:1e6fe55
I0517 19:48:09.070621 11781 dxcore.c:267] Adding new adapter via dxcore hAdapter:40000000 luid:1e6fe55 wddm version:3000
I0517 19:48:09.070814 11781 dxcore.c:226] Creating a new WDDM Adapter for hAdapter:40000040 luid:1e70354
I0517 19:48:09.110533 11781 dxcore.c:267] Adding new adapter via dxcore hAdapter:40000040 luid:1e70354 wddm version:3000
I0517 19:48:09.110710 11781 dxcore.c:325] dxcore layer initialized successfully
W0517 19:48:09.113148 11781 nvc.c:397] skipping kernel modules load on WSL
I0517 19:48:09.113981 11782 driver.c:101] starting driver service
E0517 19:48:09.193342 11782 driver.c:168] could not start driver service: load library failed: /usr/lib/wsl/drivers/nv_dispi.inf_amd64_43efafcd74b2efc9/libnvidia-ml.so.1: undefined symbol: devicesetgpcclkvfoffset
I0517 19:48:09.194047 11781 driver.c:203] driver service terminated successfully
nvidia-container-cli: initialization error: driver error: failed to process request

Getting same issue. Waiting for resolution.

Hi there - please see my post : Guide to run CUDA + WSL + Docker with latest versions (21382 Windows build + 470.14 Nvidia) . The guide goes into the steps to get it working correctly.

Most of the guides are now obsolete since Docker is supporting GPU in Windows via their own WSL2 integration, and doesn’t require the steps in the NVIDIA guido.
All the best!