I have been working through setup of Dell XPS15 (RTX 3050) W11 WSL2-Ubuntu22.04LTS with CUDA 12.2 to run GPU workloads in containers.
Steps to reproduce the behavior:
- Update Windows, Dell, NVIDIA Graphics driver
- Allow access to all users to the GPU performance counters in NVIDIA Control Panel Developer Mode
- Update WSL, Configure default version to WSL2
- Follow many steps (will add details in follow-up comment)
tl;dr:
install ubuntu 22.04 in WSL2
install cuda 12.2 in WSL2
append relevant directories to PATH and LD_LIBRARY_PATH
install nvidia-cuda-toolkit
install docker
install nvidia-container-toolkit
generate the “Container Device Interface” config json
run vespa container and install vespa-onnxruntime-cuda
try to deploy an app
Expected behavior
The app should read LD_LIBRARY_PATH, load libcublasLt.so.12, converge and deploy.
Actual behaviour
Vespa app in container could not load library libcublasLt.so.12.
nvidia-container-toolkit.log shows a searching and not finding things.
I followed steps to delete something in system32 and make the symlinks. search for windows subsystem for linux - WSL libcuda is not a symbolic link - Super User to see details.
and also added to my ~/.profile
export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH
Environment :
OS Name: Microsoft Windows 11 Pro
Version: 10.0.22621 Build 22621
System Model: XPS 15 9520
Processor: 12th Gen Intel(R) Core™ i7-12700H, 2300 Mhz, 14 Core(s), 20 Logical Processor(s)
BIOS Version/Date: Dell Inc. 1.13.1, 4/17/2023
SMBIOS Version: 3.4
BIOS Mode: UEFI
Total Physical Memory: 31.7 GB
NVIDIA Studio Driver: 536.67
NVIDIA-SMI: 535.86.01
WSL Linux kernel: 5.15.90.1-microsoft-standard-WSL2
WSL Distro Description: Ubuntu 22.04.2 LTS
“CUDA Version”: 12.2
nvcc: Cuda compilation tools, release 12.2, V12.2.91 Build cuda_12.2.r12.2/compiler.32965470_0
Vespa version
8.198.18
I am inspecting the (see attached) /var/log/nvidia-container-toolkit.log and I would like assistance in interpreting all the missing libraries in the log. What is normal and abnormal about what it contains? How can I get it to find these libraries?
nvidia-container-toolkit.log (20 KB)