Running nvprof with WSL 2 + CUDA + Ubuntu-18.04?

Hi all,

I’m incredibly excited about the progress wsl2 has made in integrating with cuda and nvidia graphics cards, and for the last few days I’ve been successfully compiling code and running it on my local machine’s gpu from within wsl 2 environment. It’s been great.

However, I have been unable to get any of the profiling tools up and running, so I thought I would post here to see if anyone has any ideas.

System setup
I installed WSL 2 and the nvidia windows driver as recommended. I’m running everything on the Ubuntu-18.04 LTS from the microsoft store. To avoid overwriting the graphics driver that comes with an “apt install” of the cuda package, I got the local installer (“cuda_11.0.1_450.36.06_linux.run”) and installed the cuda 11 toolkit without installing the nvidia driver.

** Toolkit working **
It’s clear that this initial setup is working as hoped: I am able to compile custom c++/CUDA code; it runs as expected; kernel calls and computations are all successful.

** Profiling tools not working **
The one hiccup is that the profiling tools all fail, with “Error: incompatible CUDA driver version”.
nvcc --version gives “cuda compilation tools, release 11.0, V11.0.167”, and
nvprof --version gives “Release version 11.0.167 (21)”, which all seems self-consistent.

Question
Is there an incompatibility somewhere with the windows driver? I haven’t installed any other versions of the toolkit or the drivers, so it’s not clear where the incompatibility is. One curious thing is that running nvidia-smi.exe gives an initial heading of:
NVIDIA-SMI 455.41 Drive Version: 455.41 CUDA Version: 11.1
perhaps the cuda version reported there is a typo? Or is it pointing to the ultimate root of the problem, and I have to wait for toolkit 11.1 to be released before all of these tools will work together?

Thanks for your help!

hi there,

Thanks for using CUDA on WSL 2. We don’t support CUDA profiling tools yet in WSL 2 - this is one of the features we will be adding over time. I will update the limitations section in our docs to address this point.

Also - note that NVML is not fully supported in WSL 2 either. Starting with CUDA 11, individual components of the toolkit are independently versioned (so nvcc might report a different version from say cuBLAS). Also - since R455 is a preview driver, it may report a version of CUDA that is not the released 11.0 version.

I see – thanks very much for the information!