CUPTI_ERROR_NOT_INITIALIZED is cancelling my brain on MS WSL2

Hello,

when trying to profile a pytorch script using the torch.autograd.profiler, I get the following warning:

Running your script with the autograd profiler...
WARNING:2022-06-01 13:37:49 513:513 init.cpp:129] function status failed with error CUPTI_ERROR_NOT_INITIALIZED (15)
WARNING:2022-06-01 13:37:49 513:513 init.cpp:130] CUPTI initialization failed - CUDA profiler activities will be missing
INFO:2022-06-01 13:37:49 513:513 init.cpp:132] If you see CUPTI_ERROR_INSUFFICIENT_PRIVILEGES, refer to https://developer.nvidia.com/nvidia-development-tools-solutions-err-nvgpuctrperm-cupti

Apparently, cupti is not available / initializable. As a result, I cannot see the profiling results for my CUDA activity. But I also read elsewhere that this error may appear if the installed version of cupti is not compatible with the cuda version (e.g. cause its newer).

To give you some info: on my PC, by default, there is CUDA 11.6 installed. However, I installed pytorch into a conda environment on Microsoft WSL2 (Ubuntu 20.04) with CUDA 11.3 (this is the latest stable release from the official pytorch website). It seems that in the conda cudatoolkit, there is no cupti included. Therefore, maybe, it does not work because the python script is trying to use cupti, but the cupti is either not available or not compatible with cuda 11.3? The question would then be how to install the correct cupti or how else to fi this error? I tried conda install -c anaconda cupti but it did not change anything.

The strange thing is that when running the profiling in my conda environment on windows on the same pc, it works. However, as windows has certain drawbacks when working with pytorch, it is a necessity for me to switch to linux. however, how can i do that when i cannot properly profile my code? ^^ you get the point. I appreciate any help :) thanks!

3 Likes