I’m having a lot of trouble profiling Python scripts within an Anaconda environment with NSight Systems. I’m on Windows 10.
The issue is specifically with a Python script using tensorflow-gpu. My NVIDIA driver version is 441.68 (from nvidia-smi), so it should be able to support up to CUDA 10.2.
When I run the Python script using tensorflow-gpu via NVIDIA Nsight using the execution command:
C:\Anaconda\envs\test\python.exe test.py
I get the following error:
tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version.
On the other hand, if I run the same python script using the same command in a regular command prompt, it runs successfully.
The anaconda environment I’m using has Python 3.7.x, tensorflow-gpu=1.13.x, and cudatoolkit=10.0.x. (all should be compatible with each other).
Does NVIDIA Nsight Systems somehow run it in a different context?