NVPROF with Error: incompatible CUDA driver version.

Hi Team,

I’m a beginner with CUDA, and I just got an error when I use NVPROF but I don’t know why.
Several days ago when I used only one GPU device, nvprof works fine. And the hardware and software didn’t change these days. I’m wondering what should I do if I hope use nvprof to profile the application. Thanks very much!

I used pgf90 (CUDA Fortran) with -Mcuda.

==22439== NVPROF is profiling process 22439, command: ./Xvicar3dPAT-2.4.0
======== Profiling result:
No kernels were profiled.
No API activities were profiled.
======== Error: incompatible CUDA driver version.

nvidia-smi
Mon Dec 30 09:58:00 2019
±----------------------------------------------------------------------------+
| NVIDIA-SMI 430.50 Driver Version: 430.50 CUDA Version: 10.1 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 TITAN RTX Off | 00000000:01:00.0 On | N/A |
| 41% 37C P8 25W / 280W | 301MiB / 24217MiB | 0% Default |
±------------------------------±---------------------±---------------------+
| 1 TITAN RTX Off | 00000000:05:00.0 Off | N/A |
| 40% 31C P8 25W / 280W | 1MiB / 24220MiB | 0% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
±----------------------------------------------------------------------------+

nvcc --version
nvcc: NVIDIA ® Cuda compiler driver
Copyright © 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

nvprof --version
nvprof: NVIDIA ® Cuda command line profiler
Copyright © 2012 - 2018 NVIDIA Corporation
Release version 10.0.130 (21)

pgf90 --version
pgf90 19.4-0 LLVM 64-bit target on x86-64 Linux -tp haswell
PGI Compilers and Tools
Copyright © 2019, NVIDIA CORPORATION. All rights reserved.

Hi,

Can you please reinstall the CUDA Toolkit? It seems you have different versions of CUDA toolkit and the CUDA driver. CUDA Toolkit 10.0 is compatible with Linux Driver versions >= 410.48. Though CUDA driver is backward compatible with the older CUDA toolkit, but let’s try with the fresh installation.