OpenACC, CUDA, driver mismatch issue

[imanhosseini.17]

I do:

$ nvc++ -acc -Minfo=accel -o t0 -gpu:cc75 t0.cpp
nvc++-Error-A CUDA toolkit matching the current driver version (0) or a supported older version (11.0) was not installed with this HPC SDK.

Despite:

$ nvidia-smi
Mon Apr 24 16:42:03 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.61.05    Driver Version: 531.68       CUDA Version: 12.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:01:00.0 Off |                  N/A |
| N/A   56C    P8     2W /  N/A |     19MiB /  4096MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A        19      G   /Xwayland                       N/A      |
|    0   N/A  N/A        20      G   /Xwayland                       N/A      |
+-----------------------------------------------------------------------------+

The error is weird, driver version (0)? Also, I do have nvcc and it does work:

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Jan__6_16:45:21_PST_2023
Cuda compilation tools, release 12.0, V12.0.140
Build cuda_12.0.r12.0/compiler.32267302_0
$ which nvcc
/opt/nvidia/hpc_sdk/Linux_x86_64/23.3/compilers/bin/nvcc

All of these are run from the same shell, so the nvcc we see here, is exposed to the nvc++ process and it shoud be able to pick it up.
Any thoughts what’s going on here?
More info:

$ nvc++ --version

nvc++ 23.3-0 64-bit target on x86-64 Linux -tp haswell 
NVIDIA Compilers and Tools
Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

I installed this bundled with cuda 12.

[MatColgrove]

Hi imanhosseini.17,

As far as I can tell, 531.68 is a Windows driver. Are you using WSL2?

If so, it’s likely that our runtime can’t find the CUDA driver (libcuda.so). Try setting the LD_LIBRARY_PATH in your environment to include “/usr/lib/wsl/lib” or where ever libcuda.so was installed.

Note, you can tell if this is the problem by running the ‘nvaccelinfo’ utility, rather than nvidia-smi, since it uses the same look-up as our runtime.

If you’re running native Linux, it still likely a similar issue where the nvc++ runtime can’t find the driver. Posting the output from “nvaccelinfo” would be helpful.

-Mat

[imanhosseini.17]

Thanks Mat!
I am using WSL2 and setting LD_LIBRARY_PATH fixed it for me.