CUDA version not available message with nvc++ on Ubuntu

I have recently installed version10.7 of the HPC SDK on my 64-bit Ubuntu 20.04 system with TITAN X GPU. When I try to compile a simple C++17 test program, I get a message such as “Error-CUDA version 10.2 is not available in this installation . Please read documentation for CUDA_HOME to solve this issue”. My system has the nvidia-driver-440 and nvidia-cuda-toolkit packages installed. I note that nvcc is located in the /usr/bin directory; and reports 10.1 as the release number.

https://docs.nvidia.com/hpc-sdk/index.html
https://docs.nvidia.com/hpc-sdk/compilers/c+±parallel-algorithms/index.html
https://docs.nvidia.com/hpc-sdk/compilers/hpc-compilers-user-guide/index.html#cuda-toolkit-versions

I began by following the blog post “Accelerating Standard C++ with GPUs Using stdpar”; and then worked through some suggestions via the three links above. I can only ever get the message to indicate that it is another version of CUDA that it can’t find. The command I start with is:

/opt/nvidia/hpc_sdk/Linux_x86_64/20.7/compilers/bin/nvc++ -stdpar help.cpp

I began by adding -gpu=cuda10.1 and/or -gpu=cc60 (Pascal). I also tried setting CUDA_HOME to /usr/bin. Recognising that there are a few CUDA versions included with the SDK I also tried first with the environment commands below, and then tried a few other -gpu=CUDAX.Y options.

NVARCH=`uname -s`_`uname -m`; export NVARCH
NVCOMPILERS=/opt/nvidia/hpc_sdk; export NVCOMPILERS
MANPATH=$MANPATH:$NVCOMPILERS/$NVARCH/20.7/compilers/man; export MANPATH
PATH=$NVCOMPILERS/$NVARCH/20.7/compilers/bin:$PATH; export PATH

nvaccelinfo reports the CUDA Driver Version as 10020.

Here is one C++ program which can trigger the error:

#include <algorithm>
#include <iostream>
#include <vector>
#include <execution>

int main()
{
  std::vector<int> v = {5,100,3,6,6,109,64,234,656,25,7,44,6,232,2};
  std::sort(std::execution::par_unseq, v.begin(), v.end());
  std::cout << v[0] << '\n';
  return 0;
}

Any help would be much appreciated.

Paul

Hi Paul,

Which bundle of the NVIDIA SDK did you download? There’s two. One with just the latest CUDA version (11.0) and one that includes the current and past two releases of CUDA (11,0, 10.2, and 10.1). My guess is that you installed the CUDA 11.0 only bundle.

Note that you don’t have to download the larger bundle and instead use you’re own local install of CUDA by setting the environment variable to point to your CUDA installation. In your case, it sounds like you have the CUDA 10.1 SDK installed, so you’ll also need to tell the compiler to target CUDA 10.1 on the compile line (via the “-gpu=cuda10.1” flag). By default, the compiler with use the CUDA version of your driver.

To summarize some of your options, you can do one of the following:

Download and install the HPC SDK which includes the older CUDA versions.
Update your CUDA driver to CUDA 11.0
Install the CUDA 10.2 SDK and set CUDA_HOME to point to this install
Set CUDA_HOME to your CUDA 10.1 install and add the flag “-gpu=cuda10.1”.

Hope this helps,
Mat

Hi Mat,

Thanks for getting back so quickly. It was indeed the NVIDIA HPC SDK version which includes just the latest CUDA release (nvhpc_2020_207_Linux_x86_64_cuda_11.0.tar.gz). When I use the “-gpu=cuda10.1” flag, the error message then references 10.1 instead of 10.2:

$ /opt/nvidia/hpc_sdk/Linux_x86_64/20.7/compilers/bin/nvc++ -gpu=cuda10.1 -stdpar help.cpp 
nvc++-Error-CUDA version 10.1 is not available in this installation. Please read documentation for CUDA_HOME to solve this issue

Your 4th option suggests setting CUDA_HOME. If I set this to the location of nvcc (i.e. `/usr/bin), it changes nothing. Does CUDA use the CUDA_HOME variable only to locate nvcc? (Otherwise I might be able to create another directory and use symbolic links.)

Thanks again,
Paul

No, CUDA_HOME should point to the base directory for your CUDA SDK installation, which typically is located in “/opt/cuda…” or “/usr/local/cuda…”. Though users can install CUDA in any directory they choose, so it may be elsewhere depending upon where you or your admin installed it.

We don’t use the “nvcc” driver directly but rather the underlying CUDA back-end compiler (libnvvm) and CUDA runtime libraries.

Thanks again Mat. I continued with your other suggestions. I have the latest versions of the CUDA driver and CUDA SDK packages available on Ubuntu 20.04, and so the last option which I can look at is to:

Download and install the HPC SDK which includes the older CUDA versions.

I couldn’t find information on how to uninstall the (nvhpc_2020_207_Linux_x86_64_cuda_11.0) HPC SDK (perhaps just remove the directory?), but I can of course install nvhpc_2020_207_Linux_x86_64_cuda_multi in a custom location (I chose /opt/nvidia/hpc_sdk_multi). After doing this my compilation command produces a different error message:

$ /opt/nvidia/hpc_sdk_multi/Linux_x86_64/20.7/compilers/bin/nvc++ -gpu=cuda10.1 -gpu=cc60 -stdpar ~/projects/cpp-std-parallel/help.cpp
"/opt/nvidia/hpc_sdk_multi/Linux_x86_64/20.7/compilers/include/nvhpc/stdpar_con
          fig.hpp", line 19: catastrophic error: #error directive: An
          unexpected version of Thrust which is incompatible with NVC++ is on
          the include path. NVC++ includes its own version of Thrust; no
          user-supplied version should be needed.

I then tried removing the libthrust-dev package, but this also invokes the removal of the nvidia-cuda-toolkit (and nvidia-cuda-dev). (I don’t want to do this, but I’m curious, so I do it; I can always reinstall.) After that, I add in -std=c++17 and it works; and note that the -gpu=cuda10.1 and -gpu=cc60 flags aren’t necessary (CUDA version 10.2 seems to be the default) here.

$ /opt/nvidia/hpc_sdk_multi/Linux_x86_64/20.7/compilers/bin/nvc++ -stdpar -std=c++17 help.cpp

I still need CUDA installed, so I test the versions included with the SDK. Of the three shipped with the HPC SDK, versions 10.1 and 10.2 produce the ol’ unsupported GNU version! gcc versions later than 8 are not supported! error, but version 11 works:

$ /opt/nvidia/hpc_sdk_multi/Linux_x86_64/20.7/cuda/11.0/bin/nvcc my_simple_test.cu

…but not with anything substantial … via https://github.com/NVIDIA/cuda-samples/archive/v11.0.tar.gz:

$ /opt/nvidia/hpc_sdk_multi/Linux_x86_64/20.7/cuda/11.0/bin/nvcc -I ../../Common matrixMul.cu
$ ./a.out 
[Matrix Multiply Using CUDA] - Starting...
CUDA error at ../../Common/helper_cuda.h:793 code=35(cudaErrorInsufficientDriver) "cudaGetDeviceCount(&device_count)"

This sends me back to the “gcc versions later than 8 not supported!” error. I install g++-8 and put symbolic links named gcc and g++ in each of the 3 version-numbered CUDA bin directories in the HPC SDK. 10.1 works, but so does 10.2, so I go with that newer version (11.0 fails at runtime as before):

$ ln -s /usr/bin/gcc-8 /opt/nvidia/hpc_sdk_multi/Linux_x86_64/20.7/cuda/10.2/bin/gcc
$ ln -s /usr/bin/g++-8 /opt/nvidia/hpc_sdk_multi/Linux_x86_64/20.7/cuda/10.2/bin/g++

This suits me for now. (I’m glad to stick with package managed drivers; I’ve lost too much time on NVIDIA proprietary drivers in the past.)

The only questions I would ask now are: what is the correct way to uninstall the SDK? And with CUDA 10.2 from the SDK, would CUDA_HOME be /opt/nvidia/hpc_sdk_multi/Linux_x86_64/20.7/cuda/10.2/bin?

Cheers,
Paul

what is the correct way to uninstall the SDK?

Assuming you mean the HPC SDK, then you can just delete it. i.e. “rm -rf /opt/nvidia/hpc_sdk_multi”.

And with CUDA 10.2 from the SDK, would CUDA_HOME be

I don’t set CUDA_HOME unless I’m pointing to a CUDA SDK install outside of the HPC SDK install tree so haven’t done this, but would think it would be “/opt/nvidia/hpc_sdk_multi/Linux_x86_64/20.7/cuda/10.2” (i.e. drop the bin).

-Mat

1 Like

I am struggling with a related issue. I am on CentOS 7, with CUDA 10.1, 10.2, 11 installed.
When I try to compile the code above from the OP, I get:
nvc++ cudapar.cpp -std=c++17 -O3 -stdpar -o gpu_tes -gpu=cuda10.2
#error An unexpected version of Thrust which is incompatible with NVC++ is on the include path. NVC++ includes its own version of Thrust; no user-supplied version should be needed.

My include paths, as reported by:
echo | gcc -E -Wp,-v -

/home/shared/software/cuda/hpc_sdk/Linux_x86_64/20.9/comm_libs/nvshmem/include
/home/shared/software/cuda/hpc_sdk/Linux_x86_64/20.9/comm_libs/nccl/include
/home/shared/software/cuda/hpc_sdk/Linux_x86_64/20.9/comm_libs/mpi/include
/home/shared/software/cuda/hpc_sdk/Linux_x86_64/20.9/math_libs/include
/home/shared/software/cuda/hpc_sdk/Linux_x86_64/20.9/cuda/include
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/include
/usr/local/include
/usr/include
(none of the system paths contain a directory called “thrust”)

When I use CUDA 10.2 (setting CUDA_HOME there), and pass -gpu=cuda10.2, I get the same issue. I don’t know what version of thrust might get included that causes the error…

Hmm, not sure why this is happening. Can you try adding the flag “–trace_includes” so we can see which Thrust is getting picked up? In particular, we’re looking for the "thrust/version.h” header file,

Thanks,
Mat

This one
/home/shared/software/cuda/hpc_sdk/Linux_x86_64/20.9/compilers/include/nvhpc/stdpar_config.hpp
checks for
#if THRUST_VERSION < 100910
#error An unexpected version of Thrust which is incompatible with NVC++ is on the include path. NVC++ includes its own version of Thrust; no user-supplied version should be needed.
#endif

and includes
/home/shared/software/cuda/hpc_sdk/Linux_x86_64/20.9/cuda/include/thrust/version.h
which comes through a symlink to
/home/shared/software/cuda/hpc_sdk/Linux_x86_64/20.9/cuda/11.0/include/thrust/version.h
which does
#define THRUST_VERSION 100909

In contrast to this one:
/home/shared/software/cuda/hpc_sdk/Linux_x86_64/20.9/compilers/include-stdpar/thrust/version.h
which does have
#define THRUST_VERSION 101000

If I compile with
nvc++ -I/home/shared/software/cuda/hpc_sdk/Linux_x86_64/20.9/compilers/include-stdpar vector_copy_gpu.cpp -std=c++17 -O3 -o gpu_test -stdpar

then things work!! Yey :) Not sure why it’s first checking for the wrong one.

Sorry for a lot of bold fonts, but that’s because I post the verbose output here. I’ve been battling with the same problem.

  1. I clean install NVIDIA HPC SDK from the tarball nvhpc_2021_213_Linux_x86_64_cuda_11.2.tar.gz
  2. Follow the instructions at Installation Guide Version 21.2, set all the paths
  3. Try to complile the c++ code from the beginning of this thread with

nvc++ -stdpar -std=c++17 -O3 cuda_cpp.cpp -o cuda

and get

/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/bin/tools/cpp1 --llalign -Dunix -D__unix -D__unix__ -Dlinux -D__linux -D__linux__ -D__NO_MATH_INLINES -D__LP64__ -D__x86_64 -D__x86_64__ -D__LONG_MAX__=9223372036854775807L ‘-D__SIZE_TYPE__=unsigned long int’ ‘-D__PTRDIFF_TYPE__=long int’ -D__amd64 -D__amd64__ -D__k8 -D__k8__ -D__MMX__ -D__SSE__ -D__SSE2__ -D__SSE3__ -D__SSSE3__ -D__SSE4_1__ -D__SSE4_2__ -D__AVX__ -D__AVX2__ -D__AVX512F__ -D__AVX512VL__ -D__FMA__ -D__XSAVE__ -D__POPCNT__ -D__FXSR__ -D__PGI -D__NVCOMPILER -D_GNU_SOURCE -D_PGCG_SOURCE --c++17 -I- -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/include_man -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/include-gcc70 -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/include -I/usr/include/c++/7 -I/usr/include/x86_64-linux-gnu/c++/7 -I/usr/include/c++/7/backward -I/usr/lib/gcc/x86_64-linux-gnu/7/include -I/usr/local/include -I/usr/lib/gcc/x86_64-linux-gnu/7/include-fixed -I/usr/include/x86_64-linux-gnu -I/usr/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/include_man -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/include-gcc70 -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/include -I/usr/lib/gcc/x86_64-linux-gnu/7/include -I/usr/local/include -I/usr/lib/gcc/x86_64-linux-gnu/7/include-fixed -I/usr/include/x86_64-linux-gnu -I/usr/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/include-stdpar -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/math_libs/11.2/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/cuda/11.2/include -D__PGLLVM__ -D__NVCOMPILER_LLVM__ -D__extension__= -D_ACCEL=201003 -D_OPENACC=201711 -DCUDA_VERSION=11020 -DPGI_TESLA_TARGET -D__PGI_CUDA_ARCH__=600 -D__NVCOMPILER_CUDA_ARCH__=600 -D__PGI_ACC_UMEM -D__STDPAR_GPU__ --preinclude cplus_preinclude.h --preinclude_macros cplus_macros.h --gnu_version=70500 -D__pgnu_vsn=70500 --cuda -D__CUDACC -D__NV_NO_HOST_COMPILER_CHECK --preinclude _cuda_preinclude.h --accel --preinclude openacc_predef.h --accel_managed -q -o /tmp/nvc++knXeExUg15Pk.il cuda_cpp.cpp
“/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/include/nvhpc/stdpar_config.hpp”, line 19: catastrophic error: #error directive: An unexpected version of Thrust which is incompatible with NVC++ is on the include path. NVC++ includes its own version of Thrust; no user-supplied version should be needed.
#error An unexpected version of Thrust which is incompatible with NVC++ is on the include path. NVC++ includes its own version of Thrust; no user-supplied version should be needed.
^

1 catastrophic error detected in the compilation of “cuda_cpp.cpp”.
Compilation terminated.
nvc+±Fatal-cpp1 completed with exit code 4

If I try compile with
nvc++ -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/include-stdpar -std=c++17 -gpu=cuda11.2 -stdpar=gpu cuda_cpp.cpp cuda

I get

/usr/bin/ld: cannot find cuda: No such file or directory
pgacclnk: child process exit status 1: /usr/bin/ld

I tried uninstalling and reinstalling other versions of HPC SDK with the latest and multiple version of the CUDA toolkit 11.1, 11.2, tried it with std=c++17 and without I still can’t compile it. Please help.

The full verbose output from nvc++ -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/include-stdpar -std=c++17 -gpu=cuda11.2 -stdpar=gpu cuda_cpp.cpp cuda -v

is as follows:

Export PGI_CURR_CUDA_HOME=/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/cuda/11.2
Export NVCOMPILER=/opt/nvidia/hpc_sdk/Linux_x86_64/21.2
Export PGI=/opt/nvidia/hpc_sdk
cuda_cpp.cpp:

/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/bin/tools/cpp1 --llalign -Dunix -D__unix -D__unix__ -Dlinux -D__linux -D__linux__ -D__NO_MATH_INLINES -D__LP64__ -D__x86_64 -D__x86_64__ -D__LONG_MAX__=9223372036854775807L ‘-D__SIZE_TYPE__=unsigned long int’ ‘-D__PTRDIFF_TYPE__=long int’ -D__amd64 -D__amd64__ -D__k8 -D__k8__ -D__MMX__ -D__SSE__ -D__SSE2__ -D__SSE3__ -D__SSSE3__ -D__SSE4_1__ -D__SSE4_2__ -D__AVX__ -D__AVX2__ -D__AVX512F__ -D__AVX512VL__ -D__FMA__ -D__XSAVE__ -D__POPCNT__ -D__FXSR__ -D__PGI -D__NVCOMPILER -D_GNU_SOURCE -D_PGCG_SOURCE --c++17 -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/include-stdpar -I- -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/include_man -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/include-gcc70 -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/include -I/usr/include/c++/7 -I/usr/include/x86_64-linux-gnu/c++/7 -I/usr/include/c++/7/backward -I/usr/lib/gcc/x86_64-linux-gnu/7/include -I/usr/local/include -I/usr/lib/gcc/x86_64-linux-gnu/7/include-fixed -I/usr/include/x86_64-linux-gnu -I/usr/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/include_man -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/include-gcc70 -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/include -I/usr/lib/gcc/x86_64-linux-gnu/7/include -I/usr/local/include -I/usr/lib/gcc/x86_64-linux-gnu/7/include-fixed -I/usr/include/x86_64-linux-gnu -I/usr/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/include-stdpar -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/math_libs/11.2/include -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/cuda/11.2/include -D__PGLLVM__ -D__NVCOMPILER_LLVM__ -D__extension__= -D_ACCEL=201003 -D_OPENACC=201711 -DCUDA_VERSION=11020 -DPGI_TESLA_TARGET -D__PGI_CUDA_ARCH__=600 -D__NVCOMPILER_CUDA_ARCH__=600 -D__PGI_ACC_UMEM -D__STDPAR_GPU__ --preinclude cplus_preinclude.h --preinclude_macros cplus_macros.h --gnu_version=70500 -D__pgnu_vsn=70500 --cuda -D__CUDACC -D__NV_NO_HOST_COMPILER_CHECK --preinclude _cuda_preinclude.h --accel --preinclude openacc_predef.h --accel_managed -q -o /tmp/nvc++KR1eSftnOQpg.il cuda_cpp.cpp

/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/bin/tools/cpp2 cuda_cpp.cpp -opt 2 -x 119 0xa10000 -x 122 0x40 -x 123 0x1000 -x 127 4 -x 127 17 -x 19 0x400000 -x 28 0x40000 -x 120 0x10000000 -x 70 0x8000 -x 122 1 -x 125 0x20000 -quad -x 59 4 -tp skylake -x 120 0x1000 -astype 0 -fn cuda_cpp.cpp -il /tmp/nvc++KR1eSftnOQpg.il -x 117 0x200 -x 123 0x80000000 -x 123 4 -x 119 0x20 -def __pgnu_vsn=70500 -x 70 0x40000000 -x 183 4 -x 121 0x800 -x 6 0x20000 -x 249 100 -x 120 0x200000 -x 70 0x40000000 -x 8 0x40000000 -x 164 0x800000 -x 85 0x2000 -x 85 0x4000 -x 68 0x1 -x 39 4 -x 56 0x10 -x 26 0x10 -x 26 1 -x 56 0x4000 -accel tesla -cudaroot /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/cuda/11.2 -x 176 0x100 -cudacap 60 -x 180 0x4000400 -x 121 0xc00 -x 186 0x80 -x 176 0x100 -cudacap 60 -x 163 0x1 -x 186 0x80000 -cudaver 11020 -x 194 0x40000 -cudaroot /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/cuda/11.2 -x 180 0x4000400 -x 121 0xc00 -y 210 8 -x 189 0x8000 -y 163 0xc0000000 -x 189 0x10 -y 189 0x4000000 -cudaroot /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/cuda/11.2 -x 187 0x40000 -x 187 0x8000000 -x 194 0x20000000 -x 198 0x100 -gnuvsn 70500 -x 69 0x200 -x 123 0x400 -x 137 1 -cmdline ‘+nvc++ /tmp/nvc++KR1eSftnOQpg.il -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/include-stdpar -std=c++17 -gpu=cuda11.2 -stdpar=gpu -v’ -asm /tmp/nvc++KR1eSmkz6xOs.ll
/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/bin/tools/nvdd -dcuda /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/cuda/11.2 -usenvvm -nvvm70 -reloc /tmp/pgacceV1emvCeK-uG.gpu -computecap 60 -ptx /tmp/pgacceV1em02RNCKg.ptx -o /tmp/pgaccKV1eSvTCebiX.bin -cuda11020
/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/bin/tools/nvdd -dcuda /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/cuda/11.2 -reloc -cuda11020 -fat cuda_cpp.cpp -sm 60 /tmp/pgaccKV1eSvTCebiX.bin -compute 60 /tmp/pgacceV1em02RNCKg.ptx -o /tmp/pgacceV1emdyKZsiG.fat
NVC++/x86-64 Linux 21.2-0: compilation successful

/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/share/llvm/bin/opt -O2 /tmp/nvc++KR1eSmkz6xOs.ll -S -o /tmp/nvc++eR1emYBK69Lu.llvm

/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/share/llvm/bin/llc /tmp/nvc++eR1emYBK69Lu.llvm -march=x86-64 -mcpu=native -O2 -fast-isel=0 -non-global-value-max-name-size=4294967295 -x86-cmov-converter=0 -filetype=obj -o cuda_cpp.o

/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/bin/tools/acclnk -nvidia /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/bin/tools/nvdd -cuda11020 -cudaroot /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/cuda/11.2 -cudalink -computecap=60 -init=ctxrel -init=managed -init=cuda -v /usr/bin/ld /usr/lib/x86_64-linux-gnu/crt1.o /usr/lib/x86_64-linux-gnu/crti.o /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/lib/trace_init.o /usr/lib/gcc/x86_64-linux-gnu/7/crtbegin.o --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/lib/nvhpc.ld -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/math_libs/11.2/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/cuda/11.2/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-linux-gnu/7 cuda_cpp.o cuda -rpath /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/lib -rpath /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/math_libs/11.2/lib64 -rpath /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/cuda/11.2/lib64 -rpath /usr/lib/gcc/x86_64-linux-gnu/7/…/…/…/…/lib64 -L/usr/lib/gcc/x86_64-linux-gnu/7/…/…/…/…/lib64 /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/lib/acc_init_link_cuda.o -lacchost -lacccuda -lnvhpcman -lcudadevrt -lcudart /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/lib/cuda_init_register_end.o -ldl -lcudadevice -lcudanvhpc101 -latomic -lnvhpcatm -lstdc++ -lnvomp -ldl --as-needed -lnvhpcatm -latomic --no-as-needed -lpthread --start-group -lnvcpumath -lnsnvc -lnvc --end-group -lm -lgcc -lc -lgcc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/7/crtend.o /usr/lib/x86_64-linux-gnu/crtn.o
/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/bin/tools/nvdd -dcuda /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/cuda/11.2 /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/lib/trace_init.o -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/math_libs/11.2/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/cuda/11.2/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-linux-gnu/7 cuda_cpp.o -L/usr/lib/gcc/x86_64-linux-gnu/7/…/…/…/…/lib64 /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/lib/acc_init_link_cuda.o -lacchost -lacccuda -lnvhpcman -lcudadevrt -lcudart /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/lib/cuda_init_register_end.o -ldl -lcudadevice -lcudanvhpc101 -latomic -lnvhpcatm -lstdc++ -lnvomp -ldl -lpthread -lnvcpumath -lnsnvc -lnvc -lm -lgcc -lc -lgcc_s -dolink -cuda11020 -computecap 60 -o /tmp/pgcuda531eRT4uajAz.cubin -regobj /tmp/pgcudaregX31ettM725IN.o
/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/bin/tools/nvdd -fatobj /tmp/pgcudafatb31edBEChrlg.o -o /tmp/pgcudafatb31edBEChrlg.o -cuda11020 -dcuda /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/cuda/11.2 -init=ctxrel -init=managed -init=cuda -cudalink -sm 60 /tmp/pgcuda531eRT4uajAz.cubin
/usr/bin/ld /usr/lib/x86_64-linux-gnu/crt1.o /usr/lib/x86_64-linux-gnu/crti.o /tmp/pgcudafatb31edBEChrlg.o /tmp/pgcudaregX31ettM725IN.o /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/lib/trace_init.o /usr/lib/gcc/x86_64-linux-gnu/7/crtbegin.o --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/lib/nvhpc.ld -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/math_libs/11.2/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/cuda/11.2/lib64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-linux-gnu/7 cuda_cpp.o cuda -rpath /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/lib -rpath /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/math_libs/11.2/lib64 -rpath /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/cuda/11.2/lib64 -rpath /usr/lib/gcc/x86_64-linux-gnu/7/…/…/…/…/lib64 -L/usr/lib/gcc/x86_64-linux-gnu/7/…/…/…/…/lib64 /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/lib/acc_init_link_cuda.o -lacchost -lacccuda -lnvhpcman -lcudadevrt -lcudart /opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/lib/cuda_init_register_end.o -ldl -lcudadevice -lcudanvhpc101 -latomic -lnvhpcatm -lstdc++ -lnvomp -ldl --as-needed -lnvhpcatm -latomic --no-as-needed -lpthread --start-group -lnvcpumath -lnsnvc -lnvc --end-group -lm -lgcc -lc -lgcc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/7/crtend.o /usr/lib/x86_64-linux-gnu/crtn.o
/usr/bin/ld: cannot find cuda: No such file or directory
pgacclnk: child process exit status 1: /usr/bin/ld
unlink /tmp/pgcudafatb31edBEChrlg.o
unlink /tmp/pgcudaregX31ettM725IN.o
unlink /tmp/pgcuda531eRT4uajAz.cubin
nvc+±Fatal-linker completed with exit code 1

Unlinking /tmp/nvc++KR1eSftnOQpg.il
Unlinking /tmp/nvc++eR1emZRzFO1e.s
Unlinking /tmp/nvc++KR1eSmkz6xOs.ll
Unlinking /tmp/nvc++eR1emYBK69Lu.llvm

Thanks to ofbinull I got my code running, although it looks like a hack, and a simple nvc++ -stdpar -std=c++17 cuda_cpp.cpp -o cuda should be able to do the job. So this is what worked for me:

nvc++ -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.2/compilers/include-stdpar -fast -g -Wall -stdpar -c -o cuda_cpp.o cuda_cpp.cpp

and then

nvc++ -o cuda cuda_cpp.o -stdpar

What doesn’t seem right is that I have to add a link to include-stdpar and the strange error with Thrust if I follow the instruction in this blog

Hi nightman_roman,

/usr/bin/ld: cannot find cuda: No such file or directory
pgacclnk: child process exit status 1: /usr/bin/ld

This error is simply because you accidently removed the “-o” flag before “-o cuda”, so the compiler is passing this as a file to the linker.

As for the Thrust warning message, this occurs when include paths are picking up an older version of the Thrust headers file. Could you have Thrust installed on this system under one of the system include paths? i.e. is there a “/usr/include/thrust” or “/usr/local/include/thrust” directory?

-Mat