Nvlink fatal

Hi,

I’m getting

nvlink fatal : Input file ‘/global/common/cori/software/cuda/10.0/lib64/libcudadevrt.a:cuda_device_runtime.o’ newer than toolkit (100 vs 91)

No matter which versions of cuda and pgi are used.

It even occurred when other version of cuda lib are explicitly specified for the linking line:

mpif90 -Mcuda -lcudaforblas /global/common/cori/software/cuda/9.1/lib64/libcudadevrt.a -o mycuda.x
nvlink fatal : Input file ‘/global/common/cori/software/cuda/10.0/lib64/libcudadevrt.a:cuda_device_runtime.o’ newer than toolkit (100 vs 91)

Do you know what happened here?

Thanks,

– Jing

Hi Jing,

Do you know what happened here?

Not sure, but I’m guessing that you have a mismatch someplace on which CUDA runtime you’re using to link and the nvlink version. You can try adding the verbose flag (-v) to your link and post the output. We should be able to then see where the mismatch is.

Also, try compiling and linking with “-Mcuda=cuda10.0” to explicitly set the CUDA version. Also link using “-Mcudalib=cublas” instead of explicitly adding “-lcudaforblas” so the correct cuBlas library version is brought in.

Are you setting “CUDA_HOME” in your environment? By default we’ll use the CUDA components we ship but users can use their own CUDA installation when CUDA_HOME is set. Given the error references a library in your CUDA install, I’m guessing this is set. Using “-Mcuda=cuda10.0” will override this.

-Mat

Hi Mat,

“-v” clears shows that CUDA_HOME was set to point “9.1” and this was the reason for linking error.

Then “-Mcuda=cuda10.0” was used to explicitly set it to “10.1” and the error was gone.

Thanks a lot and have a good day.

– Jin

Hi Mat,

One more thing. What’s the syntax for using multiple cu library, such as

cuBLAS, cuFFT, cuRAND, and cuSPARSE, cuSOLVER

?

Is “-Mcudalib=cublas,cufft, curand,cusolver” right?

Is “-Mcudalib=cublas,cufft, curand,cusolver” right?

Correct, a comma delineated list. However, no space between “,” and “curand”.

-Mat

Thanks.

– Jin

Hi Mat,
I met the same problem when compiling my project.

nvcc -O3 -w -arch=compute_61 -code=sm_75 -gencode=arch=compute_61,code=sm_75 -Xcompiler -fopenmp -Xcompiler -mfma main.cu -o test -I/usr/local/cuda-11.6/include -L/usr/local/cuda-11.6/lib64  -lcudart  -lcusparse   -D VALUE_TYPE=double
nvlink fatal   : Input file '/usr/local/cuda-11.6/lib64/libcudadevrt.a:cuda_device_runtime.o' newer than toolkit (116 vs 101) (target: sm_75)
make: *** [Makefile:29: make] Error 1

I’m using cuda11.6, but I “apt-get install nvidia-cuda-toolkit”(cuda 10.1) at a mistake ,while it already got a cuda-toolkit 11.6 existed. And, it turns out to make my environment a mass. As you have mentioned before, I used -v option:

nvcc -O3 -w -arch=compute_61 -code=sm_75 -gencode=arch=compute_61,code=sm_75 -v -Xcompiler -fopenmp -Xcompiler -mfma main.cu -o test -I/usr/local/cuda-11.6/include -L/usr/local/cuda-11.6/lib64  -lcudart  -lcusparse  -D VALUE_TYPE=double
#$ _SPACE_= 
#$ _CUDART_=cudart
#$ _HERE_=/usr/lib/nvidia-cuda-toolkit/bin
#$ _THERE_=/usr/lib/nvidia-cuda-toolkit/bin
#$ _TARGET_SIZE_=
#$ _TARGET_DIR_=
#$ _TARGET_SIZE_=64
#$ NVVMIR_LIBRARY_DIR=/usr/lib/nvidia-cuda-toolkit/libdevice
#$ PATH=/usr/lib/nvidia-cuda-toolkit/bin:/home/zhanglx/.vscode-server/bin/c3511e6c69bb39013c4a4b7b9566ec1ca73fc4d5/bin/remote-cli:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/zhanglx/cuda-11.6/bin:/home/zhanglx/cuda-11.6/bin:/usr/local/cuda-11.6/bin
#$ LIBRARIES=  -L/usr/lib/x86_64-linux-gnu/stubs -L/usr/lib/x86_64-linux-gnu
#$ gcc -D__CUDA_ARCH__=610 -E -x c++  -DCUDA_DOUBLE_MATH_FUNCTIONS -D__CUDACC__ -D__NVCC__  -fopenmp -mfma -O3 -w -I"/usr/local/cuda-11.6/include"  -D "VALUE_TYPE=double" -D__CUDACC_VER_MAJOR__=10 -D__CUDACC_VER_MINOR__=1 -D__CUDACC_VER_BUILD__=243 -include "cuda_runtime.h" -m64 "main.cu" > "/tmp/tmpxft_00272df0_00000000-8_main.cpp1.ii" 
#$ cicc --c++14 --gnu_version=90400 -w --allow_managed  -arch compute_61 -m64 -ftz=0 -prec_div=1 -prec_sqrt=1 -fmad=1 --include_file_name "tmpxft_00272df0_00000000-2_main.fatbin.c" -tused -nvvmir-library "/usr/lib/nvidia-cuda-toolkit/libdevice/libdevice.10.bc" --gen_module_id_file --module_id_file_name "/tmp/tmpxft_00272df0_00000000-3_main.module_id" --orig_src_file_name "main.cu" --gen_c_file_name "/tmp/tmpxft_00272df0_00000000-5_main.cudafe1.c" --stub_file_name "/tmp/tmpxft_00272df0_00000000-5_main.cudafe1.stub.c" --gen_device_file_name "/tmp/tmpxft_00272df0_00000000-5_main.cudafe1.gpu"  "/tmp/tmpxft_00272df0_00000000-8_main.cpp1.ii" -o "/tmp/tmpxft_00272df0_00000000-5_main.ptx"
#$ ptxas -w -arch=sm_75 -m64 "/tmp/tmpxft_00272df0_00000000-5_main.ptx"  -o "/tmp/tmpxft_00272df0_00000000-9_main.cubin" 
#$ fatbinary --create="/tmp/tmpxft_00272df0_00000000-2_main.fatbin" -64 --cmdline="" "--image=profile=sm_75,file=/tmp/tmpxft_00272df0_00000000-9_main.cubin" --embedded-fatbin="/tmp/tmpxft_00272df0_00000000-2_main.fatbin.c" 
#$ rm /tmp/tmpxft_00272df0_00000000-2_main.fatbin
#$ gcc -E -x c++ -D__CUDACC__ -D__NVCC__  -fopenmp -mfma -O3 -w -I"/usr/local/cuda-11.6/include"  -D "VALUE_TYPE=double" -D__CUDACC_VER_MAJOR__=10 -D__CUDACC_VER_MINOR__=1 -D__CUDACC_VER_BUILD__=243 -include "cuda_runtime.h" -m64 "main.cu" > "/tmp/tmpxft_00272df0_00000000-4_main.cpp4.ii" 
#$ cudafe++ --c++14 --gnu_version=90400 -w --allow_managed --m64 --parse_templates --gen_c_file_name "/tmp/tmpxft_00272df0_00000000-5_main.cudafe1.cpp" --stub_file_name "tmpxft_00272df0_00000000-5_main.cudafe1.stub.c" --module_id_file_name "/tmp/tmpxft_00272df0_00000000-3_main.module_id" "/tmp/tmpxft_00272df0_00000000-4_main.cpp4.ii" 
#$ gcc -D__CUDA_ARCH__=610 -c -x c++  -DCUDA_DOUBLE_MATH_FUNCTIONS -fopenmp -mfma -O3 -w -I"/usr/local/cuda-11.6/include" -m64 -o "/tmp/tmpxft_00272df0_00000000-10_main.o" "/tmp/tmpxft_00272df0_00000000-5_main.cudafe1.cpp" 
#$ nvlink --arch=sm_75 --register-link-binaries="/tmp/tmpxft_00272df0_00000000-6_test_dlink.reg.c"  -w -m64 -L"/usr/local/cuda-11.6/lib64" -lcudart -lcusparse   -L/usr/lib/x86_64-linux-gnu/stubs -L/usr/lib/x86_64-linux-gnu -cpu-arch=X86_64 -report-arch "/tmp/tmpxft_00272df0_00000000-10_main.o"  -lcudadevrt  -o "/tmp/tmpxft_00272df0_00000000-11_test_dlink.cubin"
nvlink fatal   : Input file '/usr/local/cuda-11.6/lib64/libcudadevrt.a:cuda_device_runtime.o' newer than toolkit (116 vs 101) (target: sm_75)
# --error 0x1 --
make: *** [Makefile:29: make] Error 1

should I fix my environment or add some compiling option to solve this?
it would be highly appreciated !

Hi 2509957071,

What’s the environment variable “CUDA_HOME” set to?
What’s the output from “nvcc -V” and “which nvcc”?

Your PATH contains three different directories which look to be CUDA bin directories. Though “nvcc” is most likely being invoked from “/usr/lib/nvidia-cuda-toolkit/bin” and I’m guessing it is nvcc v10.1.

Then there’s a fourth directory, “/usr/local/cuda-11.6/”, being used to find the include headers and CUDA libraries which is causing the mismatch. You probably have CUDA_HOME set to this directory.

To fix, make sure that the nvcc in your PATH matches version being used in CUDA_HOME.

-Mat