Cuda fortran link issue

Accelerated Computing HPC Compilers Legacy PGI Compilers
1

Hi All,
I ran into a linking problem with a cuda fortran code (CUF). Demonstrated below with the matmul example from the manual. When using the -fPIC flag, the linker can not find libcublas.so.

Without the ‘-fPIC’ flag:
[ktomko@oakley02 ~/CUF_Examples]$ pgf90 -Mcuda=4.1 -o mmul_pgf90 mmul.cuf

With the ‘-fPIC’ flag:
[ktomko@oakley02 ~/CUF_Examples]$ pgf90 -fPIC -Mcuda=4.1 -o mmul_pgf90 mmul.cuf
/usr/bin/ld: cannot find …/…/2011/cuda/3.2/lib64/libcublas.so

If I add the -# flag to see all of the compilation steps here is what the ld command looks like:

/usr/bin/ld /usr/lib64/crt1.o /usr/lib64/crti.o /usr/local/pgi/12.5/linux86-64/12.5/libso/trace_init.o /usr/lib/gcc/x86_64-redhat-linux/4.4.5/crtbegin.o /usr/local/pgi/12.5/linux86-64/12.5/libso/initmp.o /usr/local/pgi/12.5/linux86-64/12.5/libso/f90main.o -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 /usr/local/pgi/12.5/linux86-64/12.5/lib/pgi.ld -L/usr/local/pgi/12.5/linux86-64/12.5/libso -L/usr/local/pgi/12.5/linux86-64/12.5/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.5 /tmp/pgf90cM5dg8U334jS.o -rpath /usr/local/pgi/12.5/linux86-64/12.5/libso -rpath /usr/local/pgi/12.5/linux86-64/12.5/lib -rpath /usr/local/pgi/12.5/linux86-64/2012/cuda/4.1/lib64 -o mmul_pgf90 -lcudafor4 -lcudaforblas -L/usr/local/pgi/12.5/linux86-64/2012/cuda/4.1/lib64 -lcudart /usr/local/pgi/12.5/linux86-64/12.5/lib/nonuma.o -lpgmp -lpthread -lpgf90 -lpgf90_rpm1 -lpgf902 -lpgf90rtl -lpgftnrtl -lnspgc -lpgc -lrt -lpthread -lm -lgcc -lc -lgcc /usr/lib/gcc/x86_64-redhat-linux/4.4.5/crtend.o /usr/lib64/crtn.o
/usr/bin/ld: cannot find …/…/2011/cuda/3.2/lib64/libcublas.so
pgf90-Fatal-linker completed with exit code 1

One solution is to just not use -fPIC, however the code that I’m building is a hybrid MPI/CUF code and the MPI library invokes pgf90 with the two flags
-noswitcherror -fPIC.

The compiler version is:
[ktomko@oakley02 ~/CUF_Examples]$ pgf90 --version

pgf90 12.5-0 64-bit target on x86-64 Linux -tp nehalem
Copyright 1989-2000, The Portland Group, Inc. All Rights Reserved.
Copyright 2000-2012, STMicroelectronics, Inc. All Rights Reserved.

I have a workaround using a customized mpif90 script, so not a show stopper for me.
-Karen

2

Hi Karen,

This is a known problem that was fixed in 12.6 and only occurs in 12.5 when CUDA 4.1 is used.

Basically what happened is that our engineers forgot to update a ld script to point to the correct directory. Since the directory exists here (we still have the CUDA 3.2 directory installed), we missed the error.

Your options are to use CUDA 4.0 (default), update to 12.6, or fix the ld script.

Here’s the updated cudablas ld script

% cat /usr/local/pgi/linux86-64/12.5/libso/libcudaforblas.so
OUTPUT_FORMAT(elf64-x86-64)
GROUP ( ../../2012/cuda/4.1/lib64/libcublas.so ./libcudaforwrapblas.so )

Sorry for the error,
Mat