Problem with numa

Accelerated Computing HPC Compilers Legacy PGI Compilers
1

I have wrote en an “hybrid code” (MPI/openmp) which compile, built and executed well with Gfortran and Ifort.

I meet some difficulties with pgfortran, and I hope that you can help me.

The first one is when I want to compile code’s source with only openmp, everything seem to be ok during compiling phase, but during the link phase I met this message :

pgf90 -I/uv100/usr/local/fftw-3.3.3/include -r8 -O3 -fast -mp -Mpreprocess -Dkey_fftw_openmp -Dkey_no_adios -c GP_3D_stat_communication.f90
PGF90-W-0006-Input file empty (GP_3D_stat_communication.f90)
PGF90/x86-64 Linux 13.10-0: compilation completed with warnings
pgf90 -o GP2DS_OPENMP -mp GP_3D_stat_main.o GP_3D_stat_var_def.o GP_3D_stat_compute.o GP_3D_stat_IO.o GP_3D_stat_fft_compute.o GP_3D_stat_models.o GP_3D_stat_communication.o GP_3D_stat_allocation.o VTR_mod.o VTK_mod.o -L/uv100/usr/local/fftw-3.3.3/lib -lm -lfftw3 -lfftw3_omp
/usr/bin/ld: cannot find -lnuma
make: *** [GP2DS_OPENMP] Erreur 2

For information I’m using a UV-100 computer and my makefile is :

FC = pgf90

FFTW

FFTW3DIR =/uv100/usr/local/fftw-3.3.3
FFTW3LIB = -L$(FFTW3DIR)/lib
FFTW3INC = -I$(FFTW3DIR)/include
#if ADIOS (option, only in 3D)
ADIOS = nok
ADIOSDIR = FOR_VIZU_3D/adios-1.5.0
MXMLDIR = FOR_VIZU_3D/mxml-2.7
HDF5DIR = FOR_VIZU_3D/hdf5-1.8.11/hdf5
ADIOSINC = -I$(ADIOSDIR)/include
ADIOSLIB = -L$(ADIOSDIR)/lib
MXMLINC = -I$(MXMLDIR)/include
MXMLLIB = -L$(MXMLDIR)/lib
HDF5INC = -I$(HDF5DIR)/include
HDF5LIB = -L$(HDF5DIR)/lib

#EXEC = GP3DS_MPI
EXEC = GP2DS_OPENMP

ifeq ($(EXEC),GP3DS_MPI)
PREPROC = key_mpi
MPI = -lmpi
else
PREPROC = key_fftw_openmp
PREADIOS = key_no_adios
endif

ifeq ($(ADIOS),ok)
PREADIOS = key_adios
else
PREADIOS = key_no_adios
endif

FFLAGS = -r8 -O3 -fast -mp -Mpreprocess -D$(PREPROC) -D$(PREADIOS)
LD = $(FC)
LDFLAGS = -mp

2

Hi Philippe,

You can add “-mp=nonuma” to not link with the NUMA library or determine why the linker can’t find the library.

Often systems have “/usr/lib64/libnuma.so.1” but just need a symbolic link “/usr/lib64/libnuma.so → /usr/lib64/libnuma.so.1” added in order to resolve this library. However, during installation of the compilers if the installer doesn’t detect that your system have “libnuma.so” available, a PGI supplied dummy NUMA library will be included in your PGI installation directory. Why this dummy library wouldn’t have been added to your installation is unclear.

  • Mat
3

Hi mat,

Second option is good!
I created symbolic link.

But now I have a new " link error":
/fftw-3.3.3/lib/libfftw3_omp.a(libfftw3_omp_la-openmp.o): In function fftw_spawn_loop': openmp.c:(.text+0x10f): undefined reference to GOMP_parallel_start’
openmp.c:(.text+0x11c): undefined reference to `GOMP_parallel_end’
make: *** [GP2DS_OPENMP] Erreur 2

Due to fftw 3.3.3.
So does it mean that I need to rebuilt a fftw with PGI compiler, right ?

regards.

4

So does it mean that I need to rebuilt a fftw with PGI compiler, right ?

Correct. If it was serial code, it wouldn’t matter. However for OpenMP, GNU and PGI have different implantations which can’t be mixed.

  • Mat
5

thanks lot.

Regards.

6

Unfortunately the problem is not fixed after I built FFTW3.3.3 library with PGI compilers.

I have follow this recommandation (Account Login | PGI) without problem.
But in order to be homogeneous with my installation, I use FFTW 3.3.3.
Is it a problem ?

For information the link error is still the same.

7

“GOMP” symbols are in the GNU OpenMP runtime, so if you’re still getting them then you’re still building with gcc.

On some systems, “env” doesn’t set the environment variables as expected. You may want to try setting them in your environment before running configure.

  • Mat
8

Correct.

thx.