Problem with numa

I have wrote en an “hybrid code” (MPI/openmp) which compile, built and executed well with Gfortran and Ifort.

I meet some difficulties with pgfortran, and I hope that you can help me.

The first one is when I want to compile code’s source with only openmp, everything seem to be ok during compiling phase, but during the link phase I met this message :

pgf90 -I/uv100/usr/local/fftw-3.3.3/include -r8 -O3 -fast -mp -Mpreprocess -Dkey_fftw_openmp -Dkey_no_adios -c GP_3D_stat_communication.f90
PGF90-W-0006-Input file empty (GP_3D_stat_communication.f90)
PGF90/x86-64 Linux 13.10-0: compilation completed with warnings
pgf90 -o GP2DS_OPENMP -mp GP_3D_stat_main.o GP_3D_stat_var_def.o GP_3D_stat_compute.o GP_3D_stat_IO.o GP_3D_stat_fft_compute.o GP_3D_stat_models.o GP_3D_stat_communication.o GP_3D_stat_allocation.o VTR_mod.o VTK_mod.o -L/uv100/usr/local/fftw-3.3.3/lib -lm -lfftw3 -lfftw3_omp
/usr/bin/ld: cannot find -lnuma
make: *** [GP2DS_OPENMP] Erreur 2

For information I’m using a UV-100 computer and my makefile is :

FC = pgf90


FFTW3DIR =/uv100/usr/local/fftw-3.3.3
FFTW3INC = -I$(FFTW3DIR)/include
#if ADIOS (option, only in 3D)
ADIOS = nok
ADIOSDIR = FOR_VIZU_3D/adios-1.5.0
MXMLDIR = FOR_VIZU_3D/mxml-2.7
HDF5DIR = FOR_VIZU_3D/hdf5-1.8.11/hdf5
MXMLINC = -I$(MXMLDIR)/include
HDF5INC = -I$(HDF5DIR)/include
HDF5LIB = -L$(HDF5DIR)/lib


ifeq ($(EXEC),GP3DS_MPI)
PREPROC = key_mpi
MPI = -lmpi
PREPROC = key_fftw_openmp
PREADIOS = key_no_adios

ifeq ($(ADIOS),ok)
PREADIOS = key_adios
PREADIOS = key_no_adios

FFLAGS = -r8 -O3 -fast -mp -Mpreprocess -D$(PREPROC) -D$(PREADIOS)
LD = $(FC)

Hi Philippe,

You can add “-mp=nonuma” to not link with the NUMA library or determine why the linker can’t find the library.

Often systems have “/usr/lib64/” but just need a symbolic link “/usr/lib64/ -> /usr/lib64/” added in order to resolve this library. However, during installation of the compilers if the installer doesn’t detect that your system have “” available, a PGI supplied dummy NUMA library will be included in your PGI installation directory. Why this dummy library wouldn’t have been added to your installation is unclear.

  • Mat

Hi mat,

Second option is good!
I created symbolic link.

But now I have a new " link error":
/fftw-3.3.3/lib/libfftw3_omp.a(libfftw3_omp_la-openmp.o): In function fftw_spawn_loop': openmp.c:(.text+0x10f): undefined reference to GOMP_parallel_start’
openmp.c:(.text+0x11c): undefined reference to `GOMP_parallel_end’
make: *** [GP2DS_OPENMP] Erreur 2

Due to fftw 3.3.3.
So does it mean that I need to rebuilt a fftw with PGI compiler, right ?


So does it mean that I need to rebuilt a fftw with PGI compiler, right ?

Correct. If it was serial code, it wouldn’t matter. However for OpenMP, GNU and PGI have different implantations which can’t be mixed.

  • Mat

thanks lot.


Unfortunately the problem is not fixed after I built FFTW3.3.3 library with PGI compilers.

I have follow this recommandation ( without problem.
But in order to be homogeneous with my installation, I use FFTW 3.3.3.
Is it a problem ?

For information the link error is still the same.

“GOMP” symbols are in the GNU OpenMP runtime, so if you’re still getting them then you’re still building with gcc.

On some systems, “env” doesn’t set the environment variables as expected. You may want to try setting them in your environment before running configure.

  • Mat