compiling for OMP on AMD64

Accelerated Computing HPC Compilers Legacy PGI Compilers
1

Hi,

I am compiling a model under an AMD 64-bit system using PGIF77 and OMP. The model compiles fine, however, I am running into the following loader error: /usr/bin/ld: cannot find -lnuma. Has anyone else encountered this error? If so any help/info as to a resolution would be appreciated. Thanks.

2

Hi asrcMark,


At installation, the installer detects if your system supports NUMA by checking for the existance of “/usr/lib64/libnuma.so” library. If so, then the NUMA library will be added by default when “-mp” is used. So some possible reasons why you might get the error is: either the installer shouldn’t have added NUMA support, the library has been removed, or your trying to compile statically.

Can you post your link line without the objects? Also can you verify that “/usr/lib64/libnuma.so” exists and post which operation system your using?

Thanks,
Mat

3

Hi Mat,

Well, some interesting results. Let me outline for you. The system that I am having the NUMA load problems on is configured as follows: 64-bit RedHat Linux Enterprise 4 (kernel v2.6.9-5), with Version 6.1 of the PGI compilers. The exact same model, and Makefile compiles and builds the OMP version of my model fine on another 64-bit AMD machine that we have running here. That machine is configured as follows: 64-bit RedHat Linux Enterprise 4 (kernel v2.6.9-22.0.1), with Version 6.0 of the PGI compilers. Both systems have the /usr/lib64/libnuma.so linked to /usr/lib64/libnuma.so.1 and both are 21424 bytes long. To rule out RedHat kernel problems, we tarred and moved the PGI version 6.0 compilers over to the system where I’m having the NUMA problem and the model compiled and loaded just fine for OMP using version 6.0 of the PGI compilers on the system that I originally started having problems with.

Here is the identical build command which is used on both of the systems (Makefile is identical):

pg_linuxomp:
@rm -f $(INC)/camx.prm
@csh chktracer camx.prm.$(DOMAIN) CAMx.$(DOMAIN).pg_linuxomp
@ln -s camx.prm.$(DOMAIN) $(INC)/camx.prm
make model FC=“pgf77” FLGS=“-I$(INC) -O2 -mp -tp k8-64 -pc 64 -Mnoframe -byteswapio -Wl,-Bstatic” TARGT=“CAMx.$(DOMAIN).pg_linuxomp” DUM=dummy_omp

where ‘model’ is:

model: $(OBJCTS)
$(FC) -o $(TARGT) $(FLGS) $(OBJCTS) $(LIBS)
.f.o :
$(FC) -c -o $@ $(FLGS) $<

Hope this helps, I’ll be interested in knowing what you diagnose the problem to be! Thanks so much for your help.

Mark
ASRC

4

Hi Mark,

The problem is that your trying to link statically but the NUMA library is only available as a shared library. So I see three possible solutions.

  1. If you don’t really need the application to be linked statically, then remove “-Wl,-Bstatic” from your link.

  2. If you need the the application to be mostly static but are willing to have the NUMA library be dynamic, then add the following “rc” command to either “$PGI/linux64-64/6.1-1/bin/siterc” (your path may be different) if you want the change to occur for all users and all the compilers, or to “~/.mypgf77rc” if you only want the change to apply to your pgf77 links.

set NUMALIBS=-Bdynamic -lnuma -Bstatic ;
  1. If you need the application to be completely static and are willing to not use NUMA, then add the following to either “siterc” or “.mypgf77rc”.
set NUMALIBS= ;
set NUMAOBJS= ;

Note that I’ve submitted a feature request to add a “-mp=nonuma” flag which will allow you to disable NUMA support from the command line. Also, in order to enable NUMA support with the 6.0 release, use “-mp=numa” instead of “-mp”.

Hope this helps,
Mat

5

Hi Mat,

Thanks for the help. I was able to temporarily disable the linking of the NUMA library using the ‘siterc’ method you provided on the system running the 6.1 release. However, I would like to test using NUMA on both systems, and when I tried the ‘-mp=numa’ switch for the 6.0 release, I get a pgf77 error with an unrecognized switch error for ‘numa’. Any advice?

Thanks again for the assistance.
Mark

6

Hi Mark,

I should have been more clear. “-mp=numa” was added to the 6.0-5 release of the compilers. Earlier versions would give you the unknown keyword error.

Sorry for the confusion.
Mat

7

Hi Mat,

No problem, our Center has purchased the site licence for the PGI products and our sys admin team is in the process of upgrading all PGI products on all servers. Currently, we are waiting on the 6.1 license for my one server and when that upgrade happens I’ll run NUMA.

thanks for the help
Mark

8

I seem to be running with similar compilation/ linking problems with version 6.1 of pgf90. I want to link the rather large Fortran program statically; but there is no static NUMA library on the system (i.e. there is no libnuma.a anywhere) so I want to opt out of using NUMA (while still compiling for parallelism because the Fortran program uses OpenMP). The man page for version 6.1 says that it does have the “-mp=nonuma” option but when I try to use it, I get the unknown keyword error,
William Nicholson

9

Hi William,

Yes, the “-mp=nonuma” option should have been added to 6.1, but didn’t make it in until 6.2.

The work around is to create a “~/.mypgf90rc” file in your home directory and add the following lines:

set NUMAOBJS=;
set NUMALIBS=;

This will change will only effect your linking. You can also change your installations ‘localrc’ file (found int PGI bin directories), but this changes the setting for all users.

Hope this helps,
Mat