libpgf90rtl undefined reference omp_set_schedule

I’m getting the following link failure while trying to build a simple
hello world program using the mpif90 for an openmpi installation built with
pgi-13.10 :

/usr/local/pgi/linux86/libso/libpgf90rtl.so: undefined reference to omp_set_schedule' /usr/local/pgi/linux86/libso/libpgf90rtl.so: undefined reference to omp_get_schedule’
pgf90-Fatal-linker completed with exit code 1

Details of compiler version:

% pgf90 --version
pgf90 13.10-0 64-bit target on x86-64 Linux -tp penryn
The Portland Group - PGI Compilers and Tools
Copyright (c) 2013, NVIDIA CORPORATION. All rights reserved.

The pgf90 command issued by mpif90 is:

% mpif90 --showme
pgf90 -I/usr/mpi/pgi/openmpi-1.4.3-qlc/include
-I/usr/mpi/pgi/openmpi-1.4.3-qlc/lib64
-L/usr/mpi/pgi/openmpi-1.4.3-qlc/lib64
-lmpi_f90 -lmpi_f77 -lmpi -lopen-rte -lopen-pal
-ldl -Wl,–export-dynamic -lnsl -lutil -lpthread -ldl

I’ve issued the pgf90 command directly, and supplemented it with
-Wl,-M -Wl,–cref to obtain load map with cross references. What
I find is that libpgf90rtl is not supplying any references for the
hello world executable, but is causing the link to fail because of
the undefined reference problem.

I’ve found a workaround which is to add -Wl,–allow-shlib-undefined to
the load flags. But it doesn’t seem like I should need to do this.

Are these undefined references in libpgf90rtl needed for something?
They don’t appear in version 11.5 of the library which I was using
before updating to 13.10.

Hi Brian,

It looks like for some reason the our mp library isn’t getting added to the link. This can occur if the flag “-nomp” is added, but I don’t see it in your case.

One thing I don’t understand is why the path listed is for the 32-bit lib directory (/usr/local/pgi/linux86/libso). However, “libpgf90rtl.so” is only available in 64-bits and everything else indicates that you’re doing a 64-bit compile. Is the directory name a typo? or did things get moved around in your installation?

If things were moved, can you check if “libpgmp.so” was moved as well? Maybe try adding “-L/usr/local/pgi/linux86/libso -lpgmp”?

  • Mat

Hi Mat,

Actually the pgmp lib is being added to the link command. I should have
included this in my original post. Using the -v option to pgf90 shows that
the actual link command resulting from the pgf90 command in the original
post is:

/usr/bin/ld /usr/lib64/crt1.o /usr/lib64/crti.o
/usr/local/pgi-pgcc-pghf-13.10/linux86-64/13.10/lib/trace_init.o
/usr/lib/gcc/x86_64-redhat-linux/4.4.7/crtbegin.o
/usr/local/pgi-pgcc-pghf-13.10/linux86-64/13.10/lib/initmp.o
/usr/local/pgi-pgcc-pghf-13.10/linux86-64/13.10/lib/f90main.o -m elf_x86_64
-dynamic-linker /lib64/ld-linux-x86-64.so.2
/usr/local/pgi-pgcc-pghf-13.10/linux86-64/13.10/lib/pgi.ld
-L/usr/mpi/pgi/openmpi-1.4.3-qlc/lib64
-L/usr/local/pgi-pgcc-pghf-13.10/linux86-64/13.10/lib -L/usr/lib64
-L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -lmpi_f90 -lmpi_f77 -lmpi
-lopen-rte -lopen-pal --export-dynamic -M --cref -t --allow-shlib-undefined
/tmp/pgf90zsOblXiHsi33.o -rpath
/usr/local/pgi-pgcc-pghf-13.10/linux86-64/13.10/lib -laccapid -lpgmp -lnuma
-lpthread -lpgf90 -lpgf90_rpm1 -lpgf902 -lpgf90rtl -lpgftnrtl -lpgmp
-lnspgc -lpgc -lrt -lpthread -lm -lgcc -lc -lgcc
/usr/lib/gcc/x86_64-redhat-linux/4.4.7/crtend.o /usr/lib64/crtn.o

The soft links are resolved in the above output, so you can see that in our
installation linux86 is a link to linux86-64.

Also, in our local installation /usr/local/pgi/linux86/libso resolves to
/usr/local/pgi-pgcc-pghf-13.10/linux86-64/13.10/libso

The above link command also includes the libs that were added by the mpif90
invocation of pgf90.

However, while the link line contains
/usr/local/pgi-pgcc-pghf-13.10/linux86-64/13.10/lib
it does not contain
/usr/local/pgi-pgcc-pghf-13.10/linux86-64/13.10/libso


If I add “-L/usr/local/pgi/linux86/libso” to the pgf90 command as you
suggest (and remove -Wl,–allow-shlib-undefined) then the build of the
hello world test program works. But without explicitly adding it the build
fails.

Thanks for the suggestion to add the libso path to the link command. That
gives me a workaround which I like better than my original. But I’m still
trying to figure out whether this is the best solution. I have a suspicion
that something could be done in the openmpi installation so that it
wouldn’t be necessary for the user to add this argument to the mpif90
commandline.


Note that /usr/local/pgi/linux86/libso is in LD_LIBRARY_PATH.


I guess my questions are the following:

. Should I need to manually add “-L/usr/local/pgi/linux86/libso” to my link
command. Shouldn’t ld know to look in LD_LIBRARY_PATH which does contain
the libso path?

. Why does pgf90 only add the pgi/linux86-64/lib path and not
pgi/linux86-64/libso? Is there an option I should be using that tells
pgf90 to add the libso path?

. There is a pgmp lib in pgi/linux86-64/lib. It appears to have
definitions for the omp_set_schedule and omp_get_schedule references
which are causing the link failure. Why isn’t libpgmp.a satisfying those
references?

Thanks,
Brian

Hi Brian,

This one has got me confused!

What I don’t understand is why “libso/libpgf90rtl.so” is getting linked in the first place. Nowhere on your link line does “libso” show up. Maybe it’s getting picked up via LD_LIBRARY_PATH, but I’d think the rpath would take precedence.

. Should I need to manually add “-L/usr/local/pgi/linux86/libso” to my link
command. Shouldn’t ld know to look in LD_LIBRARY_PATH which does contain the libso path?

No, you shouldn’t. And if LD_LIBRARY_PATH is being used to find “libpgf90rtl.so”, why isn’t it finding “libpgmp.so”? odd.

. Why does pgf90 only add the pgi/linux86-64/lib path and not
pgi/linux86-64/libso? Is there an option I should be using that tells
pgf90 to add the libso path?

The default is to link using the static libraries. Adding “-Bdynamic” or “-fpic” will cause the driver to use “libso” instead of “lib”.

. There is a pgmp lib in pgi/linux86-64/lib. It appears to have
definitions for the omp_set_schedule and omp_get_schedule references
which are causing the link failure. Why isn’t libpgmp.a satisfying those
references?

While I’m not 100% positive, I don’t think a shared object can resolve it’s symbols to a static library.

The one thought that comes to mind is if someone has put links to the shared objects in your lib directory but missed “libpgmp.so”? That might cause this.

If not, then can you add the flag “-Wl,–verbose” to have the linker show us how it’s resolving the libraries?

  • Mat

Hi Mat,

I’ve run the following command:

 mpif90 -Wl,-M,--cref,--verbose test.f90 >| mpif90-test.out 2>&1

where test.f90 contains the 3 line hello world program.

The --verbose output contains:

libpgf90rtl.so needed by /usr/mpi/pgi/openmpi-1.4.3-qlc/lib64/libmpi_f90.so
found libpgf90rtl.so at /usr/local/pgi/linux86/libso/libpgf90rtl.so

So it is the openmpi libs added by the mpif90 command which is triggering
the problem, as expected.

But the --verbose output indicates the linker was able to locate
libpgf90rtl.so (I assume via the LD_LIBRARY_PATH), and yet doesn’t seem
able to figure out that omp_set_schedule is resolved by libpgmp.so in the
same directory.

So this is still feeling like something strange in the pgi shared libs.

I didn’t see a way to attach the output which contains all the linkage
info, so I put it here: ftp://ftp.cgd.ucar.edu/pub/eaton/mpif90-test.out.gz

Thanks for your help with this!
Brian

Hi Brian,

Looks like I was wrong about dynamic libs not linking with static. The verbose output shows libpgfrtl.so resolving several symbols in libpgmp.a. The question now is why all but these two symbols are getting resolved. Using nm, I see the symbol names however the order is slightly different relative to the internal references. Let me talk with some folks and see if they have ideas.

In the meantime, try adding “-Bdynamic”.


% nm libpgmp.a | grep omp_set_sche
                 U omp_set_schedule
0000000000000273 T omp_set_schedule_
                 U omp_set_schedule
0000000000000274 T omp_set_schedule__
0000000000000221 T omp_set_schedule
% nm libpgmp.a | grep omp_set_nested
0000000000000128 T omp_set_nested
                 U omp_set_nested
0000000000000084 T omp_set_nested_
                 U omp_set_nested
0000000000000084 T omp_set_nested__
  • Mat

Adding -Bdynamic works. That seems like the best solution. Or is it a
workaround? I’m not really sure whether pgf90 should be producing a
successful build or not. The openmpi libs are all shared, but shouldn’t
the link be able to successfully resolve externals from a combination of
shared and static libs?

Thanks for all your help.
Brian

LD_LIBRARY_PATH sets paths for dynamic libs that are reconciled
at runtime. But the link has to be successful first, so the proper libpgmp.so
needs to be found at link time.

Possible workaround - copy libpgmp.so into the PGI lib directory, or
put a link to the libpgmp.so in the libso directory.