Issue with /opt/pgi/linux86-64/14.7/lib/libstdz.a

Accelerated Computing HPC Compilers Legacy PGI Compilers
1

All,

I’ve been encountering some issues with PGI 14.7 and ESMF 6.3.0rp1 today and I’ve sent emails to PGI Support about them. However, when I work around the main issue, the build ends with:

/usr/bin/ld: /opt/pgi/linux86-64/14.7/lib/libstdz.a(collate.o): relocation R_X86_64_32S against `__vtbl__Q2_3std16collate__tm__2_c' can not be used when making a shared object; recompile with -fPIC
/opt/pgi/linux86-64/14.7/lib/libstdz.a: could not read symbols: Bad value

My usual fix in this case is find the original library and recompile with -fPIC, but I can’t do that here…can I? Is this something we did bad on our end when installing PGI 14.7? If so, how can we correct it? (Note: Our sysadmins install PGI for me–my only install experience is the Free PGI for OS X–so I probably don’t know if I’m even in the right ballpark.)

Or is there some flag I should pass in?

Thanks,
Matt

2

Note, I should say ESMF is built with -fpic (not -fPIC):

ESMF_F90COMPILEOPTS: -O -fpic -mcmodel=small -lpthread
ESMF_CXXCOMPILEOPTS: -O -DNDEBUG -fpic -mcmodel=small -lpthread
3

Hi Matt,

Does you’re link line include “-fpic” or “-mcmodel=medium” as well?

The FPIC compiled versions of the these libraries are over in the “libso” directory (i.e. “/opt/pgi/linux86-64/14.7/libso/libstdz.a”). When “-fpic”, “-mcmodel=medium” or “-Bdynamic” are used on the link line, these versions are used.

We’ve had this set-up for several years now so I’m not sure why your just now seeing it with 14.7. Perhaps there’s something else wrong, but that would be the first thing I’d try.

  • Mat
4

Mat,

The final link step was:

mpicxx -shared -o /ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_1_0/src/esmf/lib/libO/Linux.pgi.64.openmpi.default/libesmf_fullylinked.so AdaptiveKDTree.o AEntityFactory.o AffineXform.o all_allo.o assign_param_vals.o AxisBox.o bind_param.o BitPage.o BitTag.o 
...
zz_struct.o zz_util.o -mcmodel=small -lpthread -Wl,--no-as-needed -L/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_1_0/src/esmf/lib/libO/Linux.pgi.64.openmpi.default -L/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_1_0/x86_64-unknown-linux-gnu/pgfortran_14.7-openmpi_1.8.1/Linux/lib -L/opt/pgi/linux86-64/14.7/lib -Wl,-rpath,/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_1_0/src/esmf/lib/libO/Linux.pgi.64.openmpi.default -Wl,-rpath,/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_1_0/x86_64-unknown-linux-gnu/pgfortran_14.7-openmpi_1.8.1/Linux/lib -Wl,-rpath,/opt/pgi/linux86-64/14.7/lib -lmpi_mpifh -pgf90libs -ldl -L/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_1_0/x86_64-unknown-linux-gnu/pgfortran_14.7-openmpi_1.8.1/Linux/lib -lnetcdff -lnetcdf -L/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_1_0/x86_64-unknown-linux-gnu/pgfortran_14.7-openmpi_1.8.1/Linux/lib -lnetcdff -lnetcdf -L/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_1_0/x86_64-unknown-linux-gnu/pgfortran_14.7-openmpi_1.8.1/Linux/lib -L/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_1_0/x86_64-unknown-linux-gnu/pgfortran_14.7-openmpi_1.8.1/Linux/lib -lnetcdf -ljpeg -lmfhdf -ldf -lhdf5_hl -lhdf5 -lm -L/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_1_0/x86_64-unknown-linux-gnu/pgfortran_14.7-openmpi_1.8.1/Linux/lib -lmfhdf -ldf -lsz -ljpeg -L/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_1_0/x86_64-unknown-linux-gnu/pgfortran_14.7-openmpi_1.8.1/Linux/lib -lcurl -lssl -lcrypto -lssl -lcrypto -lz -lrt -lm -lm -L/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_1_0/x86_64-unknown-linux-gnu/pgfortran_14.7-openmpi_1.8.1/Linux/lib -lcurl -lssl -lcrypto -lssl -lcrypto -lz -lrt -lm

This seems to correspond with:

ESMF_F90LINKOPTS:  -mcmodel=small -lpthread -Wl,--no-as-needed
ESMF_CXXLINKOPTS:  -mcmodel=small -lpthread -Wl,--no-as-needed
ESMF_SL_LIBS_TO_MAKE: libesmf
ESMF_SL_SUFFIX:       so
ESMF_SL_LIBLINKER:    mpicxx
ESMF_SL_LIBOPTS:       -shared
ESMF_SL_LIBLIBS:

And from the man page for pgcpp:

       -shared
              (Linux only) Passed to the linker. Instructs the linker to generate a shared object file (dynamically linked library).
              Implies -fpic.

If that’s right, then the link line got both -mcmodel=small and -fpic (from shared).

Matt

5

I’ve taken a closer look at this and I’ve seen that this issue only appears when ESMF 6.3.0rp1 tries to link a new shared library libesmf_fullylinked.so:

#     The "libesmf_fullylinked.so" version links in everything that would be
#     specified in the final link step when building an executable. This
#     produces a fully self-contained shared library, which has the advantage
#     of being suitable to be loaded e.g. by the Python layer on top of ESMF.
#     However, it does require that ALL dependencies, including all of the 
#     3rd party libraries that are specified, MUST be available as 
#     position-independent-code (PIC). This can be difficult to ensure.

So, on one hand I’m lucky in that I don’t even link to this .so (since I never knew it existed and, well, we are a .a shop), but it does seem to indicate that maybe the collate.o in libstdz.a wasn’t compiled with -fPIC?

I’m sending a note to ESMF Support as well about this to see if they see it on their end when they build.

Matt

6

Hi Matt,

Yes “-shared” implies “-fpic” as well. The issue is that the linker is picking up the wrong version of the library. It needs to grab the “/opt/pgi/linux86-64/14.7/libso/libstdz.a” file instead of “opt/pgi/linux86-64/14.7/lib/libstdz.a”. The archive libraries in “libso” are built with “-fpic” while the “lib” version are not.

I’m wondering if it’s a mpicxx driver configure issue. “-shared” should cause the pgcpp driver to add “-L/opt/pgi/linux86-64/14.7/libso” your link but maybe mpicxx is calling ld directly and not adding it? What happens if you explicitly add “-L/opt/pgi/linux86-64/14.7/libso”?

  • Mat
% pgcpp -dryrun x.o -shared
...
/usr/bin/ld /usr/lib64/crti.o /proj/pgi/linux86-64/14.7/libso/trace_init.o /usr/lib/gcc/x86_64-redhat-linux/4.4.7/crtbeginS.o /proj/pgi/linux86-64/14.7/libso/initmp.o -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 /proj/pgi/linux86-64/14.7/lib/pgi.ld -L/proj/pgi/linux86-64/14.7/libso -L/proj/pgi/linux86-64/14.7/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 x.o -rpath /proj/pgi/linux86-64/14.7/libso -rpath /proj/pgi/linux86-64/14.7/lib -shared -lzceh -lgcc_eh --eh-frame-hdr -lstdz -lCz -lpgmp -lnuma -lpthread -lnspgc -lpgc -lm -lgcc -lc -lgcc /usr/lib/gcc/x86_64-redhat-linux/4.4.7/crtendS.o /usr/lib64/crtn.o