Static linking with pgf90 and Intel-MKL library

Dear experts,

I am trying to do static linking to generate static executable using PGI and the Intel-MKL library, and I am getting multiple linking errors (see below).

This is 2.6.32-431.20.3.el6.x86_64 Linux system, pgf90 13.10-0 64-bit target on x86-64.

Any help on what library should be installed and/or inking options added ?

Yours, Miro

milias@login.grid.umb.sk:~/Work/qch/software/dirac/trunk_release/build_pgi_mkl_i8_dbg_static/./mnt/apps/pgi/linux86-64/13.10/bin/pgf90 -Bstatic -Wl,–no-export-dynamic -Wl,-E -DVAR_PGF90 -Bstatic -i8 -g CMakeFiles/dirac.x.dir/src/main/main.F90.o -o dirac.x lib/libdirac.a lib/libxcfun.a -Wl,–start-group /mnt/apps/intel/mkl/lib/intel64/libmkl_intel_ilp64.a /mnt/apps/intel/mkl/lib/intel64/libmkl_pgi_thread.a /mnt/apps/intel/mkl/lib/intel64/libmkl_core.a /usr/lib64/libpthread.a /usr/lib64/libm.a -mp -Wl,–end-group -Wl,–start-group /mnt/apps/intel/mkl/lib/intel64/libmkl_lapack95_ilp64.a /mnt/apps/intel/mkl/lib/intel64/libmkl_intel_ilp64.a -mp -Wl,–end-group -Wl,–no-export-dynamic /mnt/apps/intel/mkl/lib/intel64/libmkl_pgi_thread.a /mnt/apps/intel/mkl/lib/intel64/libmkl_core.a /usr/lib64/libpthread.a /usr/lib64/libm.a /mnt/apps/intel/mkl/lib/intel64/libmkl_lapack95_ilp64.a -lzceh -lstdz -lCz -lstdc++ -lpthread -ldl

^L/mnt/apps/intel/mkl/lib/intel64/libmkl_core.a(mkl_aa_fw_load_orsl_lite_lib.o): In function mkl_aa_fw_load_orsl_lite_lib': ../../../../serv/offload/framework/core/mkl_aa_fw_load_orsl_lite_lib.c:(.text+0xbe): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /usr/lib64/libpthread.a(libpthread.o): In function sem_open’:
(.text+0x774d): warning: the use of mktemp' is dangerous, better use mkstemp’
/usr/lib64/libpthread.a(libpthread.o): In function __pthread_initialize_minimal_internal': (.text+0x39): undefined reference to __libc_setup_tls’
/usr/lib64/libpthread.a(libpthread.o): In function __pthread_initialize_minimal_internal': (.text+0x7e): undefined reference to _dl_cpuclock_offset’
/usr/lib64/libpthread.a(libpthread.o): In function __pthread_initialize_minimal_internal': (.text+0x275): undefined reference to __libc_init_secure’
/usr/lib64/libpthread.a(libpthread.o): In function __pthread_initialize_minimal_internal': (.text+0x2b3): undefined reference to _dl_pagesize’
/usr/lib64/libpthread.a(libpthread.o): In function __pthread_initialize_minimal_internal': (.text+0x2c4): undefined reference to _dl_init_static_tls’
/usr/lib64/libpthread.a(libpthread.o): In function __pthread_initialize_minimal_internal': (.text+0x2cf): undefined reference to _dl_wait_lookup_done’
/usr/lib64/libpthread.a(libpthread.o): In function __pthread_create_2_1': (.text+0x1151): undefined reference to _dl_stack_flags’
/usr/lib64/libpthread.a(libpthread.o): In function __pthread_create_2_1': (.text+0x1979): undefined reference to _dl_stack_flags’
/usr/lib64/libpthread.a(libpthread.o): In function vfork': (.text+0x8a41): undefined reference to __syscall_error’
/usr/lib64/libpthread.a(libpthread.o): In function __write_nocancel': (.text+0x8a68): undefined reference to __syscall_error’
/usr/lib64/libpthread.a(libpthread.o): In function __write_nocancel': (.text+0x8a9c): undefined reference to __syscall_error’
/usr/lib64/libpthread.a(libpthread.o): In function __read_nocancel': (.text+0x8ac8): undefined reference to __syscall_error’
/usr/lib64/libpthread.a(libpthread.o): In function __read_nocancel': (.text+0x8afc): undefined reference to __syscall_error’
/usr/lib64/libpthread.a(libpthread.o):(.text+0x8b28): more undefined references to __syscall_error' follow /usr/lib64/libdl.a(dlopen.o): In function dlopen’:
(.text+0x5): undefined reference to __dlopen' /usr/lib64/libdl.a(dlsym.o): In function dlsym’:
(.text+0x5): undefined reference to __dlsym' /usr/lib64/libdl.a(dlvsym.o): In function dlvsym’:
(.text+0x5): undefined reference to __dlvsym' /usr/lib64/libdl.a(dladdr.o): In function dladdr’:
(.text+0x1): undefined reference to `__dladdr’
milias@login.grid.umb.sk:~/Work/qch/software/dirac/trunk_release/build_pgi_mkl_i8_dbg_static/.

Hi Miro,

Instead of “-Bstatic”, try using “-Bstatic_pgi”. This will statically link the PGI runtime libraries, but dynamically link the system libraries. The errors are because libpthreads needs to be dynamically linked.

  • Mat

Hi Mat,
thanks for your hint - “-Bstatic_pgi” works with the Intel MKL library. The working linking command has the form of

mnt/apps/pgi/linux86-64/13.10/bin/pgf90 -g     -Bstatic_pgi -mp -Minfo -fast  CMakeFiles/example.dir/example.F90.o  -o ../bin/example  -Wl,--start-group /mnt/apps/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/libmkl_lapack95_lp64.a /mnt/apps/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/libmkl_intel_lp64.a -mp -Wl,--end-group -Wl,--start-group /mnt/apps/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/libmkl_intel_lp64.a /mnt/apps/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/libmkl_pgi_thread.a /mnt/apps/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/libmkl_core.a -ld

and the resulting executable contains only dynamic system libraries (I don’t see MKL libs - these are probably hidden as static inside exec):

milias@login.grid.umb.sk:~/Work/qch/software/software_projects/autocmake_devel/autocmake_miroi/test/fc_lapack/build_static_pgf90_pgcc_mkl/.m
[ 50%] Built target strcmp
make[2]: Warning: File `src/CMakeFiles/example.dir/link.txt' has modification time 23 s in the future
Linking Fortran executable ../bin/example
make[2]: warning:  Clock skew detected.  Your build may be incomplete.
[100%] Built target example
milias@login.grid.umb.sk:~/Work/qch/software/software_projects/autocmake_devel/autocmake_miroi/test/fc_lapack/build_static_pgf90_pgcc_mkl/.ldd bin/example 
        linux-vdso.so.1 =>  (0x00007fff58e83000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fa1d1633000)
        libnuma.so => /usr/lib64/libnuma.so (0x00007fa1d1429000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa1d120c000)
        librt.so.1 => /lib64/librt.so.1 (0x00007fa1d1004000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fa1d0d7f000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fa1d09eb000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fa1d1850000)
milias@login.grid.umb.sk:~/Work/qch/software/software_projects/autocmake_devel/autocmake_miroi/test/fc_lapack/build_static_pgf90_pgcc_mkl/.bin/example 
 dgesv test ok
milias@login.grid.umb.sk:~/Work/qch/software/software_projects/autocmake_devel/autocmake_miroi/test/fc_lapack/build_static_pgf90_pgcc_mkl/.

But the problem is in making pure (-Bstatic) static executable of pgf90+MKL. To avoid the known linking error of “/usr/bin/ld: dynamic STT_GNU_IFUNC symbol strcmp' with pointer equality in /usr/lib64/libc.a(strcmp.o)’ can not be used when making an executable”, I prepared my own strcmp (libstrcmp.a) static library, and put everything into one linking command as:

/mnt/apps/pgi/linux86-64/13.10/bin/pgf90 -Bstatic  -Mpreprocess -Kieee  -fast -O3 -Mipa=fast CMakeFiles/example.dir/example.F90.o  -o ../bin/example  -L/lib64  -L/mnt/apps/intel/composer_xe_2013_sp1.1.106/compiler/lib/intel64  -L/mnt/apps/intel/composer_xe_2013_sp1.1.106/ipp/lib/intel64  -L/mnt/apps/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64  -L/mnt/apps/intel/composer_xe_2013_sp1.1.106/tbb/lib/intel64/gcc4.4  libstrcmp.a -Wl,--start-group /mnt/apps/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/libmkl_lapack95_lp64.a /mnt/apps/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/libmkl_intel_lp64.a -mp -Wl,--end-group -Wl,--start-group /mnt/apps/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/libmkl_intel_lp64.a /mnt/apps/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/libmkl_pgi_thread.a /mnt/apps/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/libmkl_core.a /usr/lib64/libpthread.a /usr/lib64/libm.a -ldl -mp -Wl,--end-group /home/milias/Work/qch/software/software_projects/autocmake_devel/autocmake_miroi/test/fc_lapack/build_static_pgf90_mkl_6/libc.a

Well, compilation with warnings with this linking command still gave partially dynamic executable

Linking Fortran executable ../bin/example
IPA: no IPA optimizations for 1 source files
/mnt/apps/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/libmkl_core.a(mkl_aa_fw_load_orsl_lite_lib.o): In function `mkl_aa_fw_load_orsl_lite_lib':
../../../../serv/offload/framework/core/mkl_aa_fw_load_orsl_lite_lib.c:(.text+0xbe): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/lib64/libpthread.a(libpthread.o): In function `sem_open':
(.text+0x774d): warning: the use of `mktemp' is dangerous, better use `mkstemp'
make[2]: warning:  Clock skew detected.  Your build may be incomplete.
[100%] Built target example
milias@login.grid.umb.sk:~/Work/qch/software/software_projects/autocmake_devel/autocmake_miroi/test/fc_lapack/build_static_pgf90_pgcc_mkl/.ldd bin/example 
        linux-vdso.so.1 =>  (0x00007fffd3370000)
        libnuma.so => /usr/lib64/libnuma.so (0x00007f28356bc000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f283549e000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f2835296000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f2835012000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f2834c7d000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f28358de000)

but not working, unfortunately, giving Segmentation fault.

Any help, please, to convince pgf90 to produce fully static executable with the MKL library ? I have “pgf90 13.10-0 64-bit target on x86-64 Linux -tp nehalem”.

Miro

Linking the pthread library statically is generally not advised. It’s tied to the glibc version making the code difficult to move between systems or can cause the program to crash if there’s an update to system.

I know from another post you got this working, which is good, but if you do see crashes like this in the future, you may need to go back to using the dynamic pthread library.

  • Mat

This open project

https://github.com/miroi/mathlibs-tester

works with many fortran/library combinations, including pgf90 + MKL +i8+ static-linking.

See for example CDash board, https://testboard.org/cdash/index.php?project=MathLibs-tester&date=2015-08-31.

The Autocmake part, responsible for generating buildup commands, got fixed for pgf90-mkl-static linking. The problem was “-mp” flag in the linking command, which was removed when static linking activated.