MVAPICH2 and OpenMPI with NVIDIA compiler (20.7)

Has anybody successfully compiled mvapich2-2.3.4 or openmpi-4.0.4 with the nvidia 20.7 compilers?

I can’t built mvapich2 and I can’t run openmpi once built.
PGI support hasn’t got back to me, and last time I checked the mvapich ppl have yet to try using 20.7

I’ve build mvapich and openmpi from source code with numerous previous PGI versions (20.4, 19.9, 19.5, 18.7, 18.1) but 20.7 fails miserably - you get what you paid for?

I have the same issue, and I’ve been watching this topic waiting for some kind of response. I’ve been waiting to push out these compilers to our clusters, as we have used PGI for years with Open MPI without issue on both x86_64 and ppc64le. I have had some users ask about the new HPC suite, but at this point I have to recommend to them that they continue to use the older PGI as well as Intel and gcc for now. For our internal benchmarks, the Open MPI provided with the suite is 1.5x slower than gcc/openmpi.

Hi Slaughts,

Our support team was working with Sylvain_k via email and why I didn’t respond to this post.

There was some missing logic in our installer to check if customer already has InfiniBand libraries installed on their system, and use those rather than the ones we provide with Open MPI as a placeholder for customers who do not have InfiniBand (it was looking /usr/etc not /etc). Hence it was falling back to Ethernet. We’re expecting this to be addressed in the 20.9 release.

Not sure if this is the same issue that you’re seeing.

-Mat

Mat - these are two separate issues:

1- the NVIDIA installer is broken/incomplete on systems w/ IB - this was indeed fixed by removing libs by hand - leaving the NVIDIA distro of OpenMPI 3.x working, except for an annoying warning abt shared lib version.

2- I also build from source MVAPICH2 and OpenMPI 4.x - those two builds w/ 20.7 fail, they work fine w/ previous PGI compilers.

[I have 30 years of experience using PGI on HPC - some good and some memorably horrible - now back to square one w/ NVIDIA]

Sylvain.

Sylvain G. Korzennik, Ph.D. - SI/SAO HPC analyst.

Hi Sylvain,

I had assumed that you sent Customer Support both issues, but I just look at the support email and the only report I see from you is about the IB issue. Apologies that I didn’t know this was still an open issue.

We don’t typically build MVAPICH2 here, but let me give it a try and see if I can get it to build. Though any details about the error you’re seeing would be helpful.

I know there’s been problems in the past with OpenMPI 4.x (I don’t have details off-hand) but believe they’ve been resolved and we’ll be shipping a pre-built version with the 20.9 compilers.

-Mat

1- I’ve open a support ticket w/ PGI support for the MVAPICH2 and OpenMPI 4.x
2- The MVAPICH2 ppl had yet to try building w/ 20.7 when I contacted them
3- I tried various ways to build and it kept failing (adding -fPIC, lowering -O value)
4- At this point, I’m writing off 20.7 for MPI, and will wait to see what 20.9 will break ;-)

Also, while I have you attention, I contacted NVIDIA to ask abt support when our current PGI support expires (mid-2021) and got no answer. While this is over 6 mos away, knowing a price point would help us budget it for our next FY budget (that starts Oct 1),

Best,
Sylvain.

Who did you contact? We’re in a bit of transition period where NVIDIA Enterprise Services is taking over the support services (though Brent and I will still take the escalated issues, and I’ll still monitor the HPC Compiler user forums), but requests for quotes should be directed to them at enterpriseservices@nvidia.com

I emailed to renewalsales@nvidia.com (as per the NVIDIA HPC website, BTW) I will fwd it to enterpriseservices@nvidia.com - Thx.

PGI customers would be considered new under NVIDIA Enterprise Services unless you already had other services through them. Though, I would have expected some response even if you sent the note to the renewals email. Hopefully sending to the general email will help.

I know for me, 4.x compiles clean with PGI 20.7, but mpirun segfaults when you try to run anything, MPI or not. 4.x did not compile clean using nvc vs pgcc. I was able to modifly some flags to get it to build, but it segfaults as well.

As a side question, are nvc and pgcc supposed to be the same compiler, or are there differences internally?

If you don;t mind, I’d like to wait for 20.9 to see if the OpenMPI 4.x that we ship will work for you. If you still encounter issues, I’ll pull in our engineer that does our builds and see if we can trouble shoot yours.

As a side question, are nvc and pgcc supposed to be the same compiler, or are there differences internally?

With 20.7, nvc is the rebranded version of pgcc so are the same. Though pgcc did change in the 20.1 to a new C compiler (it’s actually our C++ compiler but in C mode) so depending on which PGI you’re comparing to, there may or may not be differences. In the 19.x compilers we shipped a beta version of the new C compiler called “pgcc18”.

I can wait. It shouldn’t take long once 20.9 is released to get 4.x built and tested.

I was able to build Open MPI 4.0.5 with 20.9 specifying pgcc/pgc++/pgfortran as the compilers. It built clean and doesn’t segfault now. When I try to build using nvc/nvc++/nvfortran, the build fails with the following errors:

/usr/bin/ld: .libs/comm_spawn_multiple_f08.o: relocation R_X86_64_32S against `.rodata’ can not be used when making a shared object; recompile with -fPIC

/usr/bin/ld: .libs/startall_f08.o: relocation R_X86_64_32S against `.rodata’ can not be used when making a shared object; recompile with -fPIC

/usr/bin/ld: .libs/testall_f08.o: relocation R_X86_64_32S against `.rodata’ can not be used when making a shared object; recompile with -fPIC

/usr/bin/ld: .libs/testany_f08.o: relocation R_X86_64_32S against `.rodata’ can not be used when making a shared object; recompile with -fPIC

/usr/bin/ld: .libs/testsome_f08.o: relocation R_X86_64_32S against `.rodata’ can not be used when making a shared object; recompile with -fPIC

/usr/bin/ld: .libs/type_create_struct_f08.o: relocation R_X86_64_32S against `.rodata’ can not be used when making a shared object; recompile with -fPIC

/usr/bin/ld: .libs/type_get_contents_f08.o: relocation R_X86_64_32S against `.rodata’ can not be used when making a shared object; recompile with -fPIC

/usr/bin/ld: .libs/waitall_f08.o: relocation R_X86_64_32S against `.rodata’ can not be used when making a shared object; recompile with -fPIC

/usr/bin/ld: .libs/waitany_f08.o: relocation R_X86_64_32S against `.rodata’ can not be used when making a shared object; recompile with -fPIC

/usr/bin/ld: .libs/waitsome_f08.o: relocation R_X86_64_32S against `.rodata’ can not be used when making a shared object; recompile with -fPIC

/usr/bin/ld: profile/.libs/pcomm_spawn_multiple_f08.o: relocation R_X86_64_32S against `.rodata’ can not be used when making a shared object; recompile with -fPIC

/usr/bin/ld: profile/.libs/pstartall_f08.o: relocation R_X86_64_32S against `.rodata’ can not be used when making a shared object; recompile with -fPIC

/usr/bin/ld: profile/.libs/ptestall_f08.o: relocation R_X86_64_32S against `.rodata’ can not be used when making a shared object; recompile with -fPIC

/usr/bin/ld: profile/.libs/ptestany_f08.o: relocation R_X86_64_32S against `.rodata’ can not be used when making a shared object; recompile with -fPIC

/usr/bin/ld: profile/.libs/ptestsome_f08.o: relocation R_X86_64_32S against `.rodata’ can not be used when making a shared object; recompile with -fPIC

/usr/bin/ld: profile/.libs/ptype_create_struct_f08.o: relocation R_X86_64_32S against `.rodata’ can not be used when making a shared object; recompile with -fPIC

/usr/bin/ld: profile/.libs/ptype_get_contents_f08.o: relocation R_X86_64_32S against `.rodata’ can not be used when making a shared object; recompile with -fPIC

/usr/bin/ld: profile/.libs/pwaitall_f08.o: relocation R_X86_64_32S against `.rodata’ can not be used when making a shared object; recompile with -fPIC

/usr/bin/ld: profile/.libs/pwaitany_f08.o: relocation R_X86_64_32S against `.rodata’ can not be used when making a shared object; recompile with -fPIC

/usr/bin/ld: profile/.libs/pwaitsome_f08.o: relocation R_X86_64_32S against `.rodata’ can not be used when making a shared object; recompile with -fPIC

/usr/bin/ld: .libs/abort_f08.o: relocation R_X86_64_PC32 against symbol `ompi_abort_f’ can not be used when making a shared object; recompile with -fPIC

/usr/bin/ld: final link failed: Bad value

I can probably work around these, but aren’t the compilers supposed to be the same now?

The compilers are the same, but I believe that Open MPI has some GNU autoconf templates that it uses to generate files like ./configure and libtool, and these templates are not yet aware of the rebranded NVIDIA HPC compiler names. Thus, their build infrastructure is not yet aware that the nv* compilers also support the -fPIC flag, which is needed here to link shared libraries correctly.

Until the Open MPI developers fix their GNU autoconf templates, you will need to work around this issue by either using the old PGI-branded compiler names, or by adding -fPIC to your CFLAGS, FCFLAGS, CXXFLAGS, and LDFLAGS variables.

Adding the flags is how I worked around it. It never occurred to me that it was Open MPI setting things up differently due to the compiler ‘name’.