OSU MPI micro-benchmarks fail to compile under HPC_SDK

This is all done on the Pleiades system.

wget http://mvapich.cse.ohio-state.edu/download/mvapich/osu-micro-benchmarks-5.6.3.tar.gz
qsub -I -V -l select=1:ncpus=9:ngpus=1:model=sky_gpu:mem=30GB -l place=scatter:shared -l walltime=02:00:00 -q v100@pbspl4
module purge
module load nvhpc/20.7
./configure CC=mpicc CXX=mpicxx --enable-cuda --with-cuda=/nasa/nvidia/hpc_sdk/Linux_x86_64/20.7/cuda/11.0 --prefix=/u/user/play/OSU/osu-micro-benchmarks-5.6.3/install/nvhpc20.7
mpicxx -DPACKAGE_NAME=“OSU-Micro-Benchmarks” -DPACKAGE_TARNAME=“osu-micro-benchmarks” -DPACKAGE_VERSION=“5.6.3” -DPACKAGE_STRING=“OSU-Micro-Benchmarks\ 5.6.3” -DPACKAGE_BUGREPORT=“mvapich-discuss@cse.ohio-state.edu” -DPACKAGE_URL="" -DPACKAGE=“osu-micro-benchmarks” -DVERSION=“5.6.3” -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=".libs/" -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_SYS_TIME_H=1 -DHAVE_UNISTD_H=1 -DHAVE_GETPAGESIZE=1 -DHAVE_GETTIMEOFDAY=1 -DHAVE_MEMSET=1 -DHAVE_SQRT=1 -DHAVE_CUDA_H=1 -D_ENABLE_CUDA_=1 -D_ENABLE_CUDA_KERNEL_=1 -DFIELD_WIDTH=18 -DFLOAT_PRECISION=2 -I. -I/nasa/nvidia/hpc_sdk/Linux_x86_64/20.7/cuda/11.0/include -g -O2 -c -o …/…/util/kernel.o …/…/util/kernel.cpp
line 8: error: invalid redeclaration of type name “__va_list_tag”
(declared at line 8)
struct __va_list_tag {

line 28: error: invalid redeclaration of type name “__pgi_va_list”
(declared at line 28)
typedef __va_list_tag __pgi_va_list[1];

2 errors detected in the compilation of “…/…/util/kernel.cpp”.
Makefile:557: recipe for target ‘…/…/util/kernel.o’ failed

Hi Daniel,

I took a look and it seems that “kernel.cpp” is actually a post-processed file generated by nvcc:

nvcc -cuda -maxrregcount 32 -ccbin mpicxx   -I/proj/nv/Linux_x86_64/20.7/cuda/11.0/include  --output-file ../../util/kernel.cpp.ii ../../util/ke  rnel.cu
mv ../../util/kernel.cpp.ii ../../util/kernel.cpp

Hence, the standard C++ header file definitions have already be included in the source. So when nvc++ includes these header files again, which it does by default, you get the redefinition errors.

Adding the flag “–no_preincludes” to the compilation will work around the issue.

Hope this helps,

Apparently someone already knew about this trick and put it into the OSU code.
find ./ -type f -name Makefile.am -exec grep -nH prei {} ;
./mpi/collective/Makefile.am:19:AM_CXXFLAGS = --nvcchost --no_preincludes
./mpi/one-sided/Makefile.am:23:AM_CXXFLAGS = --nvcchost --no_preincludes
./mpi/pt2pt/Makefile.am:20:AM_CXXFLAGS = --nvcchost --no_preincludes

Not sure why my first attempt didn’t pick this up, but a fresh configure and make worked fine without any code modifications.