pgi compiler only works without -acc tag... help!

Accelerated Computing HPC Compilers Legacy PGI Compilers
1

Hey guys

I’m having a problem during the make stage. My code (about 1 hundred *.cc and *.h files in multiple folders) compiles and runs correctly with GCC/G++, MPICC/MPIC++, or PGCC/PGC++. This is without any “-acc” tags in the compiler command and without using any gpu/pragma code. There are 2 loops that are tagged with pragmas for Openmp.

However, when I add “-acc” to the PGCC/PGC++ compiler command (because I want to use my gpu), I get an error about 50% through the make process. The only thing that I’ve changed is adding “-acc” to the compiler command - I made no changes to the code. If I add both “-acc” to the compiler command and a “pragma acc kernels” on a code loop, I get the same error. So, I’m guessing the tags are not the problem.

This is the error:
/usr/bin/ld: cannot find /tmp/pgcudafat8ped0PgFc9t6.o: No such file or directory

Further, that error message is surrounded by about 20 of these errors:
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crti.o(.debug_info): relocation 0 has invalid symbol index 7
…
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crti.o(.debug_ranges): relocation 0 has invalid symbol index 4
…
/usr/lib/x86_64-linux-gnu/crti.o: In function _init': /build/eglibc-ripdx6/eglibc-2.19/csu/../sysdeps/x86_64/crti.S:64: multiple definition of _init’
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crti.o(.debug_info): relocation 0 has invalid symbol index 7
…
/usr/lib/x86_64-linux-gnu/crti.o:/build/eglibc-ripdx6/eglibc-2.19/csu/…/sysdeps/x86_64/crti.S:64: first defined here
/usr/lib/x86_64-linux-gnu/crti.o: In function _fini': (.fini+0x0): multiple definition of _fini’
/usr/lib/x86_64-linux-gnu/crti.o:(.fini+0x0): first defined here
/usr/bin/ld: cannot find /tmp/pgcudafatKKXhS0yCAEkG.o: No such file or directory
/usr/lib/gcc/x86_64-linux-gnu/4.8/crtbeginS.o:(.data.rel+0x0): multiple definition of __dso_handle' /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbeginS.o:(.data.rel+0x0): first defined here /usr/lib/gcc/x86_64-linux-gnu/4.8/crtendS.o:(.tm_clone_table+0x0): multiple definition of TMC_END’
/usr/lib/gcc/x86_64-linux-gnu/4.8/crtendS.o:(.tm_clone_table+0x0): first defined here


I find that these 20 errors go away if I remove any repeated library paths in my compiler command (I.e., having 2 entries of “-L/usr/lib”). Does the PGI compiler become sensitive to repeated library paths when “-acc” is added to the compiler command? It doesn’t seem to be bothered by repeats until “-acc” is used.

However, I do not know how to get rid of the “tmp/pgcuda…no such file or directory” error. What do you think?

_ABOUT MY SYSTEM:

  • Ubuntu 14.04
  • NVS 4200m GPU 512mb memory with NVIDIA 384 driver
  • Intel I7 processor
  • PGI Community 17.10
  • Compile command is “-acc -ta=tesla:cuda8.0,cc20,time”. I’ve tried it without the -ta part also (meaning only having the “-acc” part), but the same error happens.
  • Personal amount of C/C++/compiled languages knowledge? … minimal. I’m a Python/R guy.
  • My PGI and CUDA installations and environment appear to be ok because I was able to run the code in this tutorial: https://devblogs.nvidia.com/openacc-example-part-1/_


    Immediately before failing and producing the error message, the last commands shown in the terminal are a bunch of linking commands. Two of the commands are large. There are about 50 *.o files and paths in one command.

Again, all of this compiles fine in both GCC/G++, MPICC/MPIC++, and PGCC/PGC++. The problem happens when I add “-acc” to the compiler command.

Thanks for any help or direction you can give me![/url]

2

Hi krkr4250,

When you added “-acc”, did you also accidently remove the “-c” flag?

The “crti.o” object is only added during link so if you’re getting it during a compilation, it’s most likely due to “-c” missing. “-c” tells the compiler to only compile the file and not link. Though, this doesn’t explain the multiple definition error which would indicate that “ctri.o” is being added twice.

One thing that may help is if you can post the full compilation line and output for one of the files that is getting the error. Also add the “-v” (verbose) flag so we can see all the steps the compiler is doing during compilation.

-Mat

3

When you added “-acc”, did you also accidentally remove the “-c” flag?

The only thing I did was add or remove “-acc”. I’m using autogenerated makefiles. From what I can tell, the makefiles and configure files automatically add the “-c” flag. The only thing that I do is set the values below at the beginning before calling the script that generates the ~30 makefiles:

--without-python \
--without-hdf5 \
CC=/opt/pgi/linux86-64/17.10/bin/pgcc \
CXX=/opt/pgi/linux86-64/17.10/bin/pgc++ \
F77=/opt/pgi/linux86-64/17.10/bin/pgfortran \
CFLAGS="-g -fast -v -acc -ta=tesla:cuda8.0,cc20,time -I/usr/include -Minfo=accel" \
CXXFLAGS="-g -fast -v -acc -ta=tesla:cuda8.0,cc20,time -I/usr/include -Minfo=accel"

Like I said before, this works fine in the PGI compilers if “-acc” is not there.

Also add the “-v” (verbose) flag so we can see all the steps the compiler is doing during compilation.

As you requested, I added the “-v” flag. Now, instead of making it 50% through the make process, it only makes it to about 25%. Additionally, with “-v” included, now PGI won’t work even without the “-acc” flag. Here is the new error where the execution stops:

/usr/bin/as /tmp/pgc++Mz0bYxhfmtai.s -I./ -I../../../ -I/usr/include/ -I/opt/pgi/linux86-64/2017/cuda/8.0/include/ -I/usr/include/ -I/opt/pgi/linux86-64/2017/cuda/8.0/include/ -o tlibSGJC2.o
Action(ReadTIFile(./tlibSGJC2.ti))
Unlinking /tmp/pgc++2z0bI7hPBCaW.il
Unlinking /tmp/pgc++Mz0bYxhfmtai.s
Unlinking /tmp/pgc++wz0bcwczLvrO.ll
PGCC/x86 Linux 17.10-0: compilation successful

/usr/bin/as /tmp/pgc++Vy0bnOo69uo1.s -I./ -I../../../ -I/usr/include/ -I/opt/pgi/linux86-64/2017/cuda/8.0/include/ -I/usr/include/ -I/opt/pgi/linux86-64/2017/cuda/8.0/include/ -o tlibSGJC.o
Action(ReadTIFile(./tlibSGJC.ti))
Unlinking /tmp/pgc++3y0bLlfsseQR.il
Unlinking /tmp/pgc++Vy0bnOo69uo1.s
Unlinking /tmp/pgc++Ny0b1PJF8sp-.ll
PGC/x86-64 Linux 17.10-0: compilation completed with informational messages

/usr/bin/as /tmp/pgcckM1bEiA7gZs7.s -I./ -I../../../ -I/usr/include/ -I/opt/pgi/linux86-64/2017/cuda/8.0/include/ -I/usr/include/ -I/opt/pgi/linux86-64/2017/cuda/8.0/include/ -o hcubature.o
Unlinking /tmp/pgcckM1bEiA7gZs7.s
Unlinking /tmp/pgccAM1bomRd8oDm.ll
libtool: compile: mv -f "-MD.o" ".libs/hcubature.o"
mv: invalid option -- 'M'
Try 'mv --help' for more information.
rm: invalid option -- 'M'
Try 'rm --help' for more information.
make[4]: *** [hcubature.lo] Error 1

However, a few lines above this error, I see some words that look similar to the words in my initial error message (the “/tmp/pgcudafat… no such file or directory” error). I see multiple lines with “fatbinary” and “tmp” coming from the cuda folder. Is this related to my problem?

/opt/pgi/linux86-64/2017/cuda/8.0/bin/fatbinary --64 --create=/tmp/pgnvd21ZbIvgBW5KR.fat --ident=/tmp/pgcudafatKzZbS95tA5Tw.o --image=profile=sm_20,file=/tmp/pgcudaezZbmCg57utU.cubin

/opt/pgi/linux86-64/17.10/bin/pgimport /tmp/pgnvdM1ZbYb5gq-CZ.s /tmp/pgnvd21ZbIvgBW5KR.fat -var __PGI_CUDA_LOC -ccname __PGI_CUDA_CAP -cc20 -init __pgi_uacc_set_tatime
Export LD_LIBRARY_PATH=/opt/pgi/linux86-64/2017/cuda/8.0/nvvm/lib64:
Export DYLD_LIBRARY_PATH=/opt/pgi/linux86-64/2017/cuda/8.0/nvvm/lib:
Export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Unlinking /tmp/pgnvd2EZbILO4WHia.fat
Unlinking /tmp/pgnvdMEZbYrTqqN2m.s

/opt/pgi/linux86-64/2017/cuda/8.0/bin/fatbinary --64 --create=/tmp/pgnvd26ZbIfUrWXKo.fat --ident=/tmp/pgcudafatKGZbSz79AnYu.o --image=profile=sm_20,file=/tmp/pgcudaeGZbmy4e7iPr.cubin
/usr/bin/ld /usr/lib/x86_64-linux-gnu/crt1.o /usr/lib/x86_64-linux-gnu/crti.o /tmp/pgcudafatKoZbSbGlAfG9.o /opt/pgi/linux86-64/17.10/lib/trace_init.o /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbegin.o /opt/pgi/linux86-64/17.10/lib/initmp.o --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 /opt/pgi/linux86-64/17.10/lib/pgi.ld -L/opt/pgi/linux86-64/2017/cuda/8.0/lib64 -L/opt/pgi/linux86-64/17.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib64 -L/opt/pgi/linux86-64/17.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 tSMatrix.o ./.libs/libhmat.a ../libhrutil/.libs/libhrutil.a -lreadline -llapack -lblas -lpgf90rtl -lpgf90 -lpgf90_rpm1 -lpgf902 -lpgftnrtl -lpgmp -lnuma -lpthread -lnspgc -lpgc -lrt -lm -rpath /opt/pgi/linux86-64/17.10/lib -rpath /opt/pgi/linux86-64/2017/cuda/8.0/lib64 -rpath /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib64 -o tSMatrix -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib64 /opt/pgi/linux86-64/17.10/lib/acc_init_link_cuda.o -laccapi -laccg -laccn -laccg2 -ldl -lcudadevice -lpgatm -lstdc++ -latomic -lpgmp -lnuma -lpthread -lnspgc -lpgc -lm -lgcc -lc -lgcc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/4.8/crtend.o /usr/lib/x86_64-linux-gnu/crtn.o

/usr/bin/as -o /tmp/pgcudafatKzZbS95tA5Tw.o /tmp/pgnvdM1ZbYb5gq-CZ.s
Export PGI=/opt/pgi

/opt/pgi/linux86-64/17.10/bin/pgimport /tmp/pgnvdM6ZbYXEYq3YW.s /tmp/pgnvd26ZbIfUrWXKo.fat -var __PGI_CUDA_LOC -ccname __PGI_CUDA_CAP -cc20 -init __pgi_uacc_set_tatime
Unlinking /tmp/pgnvd21ZbIvgBW5KR.fat
Unlinking /tmp/pgnvdM1ZbYb5gq-CZ.s
/usr/bin/ld /usr/lib/x86_64-linux-gnu/crt1.o /usr/lib/x86_64-linux-gnu/crti.o /tmp/pgcudafatKzZbS95tA5Tw.o /opt/pgi/linux86-64/17.10/lib/trace_init.o /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbegin.o /opt/pgi/linux86-64/17.10/lib/initmp.o --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 /opt/pgi/linux86-64/17.10/lib/pgi.ld -L/opt/pgi/linux86-64/2017/cuda/8.0/lib64 -L/opt/pgi/linux86-64/17.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib64 -L/opt/pgi/linux86-64/17.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 tlibhmat2.o ./.libs/libhmat.a ../libhrutil/.libs/libhrutil.a -lreadline -llapack -lblas -lpgf90rtl -lpgf90 -lpgf90_rpm1 -lpgf902 -lpgftnrtl -lpgmp -lnuma -lpthread -lnspgc -lpgc -lrt -lm -rpath /opt/pgi/linux86-64/17.10/lib -rpath /opt/pgi/linux86-64/2017/cuda/8.0/lib64 -rpath /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib64 -o tlibhmat2 -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib64 /opt/pgi/linux86-64/17.10/lib/acc_init_link_cuda.o -laccapi -laccg -laccn -laccg2 -ldl -lcudadevice -lpgatm -lstdc++ -latomic -lpgmp -lnuma -lpthread -lnspgc -lpgc -lm -lgcc -lc -lgcc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/4.8/crtend.o /usr/lib/x86_64-linux-gnu/crtn.o

/opt/pgi/linux86-64/17.10/bin/pgacclnk -nvidia /opt/pgi/linux86-64/17.10/bin/pgnvd -cuda8.0 -computecap=20 -init=tatime -v /usr/bin/ld /usr/lib/x86_64-linux-gnu/crt1.o /usr/lib/x86_64-linux-gnu/crti.o /opt/pgi/linux86-64/17.10/lib/trace_init.o /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbegin.o /opt/pgi/linux86-64/17.10/lib/initmp.o --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 /opt/pgi/linux86-64/17.10/lib/pgi.ld -L/opt/pgi/linux86-64/2017/cuda/8.0/lib64 -L/opt/pgi/linux86-64/17.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib64 -L/opt/pgi/linux86-64/17.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 tTextIO.o ./.libs/libhmat.a ../libhrutil/.libs/libhrutil.a -lreadline -llapack -lblas -lpgf90rtl -lpgf90 -lpgf90_rpm1 -lpgf902 -lpgftnrtl -lpgmp -lnuma -lpthread -lnspgc -lpgc -lrt -lm -rpath /opt/pgi/linux86-64/17.10/lib -rpath /opt/pgi/linux86-64/2017/cuda/8.0/lib64 -rpath /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib64 -o tTextIO -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib64 /opt/pgi/linux86-64/17.10/lib/acc_init_link_cuda.o -laccapi -laccg -laccn -laccg2 -ldl -lcudadevice -lpgatm -lstdc++ -latomic -lpgmp -lnuma -lpthread -lnspgc -lpgc -lm -lgcc -lc -lgcc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/4.8/crtend.o /usr/lib/x86_64-linux-gnu/crtn.o



The “crti.o” object is only added during link so if you’re getting it during a compilation, it’s most likely due to “-c” missing. “-c” tells the compiler to only compile the file and not link. Though, this doesn’t explain the multiple definition error which would indicate that “ctri.o” is being added twice.

  1. Why is this not a problem in the other compilers? And why does it only become a problem in PGI when I use the “-acc” flag?

  2. Is there a way to get the compiler to automatically remove duplicate paths?

One thing that may help is if you can post the full compilation line and output for one of the files that is getting the error.

Beware. It’s long and it’s messy. If this is too much of a mess, is there a way I can send you the log file (in a private message or email)?

/bin/bash ../../../libtool  --tag=CXX   --mode=link /opt/pgi/linux86-64/17.10/bin/pgc++  -g -fast -O2 -acc -ta=tesla:cuda8.0,cc20,time -I/opt/pgi/linux86-64/2017/cuda/8.0/include -I/usr/include -I/usr/lib/openmpi/include -mp -Minfo=accel -mp  -version-info 0:0:0 -L/opt/pgi/linux86-64/2017/cuda/8.0/lib64 -L/opt/pgi/linux86-64/2017/mpi/openmpi-2.1.2/lib -L/usr/include -o libscuff.la -rpath /home/thisisme/xxx/lib RWGGeometry.lo RWGSurface.lo ReadComsolFile.lo ReadGMSHFile.lo InitEdgeList.lo FIBBICache.lo PBCSetup.lo GCMatrixElements.lo GTransformation.lo PointInObject.lo Visualize.lo AssembleBEMMatrix.lo SurfaceSurfaceInteractions.lo EdgeEdgeInteractions.lo PanelCubature.lo PanelPanelInteractions.lo TaylorDuffy.lo QDFIPPI.lo QIFIPPICubature.lo QIFIPPITaylorDuffy.lo QIFIPPITaylorDuffyV2P0.lo FIPPICache.lo GBarAccelerator.lo GBarVDEwald.lo Faddeeva.lo GetFields.lo GetNearFields.lo DSIPFT.lo EMTPFT.lo EPPFT.lo MomentPFT.lo OPFT.lo GetPFT.lo GetDipoleMoments.lo GetSphericalMoments.lo GetDyadicGFs.lo ExpandCurrentDistribution.lo AssembleRHSVector.lo AssessPanelPair.lo CalcGC.lo rwlock.lo ../libhrutil/libhrutil.la ../libhmat/libhmat.la ../libSGJC/libSGJC.la ../libIncField/libIncField.la ../libMDInterp/libMDInterp.la ../libMatProp/libMatProp.la ../libTriInt/libTriInt.la ../libSpherical/libSpherical.la ../libSubstrate/libSubstrate.la ../libSSSolver/libSSSolver.la  -lm -lreadline -llapack -lblas    -L/opt/pgi/linux86-64/2017/cuda/8.0/lib64 -L/opt/pgi/linux86-64/2017/mpi/openmpi-2.1.2/lib -L/usr/include -L/opt/pgi/linux86-64/17.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib64 -lpgf90rtl -lpgf90 -lpgf90_rpm1 -lpgf902 -lpgftnrtl -lpgmp -lnuma -lpthread -lnspgc -lpgc -lrt -lm
libtool: link: rm -fr  .libs/libscuff.lax .libs/libscuff.so.0.0.0

libtool: link: (cd .libs/libscuff.lax/libhrutil.a && ar x "/home/thisisme/Downloads/xxx/src/libs/libscuff/../libhrutil/.libs/libhrutil.a")

libtool: link: (cd .libs/libscuff.lax/libhmat.a && ar x "/home/thisisme/Downloads/xxx/src/libs/libscuff/../libhmat/.libs/libhmat.a")

libtool: link: (cd .libs/libscuff.lax/libSGJC.a && ar x "/home/thisisme/Downloads/xxx/src/libs/libscuff/../libSGJC/.libs/libSGJC.a")

libtool: link: (cd .libs/libscuff.lax/libIncField.a && ar x "/home/thisisme/Downloads/xxx/src/libs/libscuff/../libIncField/.libs/libIncField.a")

libtool: link: (cd .libs/libscuff.lax/libMDInterp.a && ar x "/home/thisisme/Downloads/xxx/src/libs/libscuff/../libMDInterp/.libs/libMDInterp.a")

libtool: link: (cd .libs/libscuff.lax/libMatProp.a && ar x "/home/thisisme/Downloads/xxx/src/libs/libscuff/../libMatProp/.libs/libMatProp.a")

libtool: link: (cd .libs/libscuff.lax/libTriInt.a && ar x "/home/thisisme/Downloads/xxx/src/libs/libscuff/../libTriInt/.libs/libTriInt.a")

libtool: link: (cd .libs/libscuff.lax/libSpherical.a && ar x "/home/thisisme/Downloads/xxx/src/libs/libscuff/../libSpherical/.libs/libSpherical.a")

libtool: link: (cd .libs/libscuff.lax/libSubstrate.a && ar x "/home/thisisme/Downloads/xxx/src/libs/libscuff/../libSubstrate/.libs/libSubstrate.a")

libtool: link: (cd .libs/libscuff.lax/libSSSolver.a && ar x "/home/thisisme/Downloads/xxx/src/libs/libscuff/../libSSSolver/.libs/libSSSolver.a")

libtool: link: /opt/pgi/linux86-64/17.10/bin/pgc++  -fPIC -DPIC -shared -nostdlib /usr/lib/x86_64-linux-gnu/crti.o /opt/pgi/linux86-64/17.10/lib/trace_init.o /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbeginS.o /opt/pgi/linux86-64/17.10/lib/initmp.o  .libs/RWGGeometry.o .libs/RWGSurface.o .libs/ReadComsolFile.o .libs/ReadGMSHFile.o .libs/InitEdgeList.o .libs/FIBBICache.o .libs/PBCSetup.o .libs/GCMatrixElements.o .libs/GTransformation.o .libs/PointInObject.o .libs/Visualize.o .libs/AssembleBEMMatrix.o .libs/SurfaceSurfaceInteractions.o .libs/EdgeEdgeInteractions.o .libs/PanelCubature.o .libs/PanelPanelInteractions.o .libs/TaylorDuffy.o .libs/QDFIPPI.o .libs/QIFIPPICubature.o .libs/QIFIPPITaylorDuffy.o .libs/QIFIPPITaylorDuffyV2P0.o .libs/FIPPICache.o .libs/GBarAccelerator.o .libs/GBarVDEwald.o .libs/Faddeeva.o .libs/GetFields.o .libs/GetNearFields.o .libs/DSIPFT.o .libs/EMTPFT.o .libs/EPPFT.o .libs/MomentPFT.o .libs/OPFT.o .libs/GetPFT.o .libs/GetDipoleMoments.o .libs/GetSphericalMoments.o .libs/GetDyadicGFs.o .libs/ExpandCurrentDistribution.o .libs/AssembleRHSVector.o .libs/AssessPanelPair.o .libs/CalcGC.o .libs/rwlock.o   .libs/libscuff.lax/libhrutil.a/ProcessArguments.o .libs/libscuff.lax/libhrutil.a/ProcessOptions.o .libs/libscuff.lax/libhrutil.a/Vector.o .libs/libscuff.lax/libhrutil.a/libhrutil.o  .libs/libscuff.lax/libhmat.a/C2ML.o .libs/libscuff.lax/libhmat.a/GetEntries.o .libs/libscuff.lax/libhmat.a/HDF5IO.o .libs/libscuff.lax/libhmat.a/HMatrix.o .libs/libscuff.lax/libhmat.a/HVector.o .libs/libscuff.lax/libhmat.a/LBWrappers.o .libs/libscuff.lax/libhmat.a/SMatrix.o .libs/libscuff.lax/libhmat.a/Sort.o .libs/libscuff.lax/libhmat.a/TextIO.o  .libs/libscuff.lax/libSGJC.a/hcubature.o .libs/libscuff.lax/libSGJC.a/libSGJC.o .libs/libscuff.lax/libSGJC.a/pcubature.o  .libs/libscuff.lax/libIncField.a/GaussianBeam.o .libs/libscuff.lax/libIncField.a/IncField.o .libs/libscuff.lax/libIncField.a/IncFieldList.o .libs/libscuff.lax/libIncField.a/PlaneWave.o .libs/libscuff.lax/libIncField.a/PointSource.o .libs/libscuff.lax/libIncField.a/SphericalWave.o  .libs/libscuff.lax/libMDInterp.a/BinSearch.o .libs/libscuff.lax/libMDInterp.a/Interp1D.o .libs/libscuff.lax/libMDInterp.a/Interp2D.o .libs/libscuff.lax/libMDInterp.a/Interp3D.o .libs/libscuff.lax/libMDInterp.a/Interp4D.o .libs/libscuff.lax/libMDInterp.a/freadEC.o  .libs/libscuff.lax/libMatProp.a/UserDefinedMaterials.o .libs/libscuff.lax/libMatProp.a/libMatProp.o .libs/libscuff.lax/libMatProp.a/matheval.o .libs/libscuff.lax/libMatProp.a/node.o .libs/libscuff.lax/libMatProp.a/parser.o .libs/libscuff.lax/libMatProp.a/scanner.o .libs/libscuff.lax/libMatProp.a/symbol_table.o .libs/libscuff.lax/libMatProp.a/xmalloc.o .libs/libscuff.lax/libMatProp.a/xmath.o  .libs/libscuff.lax/libTriInt.a/BZIntegration.o .libs/libscuff.lax/libTriInt.a/CCCubature.o .libs/libscuff.lax/libTriInt.a/CCRules.o .libs/libscuff.lax/libTriInt.a/IntegrateCliffFunction.o .libs/libscuff.lax/libTriInt.a/LatticeSum.o .libs/libscuff.lax/libTriInt.a/LebedevRules.o .libs/libscuff.lax/libTriInt.a/TCR.o .libs/libscuff.lax/libTriInt.a/libTriInt.o  .libs/libscuff.lax/libSpherical.a/AmosBessel.o .libs/libscuff.lax/libSpherical.a/DifferentialOperators.o .libs/libscuff.lax/libSpherical.a/TranslationMatrices.o .libs/libscuff.lax/libSpherical.a/VectorSphericalWaves.o .libs/libscuff.lax/libSpherical.a/drc3jm.o .libs/libscuff.lax/libSpherical.a/libSpherical.o .libs/libscuff.lax/libSpherical.a/machcon.o .libs/libscuff.lax/libSpherical.a/zbsubs.o  .libs/libscuff.lax/libSubstrate.a/FullWave.o .libs/libscuff.lax/libSubstrate.a/GTwiddle.o .libs/libscuff.lax/libSubstrate.a/Interpolation.o .libs/libscuff.lax/libSubstrate.a/SommerfeldIntegrand.o .libs/libscuff.lax/libSubstrate.a/SommerfeldIntegrator.o .libs/libscuff.lax/libSubstrate.a/Static.o .libs/libscuff.lax/libSubstrate.a/libSubstrate.o .libs/libscuff.lax/libSubstrate.a/qIntegrate.o  .libs/libscuff.lax/libSSSolver.a/Excitations.o .libs/libscuff.lax/libSSSolver.a/GetPPI.o .libs/libscuff.lax/libSSSolver.a/GetPhiE.o .libs/libscuff.lax/libSSSolver.a/PostProcessing.o .libs/libscuff.lax/libSSSolver.a/SSSolver.o   -L/opt/pgi/linux86-64/2017/cuda/8.0/lib64 -L/opt/pgi/linux86-64/2017/mpi/openmpi-2.1.2/lib -L/usr/include -L/opt/pgi/linux86-64/17.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib64 -lreadline -llapack -lblas -lpgf90rtl -lpgf90 -lpgf90_rpm1 -lpgf902 -lpgftnrtl -lnspgc -lrt -laccapimp -laccgmp -laccnmp -laccg2mp -ldl -lcudadevice -lpgatm -lstdc++ -latomic -lpgmp -lnuma -lpthread -lpgc -lm -lc -lgcc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/4.8/crtendS.o /usr/lib/x86_64-linux-gnu/crtn.o /opt/pgi/linux86-64/17.10/lib/trace_init.o /opt/pgi/linux86-64/17.10/lib/initmp.o /opt/pgi/linux86-64/17.10/lib/trace_init.o /opt/pgi/linux86-64/17.10/lib/initmp.o /usr/lib/x86_64-linux-gnu/crti.o /tmp/pgcudafatYLRewiBuCzXc.o /opt/pgi/linux86-64/17.10/lib/trace_init.o /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbeginS.o /opt/pgi/linux86-64/17.10/lib/initmp.o  -O2 -mp -mp   -mp -Wl,-soname -Wl,libscuff.so.0 -o .libs/libscuff.so.0.0.0
4

I did some more experimentation. I mentioned that it compiles and runs fine without the “-acc” flag. The “-fast” flag also causes problems, but “-O2” works fine.

I noticed these lines in the terminal output :

PGC-I-0222-Redundant definition for symbol __THROW (/usr/include/x86_64-linux-gnu/sys/cdefs.h: 74)
PGC-I-0222-Redundant definition for symbol __extension__ (/usr/include/x86_64-linux-gnu/sys/cdefs.h: 351)
PGC-W-0095-Type cast required for this conversion (pcubature.c: 423)
PGC-W-0095-Type cast required for this conversion (pcubature.c: 459)
PGC/x86-64 Linux 17.10-0: compilation completed with warnings

Below is the context where those lines came from. Are those errors or just warnings? The PGI compiler doesn’t seem to mind and keeps moving if I am not using the “-acc” or “-fast” flags. Otherwise, the execution stops with the below messages.

Making all in libSGJC
make[4]: Entering directory `/home/thisisme/Downloads/xxx/src/libs/libSGJC'
source='pcubature.c' object='pcubature.lo' libtool=yes \
	DEPDIR=.deps depmode=pgcc /bin/bash ../../../depcomp \
	/bin/bash ../../../libtool  --tag=CC   --mode=compile /opt/pgi/linux86-64/17.10/bin/pgcc -DHAVE_CONFIG_H -I. -I../../..     -g -fast -v -acc -ta=tesla:cuda8.0,cc20,time -I/opt/pgi/linux86-64/2017/cuda/8.0/include -I/usr/include -Minfo=accel -c -o pcubature.lo pcubature.c
libtool: compile:  /opt/pgi/linux86-64/17.10/bin/pgcc -DHAVE_CONFIG_H -I. -I../../.. -g -fast -v -acc -ta=tesla:cuda8.0,cc20,time -I/opt/pgi/linux86-64/2017/cuda/8.0/include -I/usr/include -Minfo=accel -c pcubature.c -MD 
Export PGI=/opt/pgi

/opt/pgi/linux86-64/17.10/bin/pgc pcubature.c -debug -x 120 0x200 -opt 2 -x 119 0xa10000 -x 122 0x40 -x 123 0x1000 -x 127 4 -x 127 17 -x 19 0x400000 -x 28 0x40000 -x 120 0x10000000 -x 70 0x8000 -x 122 1 -x 125 0x20000 -quad -vect 56 -y 34 16 -x 34 0x8 -x 32 6291456 -y 19 8 -y 35 0 -x 42 0x30 -x 39 0x40 -x 199 10 -x 39 0x80 -x 59 4 -tp sandybridge -astype 0 -x 121 1 -stdinc /opt/pgi/linux86-64/17.10/include-gcc48:/opt/pgi/linux86-64/17.10/include:/usr/lib/gcc/x86_64-linux-gnu/4.8/include:/usr/local/include:/usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed:/usr/include/x86_64-linux-gnu:/usr/include -def unix -def __unix -def __unix__ -def linux -def __linux -def __linux__ -def __NO_MATH_INLINES -def __LP64__ -def __x86_64 -def __x86_64__ -def __LONG_MAX__=9223372036854775807L -def '__SIZE_TYPE__=unsigned long int' -def '__PTRDIFF_TYPE__=long int' -def __THROW= -def __extension__= -def __amd_64__amd64__ -def __k8 -def __k8__ -def __SSE__ -def __MMX__ -def __SSE2__ -def __SSE3__ -def __SSSE3__ -def __STDC_HOSTED__ -predicate '#machine(x86_64) #lint(off) #system(posix) #cpu(x86_64)' -idir . -idir ../../.. -idir /opt/pgi/linux86-64/2017/cuda/8.0/include -idir /usr/include -idir /opt/pgi/linux86-64/2017/cuda/8.0/include -def _ACCEL=201003 -def _OPENACC=201510 -def __CUDA_API_VERSION=8000 -def PGI_TESLA_TARGET -def HAVE_CONFIG_H -cmdline '+pgcc pcubature.c -DHAVE_CONFIG_H -I. -I../../.. -g -fast -Mvect=sse -Mcache_align -Mflushz -Mpre -v -acc -ta=tesla:cuda8.0,cc20,time -I/opt/pgi/linux86-64/2017/cuda/8.0/include -I/usr/include -Minfo=accel -c -MD' -x 123 0x80000000 -x 123 4 -preinclude _c_macros.h -x 119 0x20 -def __pgnu_vsn=40804 -autoinl 10 -x 168 400 -x 174 128000 -x 14 0x200000 -x 46 4 -x 123 8 -x 120 0x200000 -x 70 0x40000000 -x 164 0x800000 -accel tesla -x 186 0x80000 -x 180 0x4000400 -x 121 0xc00 -x 194 0x40000 -x 180 0x4000400 -x 121 0xc00 -x 163 0x1 -x 186 0x80000 -cudaver 8.0 -x 194 0x40000 -x 176 0x100 -cudacap 20 -x 189 0x8000 -y 163 0xc0000000 -x 192 0x40000000 -x 189 0x10 -y 189 0x4000000 -x 9 1 -x 42 0x14200000 -x 72 0x1 -x 136 0x11 -quad -x 119 0x10000000 -x 129 0x40000000 -x 129 2 -x 164 0x1000 -x 0 0x1000000 -x 2 0x100000 -x 0 0x2000000 -x 161 16384 -x 162 16384 -asm /tmp/pgccflFpqaS0ECe.s
PGC-I-0222-Redundant definition for symbol __THROW (/usr/include/x86_64-linux-gnu/sys/cdefs.h: 74)
PGC-I-0222-Redundant definition for symbol __extension__ (/usr/include/x86_64-linux-gnu/sys/cdefs.h: 351)
[b]PGC-W-0095-Type cast required for this conversion (pcubature.c: 423)
PGC-W-0095-Type cast required for this conversion (pcubature.c: 459)
PGC/x86-64 Linux 17.10-0: compilation completed with warnings
[/b]
/usr/bin/as /tmp/pgccflFpqaS0ECe.s -I./ -I../../../ -I/opt/pgi/linux86-64/2017/cuda/8.0/include/ -I/usr/include/ -I/opt/pgi/linux86-64/2017/cuda/8.0/include/ -o pcubature.o
Unlinking /tmp/pgccflFpqaS0ECe.s
Unlinking /tmp/pgccTlFhsIIgy6k.ll
libtool: compile: mv -f "-MD.o" ".libs/pcubature.o"
mv: invalid option -- 'M'
Try 'mv --help' for more information.
rm: invalid option -- 'M'
Try 'rm --help' for more information.
make[4]: *** [pcubature.lo] Error 1
make[4]: Leaving directory `/home/thisisme

Could it be that there is a typecast problem in the “pcubature” file? If so, why is it only a problem when “-acc” or “-fast” is used?

Those lines in the terminal output are complaining about typecasting in pcubature.c:423 and pcubature.c:453. Here is that section of the code (starting at line 407):

#define DEFAULT_MAX_NBUF (1U << 20)

int pcubature_v_log(unsigned fdim, integrand_v f, void *fdata,
                    unsigned dim, const double *xmin, const double *xmax,
                    size_t maxEval, double reqAbsError, double reqRelError,
                    error_norm norm,
                    double *val, double *err,
                    const char *LogFileName)
{
     int ret;
     size_t nbuf = 0;
     unsigned m[MAXDIM];
     double *buf = NULL;
     memset(m, 0, sizeof(unsigned) * dim);
     ret = pcubature_v_buf(fdim, f, fdata, dim, xmin, xmax,
				  maxEval, reqAbsError, reqRelError, norm,
				  m, &buf, &nbuf, DEFAULT_MAX_NBUF, val, err, LogFileName);
     free(buf);
     return ret;
}

int pcubature_v(unsigned fdim, integrand_v f, void *fdata,
                unsigned dim, const double *xmin, const double *xmax,
                size_t maxEval, double reqAbsError, double reqRelError,
                error_norm norm,
                double *val, double *err)
{
  return pcubature_v_log(fdim, f, fdata, dim, xmin, xmax,
                         maxEval, reqAbsError, reqRelError,
                         norm, val, err, 0);
}

#include "vwrapper.h"

int pcubature_log(unsigned fdim, integrand f, void *fdata,
	          unsigned dim, const double *xmin, const double *xmax,
	          size_t maxEval, double reqAbsError, double reqRelError,
	          error_norm norm, double *val, double *err, 
                  const char *LogFileName)
{
     int ret;
     size_t nbuf = 0;
     unsigned m[MAXDIM];
     double *buf = NULL;
     fv_data d;

     d.f = f; d.fdata = fdata;
     memset(m, 0, sizeof(unsigned) * dim);
     ret = pcubature_v_buf(
	  fdim, fv, &d, dim, xmin, xmax, 
	  maxEval, reqAbsError, reqRelError, norm,
	  m, &buf, &nbuf, 16 /* max_nbuf > 0 to amortize function overhead */,
	  val, err, LogFileName);
     free(buf);
     return ret;
}

int pcubature(unsigned fdim, integrand f, void *fdata,
	      unsigned dim, const double *xmin, const double *xmax,
	      size_t maxEval, double reqAbsError, double reqRelError,
	      error_norm norm, double *val, double *err)
{
 return pcubature_log(fdim, f, fdata, dim, xmin, xmax,
	              maxEval, reqAbsError, reqRelError,
	              norm, val, err, 0);
}
5

Hi krkr4250,

I high doubt the compiler warnings have anything to do with this. You might want to look at the code later to see if adding a explicit cast fixes those, but a warning wouldn’t cause the behavior.

It’s also very odd that using anything but “-O2” causes problems. Especially “-v” which doesn’t effect compilation at all, and only has the compiler driver print out the different utilities it’s running.

Did you also add “-ta=tesla:cuda8.0,cc20,time” or does libtool do that when “-acc” is defined? “cc20” targets the old Fermi cards. It’s not a problem for the compiler but is an old flag that I don’t see very often any more.

It also begs the question of what else libtool is doing?

For example there’s this line:

libtool: compile: /opt/pgi/linux86-64/17.10/bin/pgcc -DHAVE_CONFIG_H -I. -I…/…/… -g -fast -v -acc -ta=tesla:cuda8.0,cc20,time -I/opt/pgi/linux86-64/2017/cuda/8.0/include -I/usr/include -Minfo=accel -c pcubature.c -MD

Why is “-MD” being adding? This flag tells the compiler to create a makefile dependency file for the source.

However, it doesn’t look like “-MD” is actually passed to the compiler but instead libtool thinks it’s an object and tries to move it to the “.libs” directory which in turn causes an error:

libtool: compile: mv -f “-MD.o” “.libs/pcubature.o”
mv: invalid option – ‘M’
Try ‘mv --help’ for more information.
rm: invalid option – ‘M’
Try ‘rm --help’ for more information.
make[4]: *** [pcubature.lo] Error 1
make[4]: Leaving directory `/home/thisisme

It seems like the problems may be due to libtool rather than the compiler. Maybe your libtool version needs to be updated?

Is this source available that I can download it?

-Mat

6

I high doubt the compiler warnings have anything to do with this. You might want to look at the code later to see if adding a explicit cast fixes those, but a warning wouldn’t cause the behavior.

Yes, I added some casts and those warnings stopped.

Did you also add “-ta=tesla:cuda8.0,cc20,time” or does libtool do that when “-acc” is defined?

I tried several things. With and without “-acc”, with and without “-ta=…”, and with and without all of the permutations within and between. The only time compiling failed was when “-acc” or “-fast” was present.

“cc20” targets the old Fermi cards. It’s not a problem for the compiler but is an old flag that I don’t see very often any more.

Hah! It’s a ~5 year old laptop that I got for almost free. It’s old, but 48 cores is still useful.

It also begs the question of what else libtool is doing? … However, it doesn’t look like “-MD” is actually passed to the compiler but instead libtool thinks it’s an object and tries to move it to the “.libs” directory which in turn causes an error…

Right, I too noticed that “-MD” looked like a mistake. My libtool is GNU libtool 2.4.2. I searched the Makefile for “-MD”, and below is what I found:

.cc.o:
#	$(AM_V_CXX)$(CXXCOMPILE) -MT $@ -MD -MP -MF $(DEPDIR)/$*.Tpo -c -o $@ $<
#	$(AM_V_at)$(am__mv) $(DEPDIR)/$*.Tpo $(DEPDIR)/$*.Po
	$(AM_V_CXX)source='$<' object='$@' libtool=no \
	DEPDIR=$(DEPDIR) $(CXXDEPMODE) $(depcomp) \
	$(AM_V_CXX_no)$(CXXCOMPILE) -c -o $@ $<

.cc.obj:
#	$(AM_V_CXX)$(CXXCOMPILE) -MT $@ -MD -MP -MF $(DEPDIR)/$*.Tpo -c -o $@ `$(CYGPATH_W) '$<'`
#	$(AM_V_at)$(am__mv) $(DEPDIR)/$*.Tpo $(DEPDIR)/$*.Po
	$(AM_V_CXX)source='$<' object='$@' libtool=no \
	DEPDIR=$(DEPDIR) $(CXXDEPMODE) $(depcomp) \
	$(AM_V_CXX_no)$(CXXCOMPILE) -c -o $@ `$(CYGPATH_W) '$<'`

.cc.lo:
#	$(AM_V_CXX)$(LTCXXCOMPILE) -MT $@ -MD -MP -MF $(DEPDIR)/$*.Tpo -c -o $@ $<
#	$(AM_V_at)$(am__mv) $(DEPDIR)/$*.Tpo $(DEPDIR)/$*.Plo
	$(AM_V_CXX)source='$<' object='$@' libtool=yes \
	DEPDIR=$(DEPDIR) $(CXXDEPMODE) $(depcomp) \
	$(AM_V_CXX_no)$(LTCXXCOMPILE) -c -o $@ $<

I did make some progress (maybe) though. I copied and pasted the commands that were failing. I removed the “-MD” from the end and they compiled. But there are a lot of files to do this for, and I don’t know if simply leaving “-MD” off the end is equal to what libtool was trying to accomplish. But it at least compiled all of the files after I copied and pasted the commands for each file (about 20 of them). I probably should just make a bash script that goes through the list of files. But again, I don’t know if it really is as simple as leaving off the “-MD”.

After sneaking past all of the “-MD” errors, I ended up running into a new error. Now it says ~" can’t relocate this file to 3xx4343; try recompiling with -pfic"

^I’ll reproduce the error later so I can give you the exact wording of the error.

For now, I’m going to recompile the entire thing with -pfic like it suggested and again sneak past the “-MD” errors. Hopefully, that will work.

Is this source available that I can download it?

Yeah. Can I send you the link in a forum message or an email? I don’t want to post it here so publicly.

7

An update…

Ok. I got past the errors caused by putting “-v” in. For whatever reason, when using “-v”, it truncated the commands after the “-MD” flag and treated “-MD” as “-MD.o”.

When I don’t use “-v”, the command looks like this after “-MD”:

sudo /opt/pgi/linux86-64/17.10/bin/pgcc -DHAVE_CONFIG_H -I. -I../../.. -g -O2 -acc -v -Minfo -I/usr/include -c pcubature.c -MD  -fpic -DPIC -o .libs/pcubature.o

When I use “-v”, the elements on the end get cut off when using “-MD”:

sudo /opt/pgi/linux86-64/17.10/bin/pgcc -DHAVE_CONFIG_H -I. -I../../.. -g -O2 -acc -v -Minfo -I/usr/include -c pcubature.c -MD

To work around this, I manually entered the correct untruncated line for all of the files. Now, I am able to reach the original error again. Here it is:

/usr/bin/ld: cannot find /tmp/pgcudafatMLKhY6YriW4H.o: No such file or directory

From what I can tell, “pgcudafat” files are produced by the compiler program named “fat binary” or “nvlink”:

/opt/pgi/linux86-64/2017/cuda/7.5/bin/nvlink --arch=sm_50 -m64 -L/opt/pgi/linux86-64/17.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib64 -L/opt/pgi/linux86-64/17.10/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib64 /opt/pgi/linux86-64/17.10/lib/trace_init.o /opt/pgi/linux86-64/17.10/lib/initmp.o tcmatheval.o ./.libs/libcmatheval.a -lreadline -llapack -lblas -lpgf90rtl -lpgf90 -lpgf90_rpm1 -lpgf902 -lpgftnrtl -lpgmp -lnuma -lpthread -lnspgc -lpgc -lrt -lm /opt/pgi/linux86-64/17.10/lib/acc_init_link_cuda.o -laccapi -laccg -laccn -laccg2 -ldl -lcudadevice -lpgatm -lstdc++ -latomic -lpgmp -lnuma -lpthread -lnspgc -lpgc -lm -lgcc -lc -lgcc -lgcc_s -o /tmp/pgcudah76bv0R0GCtJ.cubin
/opt/pgi/linux86-64/17.10/bin/pgnvd -fatobj /tmp/pgcudafat-76b9wQ5PepC.o -o /tmp/pgcudafat-76b9wQ5PepC.o -cuda7.5 -v -sm 30 /tmp/pgcudax76bfo6KqQ_p.cubin -sm 35 /tmp/pgcudap76bT_Rty8tM.cubin -sm 50 /tmp/pgcudah76bv0R0GCtJ.cubin
Export LD_LIBRARY_PATH=/opt/pgi/linux86-64/2017/cuda/7.5/nvvm/lib64:
Export DYLD_LIBRARY_PATH=/opt/pgi/linux86-64/2017/cuda/7.5/nvvm/lib:
Export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin

/opt/pgi/linux86-64/2017/cuda/7.5/bin/fatbinary --64 --create=/tmp/pgnvdSc7beBfSvEqX.fat --key=x_dlink --ident=/tmp/pgcudafat-76b9wQ5PepC.o --image=profile=sm_30,file=/tmp/pgcudax76bfo6KqQ_p.cubin --image=profile=sm_35,file=/tmp/pgcudap76bT_Rty8tM.cubin --image=profile=sm_50,file=/tmp/pgcudah76bv0R0GCtJ.cubin

/opt/pgi/linux86-64/17.10/bin/pgimport /tmp/pgnvdmc7bKZWYkssv.s /tmp/pgnvdSc7beBfSvEqX.fat -var __PGI_CUDA_LOC -ccname __PGI_CUDA_CAP -cc30 -cc35 -cc50

I see lots of files starting with “/tmp/pgcudafat…”, but I don’t see any that are “/tmp/pgcudafatMLKhY6YriW4H.o”. Any ideas on how I can find where that file is supposed to be created from? How does the compiler know to look for that file or that it is supposed to exist?