nvlink error : Multiple definitions on compile

Accelerated Computing HPC Compilers Legacy PGI Compilers
1

All,

I have a nice chunk of CUDA Fortran/accelerator pragma code that worked with PGI 12.8. With 13 (up to 13.10), it didn’t work, dying with:

call to cuStreamSynchronize returned error 700: Launch failed
call to cuStreamSynchronize returned error 700: Launch failed
call to cuStreamSynchronize returned error 700: Launch failed
call to cuStreamSynchronize returned error 700: Launch failed

So, when 14.1 came out, I tried it and, yep, the same failure. So, I had PGI 14.2 installed and now, well, I can’t even get it to compile. Instead, upon a link I get:

/ford1/share/gmao_SIteam/MPI/openmpi-1.7.4-pgi-14.2.0/bin/mpif90 -o StandAlone_FV3_Dycore.x StandAlone_FV3_Dycore.o AppGridCreate.o LatLon2Cube.o Cube2LatL
on.o libFVdycoreCubed_GridComp.a -L/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pgfortran_14.2-openmpi_1.7.4/Linux/lib -L
/home/mathomp4/Models/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14/GEOSagcm/Linux/lib  -pgcpplibs -tp=nehalem-64             /home/mathomp4/Model
s/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14/GEOSagcm/Linux/lib/libfvdycore.a \
                                   /home/mathomp4/Models/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14/GEOSagcm/Linux/lib/libGEOS_Shared.a /home/m
athomp4/Models/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14/GEOSagcm/Linux/lib/libMAPL_Base.a /home/mathomp4/Models/Ganymed-4_0_UNSTABLE-RRTMGPU-
withPGIedits-GPUFV3-PGI14/GEOSagcm/Linux/lib/libGMAO_hermes.a /home/mathomp4/Models/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14/GEOSagcm/Linux/l
ib/libMAPL_cfio_r4.a /home/mathomp4/Models/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14/GEOSagcm/Linux/lib/libGMAO_gfio_r4.a /home/mathomp4/Model
s/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14/GEOSagcm/Linux/lib/libGMAO_mpeu.a libFVdycoreCubed_GridComp.a /home/mathomp4/Models/Ganymed-4_0_UN
STABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14/GEOSagcm/Linux/lib/libfvdycore.a /ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pg
fortran_14.2-openmpi_1.7.4/Linux/lib/libesmf.a \
                                   /home/mathomp4/Models/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14/GEOSagcm/Linux/lib/libGFDL_fms.a   \
                                   -L/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pgfortran_14.2-openmpi_1.7.4/Linux/lib 
-lnetcdff -lnetcdf -L/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pgfortran_14.2-openmpi_1.7.4/Linux/lib -L/ford1/share/g
mao_SIteam/Baselibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pgfortran_14.2-openmpi_1.7.4/Linux/lib -lnetcdf -lmfhdf -ldf -lhdf5_hl -lhdf5 -lm -L/ford1
/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pgfortran_14.2-openmpi_1.7.4/Linux/lib -lmfhdf -ldf -lsz -ljpeg -L/ford1/share/gma
o_SIteam/Baselibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pgfortran_14.2-openmpi_1.7.4/Linux/lib -lcurl -lz -lrt -lm -lm -L/ford1/share/gmao_SIteam/Ba
selibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pgfortran_14.2-openmpi_1.7.4/Linux/lib -lcurl -lz -lrt -lm -L/ford1/share/gmao_SIteam/MPI/openmpi-1.7.4
-pgi-14.2.0/lib -I/ford1/share/gmao_SIteam/MPI/openmpi-1.7.4-pgi-14.2.0/lib -Wl,-rpath -Wl,/ford1/share/gmao_SIteam/MPI/openmpi-1.7.4-pgi-14.2.0/lib -L/for
d1/share/gmao_SIteam/MPI/openmpi-1.7.4-pgi-14.2.0/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lopen-rte -lopen-pal -lm -ldl -lrt -lnsl -
lutil  \
                                   -ldl -lstd -lrt -lC -Mcuda=nofma,ptxinfo,5.0,cc20 -acc -ta=nvidia:nofma,5.0,cc20 -Minfo=accel,par,ccff
nvlink error   : Multiple definitions of 'a2b_ord2_dev4_491_gpu'
nvlink error   : Multiple definitions of 'a2b_ord2_dev4_479_gpu'
nvlink error   : Multiple definitions of 'a2b_ord2_dev4_470_gpu'
nvlink error   : Multiple definitions of 'a2b_ord2_dev4_461_gpu'
nvlink error   : Multiple definitions of 'a2b_ord2_dev4_452_gpu'
nvlink error   : Multiple definitions of 'a2b_ord2_dev4_443_gpu'
nvlink error   : Multiple definitions of 'a2b_ord2_dev4_438_gpu'
nvlink error   : Multiple definitions of 'a2b_ord2_dev4_433_gpu'
nvlink error   : Multiple definitions of 'a2b_ord2_dev4_428_gpu'
nvlink error   : Multiple definitions of 'a2b_ord2_dev4_419_gpu'
nvlink error   : Multiple definitions of 'a2b_ord2_dev8_386_gpu'
nvlink error   : Multiple definitions of 'a2b_ord2_dev8_306_gpu'
nvlink error   : Multiple definitions of 'tp_core_cuda_getedgefluxes_'
nvlink error   : Multiple definitions of 'tp_core_cuda_filldm_'
...
nvlink error   : Multiple definitions of 'dyn_core_499_gpu'
nvlink error   : Multiple definitions of 'dyn_core_498_gpu'
nvlink error   : Multiple definitions of '$ADDRESS$_dyn_core_mod_16'
nvlink error   : Multiple definitions of '_dyn_core_mod_16'
nvlink error   : Multiple definitions of 'tracer_2d_1l_234_gpu'
nvlink error   : Multiple definitions of 'tracer_2d_1l_216_gpu'
nvlink error   : Multiple definitions of 'tracer_2d_1l_201_gpu'
nvlink error   : Multiple definitions of 'tracer_2d_1l_186_gpu'
nvlink error   : Multiple definitions of 'tracer_2d_1l_165_gpu'
nvlink error   : Multiple definitions of 'tracer_2d_1l_152_gpu'
nvlink error   : Multiple definitions of 'tracer_2d_1l_145_gpu'
nvlink error   : Multiple definitions of 'tracer_2d_1l_139_gpu'
nvlink error   : Multiple definitions of 'tracer_2d_1l_133_gpu'
nvlink error   : Multiple definitions of 'tracer_2d_1l_102_gpu'
nvlink error   : Multiple definitions of 'tracer_2d_1l_92_gpu'
pgacclnk: child process exit status 2: /opt/pgi/linux86-64/14.2/bin/pgnvd
gmake[6]: *** [StandAlone_FV3_Dycore.x] Error 2

where I’ve edited out quite a few here.

Now, as I said, this code compiled just fine with PGI 14.1/Open MPI 1.7.3 (well, it compiled, since it didn’t run, it probably didn’t actually compile correctly). This is using PGI 14.2/Open MPI 1.7.4, but I’m fairly certain this isn’t an Open MPI issue.

Has anyone seen this?

I will also say that with PGI 14.2, every so often I get ‘crt1.reg.c’ being left around by the compiler. It’s not part of my code, so I’m thinking maybe when the code dies, the compiler/linker doesn’t correctly clean it up?

Thanks,
Matt

2

Answering my own question, the error means exactly what you’d expect: multiple definitions!

A careful look at how I was linking the .x reveals that the library I made from all the code, libfvdycore.a, was indeed being linked twice. I guess PGI 14.2 has become stricter than 14.1.

However, I now get a new error:

(380) $ /ford1/share/gmao_SIteam/MPI/openmpi-1.7.4-pgi-14.2.0/bin/mpif90 -o StandAlone_FV3_Dycore.x StandAlone_FV3_Dycore.o AppGridCreate.o LatLon2Cube.o Cube2LatLon.o libFVdycoreCubed_GridComp.a -L/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pgfortran_14.2-openmpi_1.7.4/Linux/lib -L/home/mathomp4/Models/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14/GEOSagcm/Linux/lib -pgcpplibs -tp=nehalem-64 /home/mathomp4/Models/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14/GEOSagcm/Linux/lib/libfvdycore.a /home/mathomp4/Models/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14/GEOSagcm/Linux/lib/libGEOS_Shared.a /home/mathomp4/Models/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14/GEOSagcm/Linux/lib/libMAPL_Base.a /home/mathomp4/Models/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14/GEOSagcm/Linux/lib/libGMAO_hermes.a /home/mathomp4/Models/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14/GEOSagcm/Linux/lib/libMAPL_cfio_r4.a /home/mathomp4/Models/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14/GEOSagcm/Linux/lib/libGMAO_gfio_r4.a /home/mathomp4/Models/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14/GEOSagcm/Linux/lib/libGMAO_mpeu.a /ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pgfortran_14.2-openmpi_1.7.4/Linux/lib/libesmf.a /home/mathomp4/Models/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14/GEOSagcm/Linux/lib/libGFDL_fms.a -L/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pgfortran_14.2-openmpi_1.7.4/Linux/lib -lnetcdff -lnetcdf -L/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pgfortran_14.2-openmpi_1.7.4/Linux/lib -L/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pgfortran_14.2-openmpi_1.7.4/Linux/lib -lnetcdf -lmfhdf -ldf -lhdf5_hl -lhdf5 -lm -L/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pgfortran_14.2-openmpi_1.7.4/Linux/lib -lmfhdf -ldf -lsz -ljpeg -L/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pgfortran_14.2-openmpi_1.7.4/Linux/lib -lcurl -lz -lrt -lm -lm -L/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pgfortran_14.2-openmpi_1.7.4/Linux/lib -lcurl -lz -lrt -lm -L/ford1/share/gmao_SIteam/MPI/openmpi-1.7.4-pgi-14.2.0/lib -I/ford1/share/gmao_SIteam/MPI/openmpi-1.7.4-pgi-14.2.0/lib -Wl,-rpath -Wl,/ford1/share/gmao_SIteam/MPI/openmpi-1.7.4-pgi-14.2.0/lib -L/ford1/share/gmao_SIteam/MPI/openmpi-1.7.4-pgi-14.2.0/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lopen-rte -lopen-pal -lm -ldl -lrt -lnsl -lutil -ldl -lstd -lrt -lC -Mcuda=nofma,ptxinfo,5.0,cc20 -acc -ta=nvidia:nofma,5.0,cc20 -Minfo=accel,par,ccff 
In file included from /opt/pgi/linux86-64/14.2/include_acc/linkstub50.c:10:
./crt1.reg.c:7: error: redefinition of ‘__cudaRegisterLinkedBinary_home_mathomp4_Models_Ganymed_4_0_UNSTABLE_RRTMGPU_withPGIedits_GPUFV3_PGI14_GEOSagcm_Linux_lib_libfvdycore_a_84c82fc1’
./crt1.reg.c:1: note: previous definition of ‘__cudaRegisterLinkedBinary_home_mathomp4_Models_Ganymed_4_0_UNSTABLE_RRTMGPU_withPGIedits_GPUFV3_PGI14_GEOSagcm_Linux_lib_libfvdycore_a_84c82fc1’ was here
./crt1.reg.c:8: error: redefinition of ‘__cudaRegisterLinkedBinary_home_mathomp4_Models_Ganymed_4_0_UNSTABLE_RRTMGPU_withPGIedits_GPUFV3_PGI14_GEOSagcm_Linux_lib_libfvdycore_a_84c82fc1’
./crt1.reg.c:1: note: previous definition of ‘__cudaRegisterLinkedBinary_home_mathomp4_Models_Ganymed_4_0_UNSTABLE_RRTMGPU_withPGIedits_GPUFV3_PGI14_GEOSagcm_Linux_lib_libfvdycore_a_84c82fc1’ was here
./crt1.reg.c:10: error: redefinition of ‘__cudaRegisterLinkedBinary_home_mathomp4_Models_Ganymed_4_0_UNSTABLE_RRTMGPU_withPGIedits_GPUFV3_PGI14_GEOSagcm_src_GEOSgcs_GridComp_GEOSgcm_GridComp_GEOSagcm_GridComp_GEOSsuperdyn_GridComp_FVdycoreCubed_GridComp_libFVdycoreCubed_GridComp_a_84c82fc1’
./crt1.reg.c:9: note: previous definition of ‘__cudaRegisterLinkedBinary_home_mathomp4_Models_Ganymed_4_0_UNSTABLE_RRTMGPU_withPGIedits_GPUFV3_PGI14_GEOSagcm_src_GEOSgcs_GridComp_GEOSgcm_GridComp_GEOSagcm_GridComp_GEOSsuperdyn_GridComp_FVdycoreCubed_GridComp_libFVdycoreCubed_GridComp_a_84c82fc1’ was here
pgacclnk: child process exit status 2: /opt/pgi/linux86-64/14.2/bin/pgnvd

Any ideas about this?

3

More information:

I’m now getting 14.2 issues in places without GPU code when I make in parallel (aka make -j4). For example, I’m getting errors like this:

/opt/pgi/linux86-64/14.2/include_acc/linkstub50.c:10:17: error: crt1.reg.c: No such file or directory
pgacclnk: child process exit status 2: /opt/pgi/linux86-64/14.2/bin/pgnvd
/opt/pgi/linux86-64/14.2/include_acc/linkstub50.c:10:17: error: crt1.reg.c: No such file or directory
pgacclnk: child process exit status 2: /opt/pgi/linux86-64/14.2/bin/pgnvd

Something’s gone a bit pear-shaped…

4

Hi Matt,

The “crt1.ref.c” file contains the CUDA startup registration calls and gets generated at link time. My best guess is that you must be performing multiple concurrent links and each link is clobbering the other’s file. Though, can you confirm that multiple links are occurring? Does the problem go away if you only have one link?

Either way, our file naming here is obvious not unique enough. Hence, we’ve add TPR #19969 to address this and expect to have it into 14.4.

Thanks,
Mat

5

Okay, I just did an experiment. First, I removed the model directory I was trying to build and then re-checked it out from CVS. I then removed all the /tmp/pgcuda* files I had (and, yowza, there are a lot).

Then I did a simple, basic, non-parallel ‘make install’ so I shouldn’t be having any issues with that (though, as I said, 14.1 can do it just fine) in case I was linking a couple files at the same time. After all this I get the same error when trying to link StandAlone_FV3_Dycore.x:

/ford1/share/gmao_SIteam/MPI/openmpi-1.7.4-pgi-14.2.0/bin/mpif90 -o StandAlone_FV3_Dycore.x StandAlone_FV3_Dycore.o AppGridCreate.o LatLon2Cube.o Cube2LatL
on.o libFVdycoreCubed_GridComp.a -L/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pgfortran_14.2-openmpi_1.7.4/Linux/lib -L
/home/mathomp4/Models/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14-2014Mar06-PGI142/GEOSagcm/Linux/lib  -pgcpplibs -tp=nehalem-64             /ho
me/mathomp4/Models/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14-2014Mar06-PGI142/GEOSagcm/Linux/lib/libfvdycore.a \
                                   /home/mathomp4/Models/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14-2014Mar06-PGI142/GEOSagcm/Linux/lib/libGEOS
_Shared.a /home/mathomp4/Models/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14-2014Mar06-PGI142/GEOSagcm/Linux/lib/libMAPL_Base.a /home/mathomp4/Mo
dels/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14-2014Mar06-PGI142/GEOSagcm/Linux/lib/libGMAO_hermes.a /home/mathomp4/Models/Ganymed-4_0_UNSTABLE
-RRTMGPU-withPGIedits-GPUFV3-PGI14-2014Mar06-PGI142/GEOSagcm/Linux/lib/libMAPL_cfio_r4.a /home/mathomp4/Models/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GP
UFV3-PGI14-2014Mar06-PGI142/GEOSagcm/Linux/lib/libGMAO_gfio_r4.a /home/mathomp4/Models/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14-2014Mar06-PGI
142/GEOSagcm/Linux/lib/libGMAO_mpeu.a /ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pgfortran_14.2-openmpi_1.7.4/Linux/lib
/libesmf.a \
                                   /home/mathomp4/Models/Ganymed-4_0_UNSTABLE-RRTMGPU-withPGIedits-GPUFV3-PGI14-2014Mar06-PGI142/GEOSagcm/Linux/lib/libGFDL
_fms.a   \
                                   -L/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pgfortran_14.2-openmpi_1.7.4/Linux/lib 
-lnetcdff -lnetcdf -L/ford1/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pgfortran_14.2-openmpi_1.7.4/Linux/lib -L/ford1/share/g
mao_SIteam/Baselibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pgfortran_14.2-openmpi_1.7.4/Linux/lib -lnetcdf -lmfhdf -ldf -lhdf5_hl -lhdf5 -lm -L/ford1
/share/gmao_SIteam/Baselibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pgfortran_14.2-openmpi_1.7.4/Linux/lib -lmfhdf -ldf -lsz -ljpeg -L/ford1/share/gma
o_SIteam/Baselibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pgfortran_14.2-openmpi_1.7.4/Linux/lib -lcurl -lz -lrt -lm -lm -L/ford1/share/gmao_SIteam/Ba
selibs/GMAO-Baselibs-4_0_4/x86_64-unknown-linux-gnu/pgfortran_14.2-openmpi_1.7.4/Linux/lib -lcurl -lz -lrt -lm -L/ford1/share/gmao_SIteam/MPI/openmpi-1.7.4
-pgi-14.2.0/lib -I/ford1/share/gmao_SIteam/MPI/openmpi-1.7.4-pgi-14.2.0/lib -Wl,-rpath -Wl,/ford1/share/gmao_SIteam/MPI/openmpi-1.7.4-pgi-14.2.0/lib -L/for
d1/share/gmao_SIteam/MPI/openmpi-1.7.4-pgi-14.2.0/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lopen-rte -lopen-pal -lm -ldl -lrt -lnsl -
lutil  \
                                   -ldl -lstd -lrt -lC -Mcuda=nofma,ptxinfo,5.0,cc20 -acc -ta=nvidia:nofma,5.0,cc20 -Minfo=accel,par,ccff
In file included from /opt/pgi/linux86-64/14.2/include_acc/linkstub50.c:10:
./crt1.reg.c:7: error: redefinition of ‘__cudaRegisterLinkedBinary_home_mathomp4_Models_Ganymed_4_0_UNSTABLE_RRTMGPU_withPGIedits_GPUFV3_PGI14_2014Mar06_PG
I142_GEOSagcm_Linux_lib_libfvdycore_a_84c82fc1’
./crt1.reg.c:1: note: previous definition of ‘__cudaRegisterLinkedBinary_home_mathomp4_Models_Ganymed_4_0_UNSTABLE_RRTMGPU_withPGIedits_GPUFV3_PGI14_2014Ma
r06_PGI142_GEOSagcm_Linux_lib_libfvdycore_a_84c82fc1’ was here
./crt1.reg.c:8: error: redefinition of ‘__cudaRegisterLinkedBinary_home_mathomp4_Models_Ganymed_4_0_UNSTABLE_RRTMGPU_withPGIedits_GPUFV3_PGI14_2014Mar06_PG
I142_GEOSagcm_Linux_lib_libfvdycore_a_84c82fc1’
./crt1.reg.c:1: note: previous definition of ‘__cudaRegisterLinkedBinary_home_mathomp4_Models_Ganymed_4_0_UNSTABLE_RRTMGPU_withPGIedits_GPUFV3_PGI14_2014Ma
r06_PGI142_GEOSagcm_Linux_lib_libfvdycore_a_84c82fc1’ was here
In file included from /opt/pgi/linux86-64/14.2/include_acc/linkstub50.c:10:
./crt1.reg.c:10: error: redefinition of ‘__cudaRegisterLinkedBinary_home_mathomp4_Models_Ganymed_4_0_UNSTABLE_RRTMGPU_withPGIedits_GPUFV3_PGI14_2014Mar06_P
GI142_GEOSagcm_src_GEOSgcs_GridComp_GEOSgcm_GridComp_GEOSagcm_GridComp_GEOSsuperdyn_GridComp_FVdycoreCubed_GridComp_libFVdycoreCubed_GridComp_a_84c82fc1’
./crt1.reg.c:9: note: previous definition of ‘__cudaRegisterLinkedBinary_home_mathomp4_Models_Ganymed_4_0_UNSTABLE_RRTMGPU_withPGIedits_GPUFV3_PGI14_2014Ma
r06_PGI142_GEOSagcm_src_GEOSgcs_GridComp_GEOSgcm_GridComp_GEOSagcm_GridComp_GEOSsuperdyn_GridComp_FVdycoreCubed_GridComp_libFVdycoreCubed_GridComp_a_84c82f
c1’ was here
pgacclnk: child process exit status 2: /opt/pgi/linux86-64/14.2/bin/pgnvd
gmake[6]: *** [StandAlone_FV3_Dycore.x] Error 2

Are there multiple links going on here? As I said, this is serial, but perhaps PGI is doing more underneath than I know?