CUDA Fortran Change with 11.3?

I’m trying to get a model running on a new GPU machine. My old machine which has Teslas and PGI 11.2 can compile this code with no problem. However, on the new machine which has Fermis and PGI 11.3, when it hits one part of the code it spits out an ICE:

/gpfsm/dnb31/tdirs/login/borggpu01.5175.mathomp4/pgcudaforatPcaUl_s8IA.h(116): error: incomplete type is not allowed

/gpfsm/dnb31/tdirs/login/borggpu01.5175.mathomp4/pgcudaforatPcaUl_s8IA.h(135): error: incomplete type is not allowed

2 errors detected in the compilation of "/gpfsm/dnb31/tdirs/login/borggpu01.5175.mathomp4/pgnvdpuPcTI1RXd4U.nv0".
PGF90-F-0000-Internal compiler error. pgnvd job exited with nonzero status code       0 (GEOS_SolarGridComp.F90: 4474)
PGF90/x86-64 Linux 11.3-0: compilation aborted
gmake[6]: *** [GEOS_SolarGridComp.o] Error 2

However, if I change the GNUmakefile in that part of the model to add -V11.1 (which is also installed on this machine), the code compiles just fine. Note that the file which has this issue isn’t the main computational kernel, but rather the part of the code that calls the GPU kernel so it’s main CUDA Fortran excitement is a combination of allocates, deallocates, copies and a call to the kernel.

So, I guess I have two questions. One: was there a change between 11.1 and 11.3 that might cause this and can I disable/correct for it? (The acc1rc files do seem to have differences, but it’s a bit esoteric for me to decode.) Two: can I mix -V11.1 and -V11.3 code? (That is, can I compile just this portion with 11.1 and then the rest with 11.3 and link with 11.3?)

FYI: My CUDA compile target is -Mcuda=nofma,maxregcount:64,fastmath,ptxinfo,3.2


Hi Matt,

Can you please send a reproducing example to PGI Customer Service? Most likely we fixed something in 11.3 that then caused your code to break. Sigh.


An example has been sent, but don’t thank me yet. You’re getting a bit of a tarball because of all the needed includes to compile it. But, hopefully you’ll be able to figure it out.

I have also confirmed with the example that 11.2 is also good (albeit on a different machine), so the error is due to some change between 11.2 and 11.3.

Addendum. I just added in more GPU code and there are three other sections of my code encountering the “error: incomplete type is not allowed” ICE. So, it doesn’t seem specific to this code, but perhaps something in my code style could be triggering it? (I do a bit of cutting, pasting, and templating.)