nvfortran 22.X (or so) used to have -Mipa, which in my case was used with OpenACC. This would enable LTO, then later tell you that LTO is not supported for your device. This made performance a bit of a challenge and required some hacky workarounds.
With nvhpc 23.11, I see LTO enabled at the start of compilation and don’t see a warning later on telling me it could not LTO. However, I did not find anything in the nvhpc release notes discussing LTO, which would be an important feature addition.
My question: Does LTO, in particular for the case of inlining functions and subroutines across module files, “work”? If not, is it expected at some point?
We’d like to bring LTO back at some point, but unfortunately we can’t give you a timeline on when that would occur.
Though for cross-file inlining you can try the two-pass method. First create an inline extract library but compiling all files with “-Mextract=lib:libname” replacing “libname” with what you’d like to call it. Then compile with “-Minline=lib:libname” to use the extract library. Inlining is performed prior to the device code generation.
Hi Mat! I am trying to implement your suggestion but the compiler remarks
NVFORTRAN-W-0268-Cannot inline subprogram; common block mismatch
at seemingly every callsite. I don’t see common blocks anywhere in our codebase, so I’m not sure why we get this warning. I’m relatively new to Fortran and GPU programming, so I’m wondering if the OpenACC directives we use are generating them.
Is this a reasonable guess or does this message seem strange?
EDIT: I read the logs a bit closer and it seems only a few modules are failing to inline, some of which don’t have OpenACC kernels. I still find it odd that it’s complaining about common blocks, unless namelists are also common blocks.
No, inlining is applied before OpenACC so the directives shouldn’t affect them.
As for the message, under-the-hood module data are common blocks, so it’s likely has something to do with your modules. Though it’s been awhile since I’ve seen this messages so forgotten off hand the specifics of what triggers it and the person I’d normally ask is out on vacation right now.
Do you have a reproducing example I could look at?
I unfortunately only have a branch on my fork of MFC: https://github.com/AiredaleDev/MFC/tree/two_stage_ipo that produces the warning. I know that the compiler is able to inline some subroutines across module boundaries (notably s_compute_speed_of_sound) using -Mextract and -Minline but it still complains about many subroutines it tries and fails to inline because of the common block mismatch.
I also noticed that we expose some global variables in each program’s m_global_parameters.fpp, but these aren’t explicitly in a common block. Could this be related?
Lastly, I also changed the CMake to support GH-200 Unified Memory to address another issue, and I noticed that when I build MFC using it (pass --unified to mfc.sh build) the compiler will eventually fail to build with: NVFORTRAN-S-1066-The -cuda flag should be used with CUDA DEVICE variable - ..inline
I tried passing this flag to -cuda in CMake but it didn’t fix the issue. Do you know what could possibly prompt this?
We are facing the same problem with inlining across modules.
Cannot inline subprogram; common block mismatch
We are using openacc to generate GPU kernels. The performance dropped by >20% after I moved subroutines from the ‘integrator’ module to the ‘physics’ module where they actually belong. I suspect it is due to the inability to inline them in the kernel.
I used the two-pass compilation you recommended. I guess the issue is the different module scope that the subroutines see. Shouldn’t it help to declare the subroutines as pure? (Tried that, still not inlining).
Is there any workaround? Loosing this much performance is really painful.
Unfortunately I don’t have a good understanding on why the module data (
i.e. common block) is being seen as a mismatch, so don’t have a work around for you. If you happen to have a reproducing example you can share, that would be appreciated. I can try again to reproduce it with some of my codes, but thus far haven’t had luck doing so.
Once I can reproduce the problem, I can then attempt to determine the root cause and offer suggestions. Also, if compiler improvements are needed, I can file reports with engineering.