Nvfortran OpenACC performance regression: WENO5 reconstruction kernel ~1.7x slower since 24.7 (24.5 good, 24.7 first bad)

Accelerated Computing HPC Compilers nvc, nvc++ and nvfortran
, ,
1

Affected versions

Bisected release-by-release; the regression was introduced in 24.7. The full application (MFC, \3D_performance_test\ benchmark, grind time in ns/gp/eq/rhs, lower is better) and this MWE (kernel ms/call) flip at the same boundary. MWE numbers below are from a single sweep over all seven SDKs on an otherwise-idle A100-80GB (2026-07-15); the kernel’s register count from ptxas (`build_.log`) steps at the same release:

nvfortran status app grind time MWE ms/call vs 24.3 registers
24.3-0 good 0.653 4.677 1.00x 180
24.5-0 good 0.652 4.660 1.00x 184
24.7-0 bad 0.830 7.901 1.69x 196
24.9-0 bad 0.838 7.900 1.69x 196
25.1-0 bad 0.847 7.922 1.69x 196
25.11-0 bad 0.837 7.911 1.69x 196
26.3-0 bad 0.838 7.906 1.69x 196

Performance is flat within each regime (24.3–24.5 and 24.7–26.3): this is a single step change at 24.7, not a gradual drift, and nothing recovers through 26.3. The checksum is 2.047598784747795E+08 for every version, identical to all printed digits.

Hardware / software environment

  • GPU: NVIDIA A100 80GB PCIe (sm_80)
  • Driver: 590.48.01 (CUDA 13.1), host: x86-64 Linux (Icelake server)
  • Target flags: -fast -O3 -acc -gpu=cc80,keep,ptxinfo -Minfo=accel
  • The regression also reproduces with -gpu=cuda12.9 on 26.3 (7.94 ms/call, identical checksum — see reference/cuda12_9_build_26_3.log and reference/cuda12_9_run_26_3.log), so it is in the Fortran front/middle end, not in the bundled ptxas.

Description

MFC (GitHub - MFlowCode/MFC: Exascale multiphase flow solver — 2025 Gordon Bell Prize Finalist | 200T grid points on 43K+ GPUs · GitHub), a compressible-flow CFD solver written in Fortran + OpenACC, shows a 28% end-to-end slowdown going from NVHPC 24.3 to 25.11/26.3 on A100. Profiling isolated the loss to its WENO5 reconstruction kernel (m_weno_s_weno_1071_gpu and its y/z variants), which goes from 5.0 ms/call to 8.3 ms/call (+66%). The attached tarball contains a single-file extraction of that kernel that reproduces the regression standalone: the same loop nest (gang vector collapse(3) over the grid with a seq inner loop over 8 field variables), small private arrays, the mapped-WENO (WENO-M) weight formula, and compile-time-constant scheme parameters (MFC’s “case optimization” mode, which amplifies the regression from ~1.1x to ~1.7x). Comparing SASS, the newer compilers stop CSE-ing/hoisting loop-invariant subexpressions: 26.3 emits ~40% more instructions for the identical source, with nearly double the DFMA count and 3x the MOVs (same 60 LDG global loads, zero spills in both), and adds per-iteration recomputation plus extra branch convergence (BSSY/BSYNC) that the 24.3 code did not need. -Minfo=accel output is identical between versions (vector(128), same implicit reductions, same “Local memory used” line), so the schedule is unchanged — only the quality of the generated device code differs.

Reproduction

./run_comparison.sh

With no arguments this builds weno_mwe.f90 with 24.3 and 26.3 (each environment sourced in its own subshell), runs both binaries on the same GPU, and prints a table. If those SDKs are not present in ./hpc_sdk_24_3 / ./hpc_sdk_26_3 next to the script, they are downloaded and installed there automatically via nvhpcInstall.sh (looked for in this directory, then one level up).

The full sweep in the tables above was produced by passing all seven environments explicitly — any list of setupEnv.sh paths works and is built/run in the order given:

./run_comparison.sh <24.3-prefix>/setupEnv.sh <24.5-prefix>/setupEnv.sh \
                    <24.7-prefix>/setupEnv.sh <24.9-prefix>/setupEnv.sh \
                    <25.1-prefix>/setupEnv.sh <25.11-prefix>/setupEnv.sh \
                    <26.3-prefix>/setupEnv.sh

Additional SDKs for the sweep can be installed the same way the bootstrap does it, e.g. ./nvhpcInstall.sh -v 24.7 -p ./hpc_sdk_24_7, which writes the corresponding `setupEnv.sh` into the install prefix.

Compilation is exactly:

nvfortran -fast -O3 -acc -gpu=cc80,keep,ptxinfo -Minfo=accel weno_mwe.f90 -o weno_mwe\_<ver>

The binary warms up 10 iterations, then times 100 launches of the kernel (system_clock around an !$acc wait-bounded region) and prints ms/call plus a checksum (sum of |values| of both output arrays) so nothing is dead-code eliminated and cross-compiler agreement can be verified.

(Version labels in the output and in artifact names come from nvfortran --version of whichever compiler each environment provides, not from the paths passed on the command line.)

Expected vs actual (measured on the machine above, A100-80GB, idle GPU)

nvfortran ms/call checksum ratio vs 24.3
24.3-0 4.677 2.047598784747795E+08 1.00x
24.5-0 4.660 2.047598784747795E+08 1.00x
24.7-0 7.901 2.047598784747795E+08 1.69x
24.9-0 7.900 2.047598784747795E+08 1.69x
25.1-0 7.922 2.047598784747795E+08 1.69x
25.11-0 7.911 2.047598784747795E+08 1.69x
26.3-0 7.906 2.047598784747795E+08 1.69x

Expected: ms/call roughly equal across versions. Actual: every release from 24.7 onward is ~1.7x slower, in a single step at 24.7. Checksums agree to all printed digits, so all compilers generate numerically equivalent code — only performance regresses. Timing measurements taken with the GPU otherwise idle (kernel-only timing brackets an !$acc wait, so contention from other jobs inflates absolute numbers and compresses the ratio).

SASS / ptxas evidence

MWE kernel m_weno_mwe_s_weno_173_gpu (dumps in reference/sass_mwe_*.txt):

metric 24.3 26.3
SASS instructions 752 1056
DFMA 119 222
MOV 42 126
LDG 60 60
BSSY/BSYNC 16/16 27/27
registers 180 196
spills 0 0

Full-application kernel m_weno_s_weno_1071_gpu from MFC, for comparison (dumps and ptxas/-Minfo excerpts in reference/):

metric 24.3 26.3
SASS instructions 760 1064
DFMA 119 222
MOV 43 128
LDG 60 60
registers 186 198
spills 0 0

The MWE reproduces the full application’s instruction-count and opcode-mix divergence almost exactly, so it exercises the same code-generation defect.

Files

All files below are in the attached tarball:

  • weno_mwe.f90 — self-contained reproducer (no dependencies beyond OpenACC)
  • build.sh <setupEnv.sh> — builds with one compiler, logs -Minfo/ptxas to build_<ver>.log
  • run_comparison.sh — good-vs-bad comparison; bootstraps its own SDKs if needed
  • nvhpcInstall.sh — downloads/installs an NVHPC SDK and generates its `setupEnv.sh` (used by the bootstrap)
  • build_<ver>.log — compile logs for all seven tested versions (24.3, 24.5, 24.7, 24.9, 25.1, 25.11, 26.3)
  • reference/
    • sass_mwe_24_3.txt, sass_mwe_26_3.txt — cuobjdump SASS of the MWE kernel
    • sass_weno_243.txt, sass_weno_263.txt — cuobjdump SASS of the original MFC kernel
    • mfc_minfo_s_weno_243.txt, mfc_minfo_s_weno_263.txt — -Minfo=accel for the original subroutine
    • mfc_ptxas_m_weno_243.txt, mfc_ptxas_m_weno_263.txt — ptxas info for the original kernels

(The weno_mwe_<ver> binaries and keep\_<ver>/ intermediates are not shipped; a run regenerates them.)

weno_mwe_nvfortran_24_7_regression.tar.gz (84.7 KB)

Source provenance

Extracted from MFC src/simulation/m_weno.fpp, subroutine s_weno, weno_order == 5 / weno_dir == 1 branch, with Fypp macros expanded to the exact OpenACC directives MFC generates and the case-optimization constants (weno_polyn=2, weno_num_stencils=2, mapped_weno=.true., etc.) baked in as Fortran parameters, as in MFC’s optimized production builds. Grid: 200^3 interior cells + 4 ghost layers, 8 reconstructed variables, double precision, synthetic smooth input data.

Happy to run additional experiments (Nsight Compute profiles, other flags or versions) on request. A TPR number for tracking would be appreciated.

AI/LLM Disclosure: The code and descriptions in this post were generated with the assistance of Claude Code

2

Hi bwilfong3,

Thanks for the report and reproducer!

This is due to a known change for 24.7 where, from the release notes:

  • The HPC compilers will not perform reciprocal rewrites at optimization level -O3 or below; reciprocal rewrites are enabled with the -Mfprelaxed or -Ofast options.

This was done due to concerns over numerical accuracy.

For weno, the performance delta is all coming from the few lines like “alpha/sum(alpha)” no longer do the reciprocal writes by default.

To get the performance back, either add the flag “-Mfprelaxed” or “-Ofast” (which implies -Mfprelaxed). “-fast” is slightly different and does not include relaxed precision.

% nvfortran -fast -O3 -acc -V24.7 weno_mwe.f90 -o weno_mwe_247_fast
% nvfortran -Ofast -O3 -acc -V24.7 weno_mwe.f90 -o weno_mwe_247_Ofast
% weno_mwe_247_fast
ms_per_call =     6.2983
checksum    =   2.047598784747795E+08
% weno_mwe_247_Ofast
ms_per_call =     3.5510
checksum    =   2.047598784747795E+08

-Mat

3

This did the trick! Thanks Mat!