NVHPC 26.1 fort2 TERMINATED by signal 11

mikostul · February 14, 2026, 8:19am

Hello,

I’m converting the MAS code (PredSci) from OpenACC to OpenMP target offload and I’m hitting an internal compiler crash in nvfortran.

Upstream repository

MAS original:GitHub - predsci/MAS: Magnetohydrodynamic Algorithm outside a Sphere (MAS)

My fork / OpenMP port

GitHub - mstulajter/MAS at openmp

How to reproduce

./build.sh conf/nvidia_gpu_psi.conf

Compile flags (from conf/nvidia_gpu_psi.conf )

-O3 -march=native -mp=gpu -stdpar=gpu -gpu=ccnative,mem:separate -Minfo=accel

Environment

NVHPC 26.1
Linux x86_64
nvfortran path:/opt/nvidia/hpc_sdk/Linux_x86_64/26.1/compilers/bin/tools/fort2

Failure

Compilation fails while building mas.o with:

nvfortran-Fatal-/opt/nvidia/hpc_sdk/Linux_x86_64/26.1/compilers/bin/tools/fort2 TERMINATED by signal 11
make: *** [Makefile:25: mas.o] Error 2

The -Minfo=accel output shows GPU code generation proceeding normally prior to the crash.

Questions

What is the recommended way to isolate which construct or routine is triggering a fort2 SIGSEGV?
Is mixing -mp=gpu and -stdpar=gpu an issues in larger codes?
Does -gpu=mem:separate interact in any problematic way with OpenMP offload?
Is this likely a compiler bug in 26.1?

scamp1 · February 15, 2026, 10:58pm

So we have a development compiler that I think dumps more information than the release compiler, which doesn’t necessarily help you directly solve these kind of things, but I took the time to play with this and try to understand what was going on.

First, I found that I had to increase my stacksize to get the mas_cpp.f90 to not crash out immediately. Then, depending on where in your code my compilation crashed, I was able to determine the subroutine or function that the compiler found problematic.

Problems I found:

function cgdot - the compiler is really struggling with this function. It ends up in an infinite compile that can eat all the memory in the system. I have reported this as issue #38252 to engineering because it shouldn’t behave like that. But luckily, that’s not how I would recommend approaching that problem anyway. I think you should rewrite that approach using the omp loop construct. It’s not as widely supported by other compilers, but it’s vastly more efficient and essentially easier to map in understanding to OpenACC constructs, which - if you’re doing conversions, may make the coding more straight forward. If I hear back from engineering on the problem, I’ll let you know.

subroutines heating and advparticles - these choke out because procedural calls are not allowed in do concurrent structures yet, so you likely need to reapproach how you handle these calls and are parallelizing the approach. I don’t have any good advice here, but the approach you’re currently taking doesn’t work for these using do_concurrent directly.

So emptying those functions and subroutines inside the file, along with increasing the compilation stacksize, I was able to get the mas_cpp.f90 file to compile. That tells me that if you sort out your parallelization approaches with those, you should be able to resolve all your issues.

Now - a more broad suggestion. I think you would have a much better time with this code if you were to invest the time needed to break it into smaller modules and pieces. A 70,000 line code is a lot and it can limit what the compiler can do in some ways. I think a lot of the issues you see here is that the compiler doesn’t assume EVERYTHING will be in the same file, and that causes problems with correct error messages and such.

In the past we’ve seen major speedups in mega-monolithic codes like this from breaking them into smaller more manageable pieces. It would likely benefit the CPU only version of the code, as well as the OpenACC versions, I suspect. It would also make the compiler errors much more obvious and easier to handle - compiling the heating and advparticles subroutines as standalones caused the compiler to immediately dump a more useful error message, for example:

[scamp]$ mpif90 heating.F90 -mp=gpu -stdpar=gpu -c -o test.o -O3 -gpu=mem:separate
NVFORTRAN-W-1288-Use of the USE_DEVICE_PTR clause is deprecated, use USE_DEVICE_ADDR clause instead (heating.F90: 77)
  0 inform,   1 warnings,   0 severes, 0 fatal for heating
NVFORTRAN-S-1074-Procedure call in Do Concurrent is not supported yet (heating.F90: 117)
NVFORTRAN-S-1074-Procedure call in Do Concurrent is not supported yet (heating.F90: 290)
  0 inform,   0 warnings,   2 severes, 0 fatal for heating
[scamp]$

With regards to your questions:

Normally I carve away code and recompile - if the issue is still there, I carve away more. If the issue is gone on recompile, I put that piece of code back, and carve in a different place. It allows you to end up with a really concentrated reproducer at the end. However, working with a 70,000 line code that has potentially multiple issues presents a problem with that, right? So for you, I’d work on breaking it up into smaller pieces and seeing if I could more finely identify the compilation errors that way, and then chase finer reproducers.
Not that I know of. They should play well together. If you run into any edge cases where they don’t, please let us know!
I don’t believe so - I think it works as expected.
This is a mixed bag - I don’t think the compiler is supposed to emit unhelpful error messages to the user. So, by definition, that’s a compiler bug. However, since we diagnosed all the issues - it’s really a combination of throwing too much at the compiler at one time causing stuff to be muddied, along with a real compiler issue, and some not-allowed parallelization approaches.

Thank you for your question, and I hope this helps! Please let us know if you run into any other issues!

Cheers,

Seth.

mikostul · February 20, 2026, 8:51am

I was able to reduce this to a very small reproducer that looks like a compiler/device mapping issue around referencing a module allocatable array directly inside a do concurrent within an OpenMP target region when calling a declare target function.

module fields
  integer, parameter :: n = 10
  real, dimension(:), allocatable :: r, h
contains
!$omp declare target (pval)
  pure function pval(x)
    real, intent(in) :: x
    real :: pval
    pval = x
  end function
end module

subroutine heating
  use fields
  implicit none
  integer :: i

!$omp target enter data map(to: r) map(to: h)
!$omp target map(tofrom: h)
  do concurrent (i = 1 : n)
    h(i) = pval(r(i))
  enddo
!$omp end target

!$omp target exit data map(from: h) map(delete: r)
end subroutine heating

program small_reproducer
  use fields
  implicit none

  allocate(r(n), h(n))
  r = 1.0
  h = 0.
  call heating
  deallocate(h, r)
end program small_reproducer

When I compile with
mpif90 -stdpar=gpu -gpu=mem:separate -mp=gpu -O3 -o small_reproducer small_reproducer .F90
I get the following error:

NVFORTRAN-S-0155-Compiler failed to translate accelerator region (see -Minfo messages): Unknown variable reference (…)
…
NVFORTRAN-F-0704-Compilation aborted due to previous errors.

If I compile with -gpu=mem:unified, it compiles fine.

If I introduce a local scalar temp and pass that instead, it compiles cleanly with -gpu=mem: separate :

module fields
  integer, parameter :: n = 10
  real, dimension(:), allocatable :: r, h
contains
!$omp declare target (pval)
  pure function pval(x)
    real, intent(in) :: x
    real :: pval
    pval = x
  end function
end module

subroutine heating
  use fields
  implicit none
  integer :: i
  real :: rtmp

!$omp target enter data map(to: r) map(to: h)
!$omp target map(tofrom: h)
  do concurrent (i = 1 : n)
    rtmp = r(i)
    h(i) = pval(rtmp)
  enddo
!$omp end target

!$omp target exit data map(from: h) map(delete: r)
end subroutine heating

program small_reproducer
  use fields
  implicit none

  allocate(r(n), h(n))
  r = 1.0
  h = 0.
  call heating
  deallocate(h, r)
end program small_reproducer

Observations:

The failure only occurs with -gpu=mem:separate. Compiling the exact same source with -gpu=mem:unified works without modification.
The issue is triggered specifically by passing an array element expression (r(i)) directly as an actual argument to a declare target procedure inside a do concurrent within an OpenMP target region.
Introducing a local scalar temporary (rtmp = r(i)) and passing that instead makes the code compile cleanly under -gpu=mem:separate.
The function itself is trivial and pure, so the failure appears to be related to device mapping / lowering of the array element reference rather than anything semantic in the Fortran code.
The diagnostic (Unknown variable reference) suggests this may be a compiler issue.

scamp1 · February 20, 2026, 4:45pm

Hi - when I compile your first program in your latest response above, I get:

scamp$ nvfortran test.F90 -stdpar=gpu -gpu=mem:separate -mp=gpu -O3 -c
NVFORTRAN-S-0186-Argument missing for formal argument x (test.F90: 21)
  0 inform,   0 warnings,   1 severes, 0 fatal for heating

Maybe some syntax or something was changed as you tried to copy it in? It definitely looks like pval should take a real argument. However, if I give it a real argument (0.0, for example), I don’t get the crash you’re seeing.

If you can identify where your copy over went wrong, I can get the weird behavior reported to engineering. : )

mikostul · February 20, 2026, 5:27pm

Hello,

I fixed the first example. The line should have been as below:

scamp1 · February 20, 2026, 6:04pm

So I see compiler crashes in both cases - unified and separate - however, I don’t think either one should fail, so I’m going to open an issue for this with engineering. It is tagged as TPR#38274.

The difference in behavior between yours and mine could come down to simply having different compilers or cuda versions, I’m not sure. When I hear back from engineering, I’ll let you know!

And also, again, great job at cutting out simple reproducers from that huge code! Your hard work will benefit the compiler and everyone that uses it!

mikostul · February 26, 2026, 9:25am

I also wanted to add another reproducer. Not sure if this is the same issue or related. I compile all codes with:

mpif90 -stdpar=gpu -O3 -o small_reproducer small_reproducer.F90 -gpu=mem:separate -mp=gpu  -Minfo=accel

Reproducer 1: do concurrent … reduce inside omp target fails

module stubs
  use iso_fortran_env
  integer, parameter :: r_typ = REAL64
contains
  pure subroutine f1(n, x)
    integer, intent(in) :: n
    real(r_typ), dimension(n), intent(in) :: x
    real(r_typ) :: a
    a = 0._r_typ
  end subroutine f1
  pure subroutine f2(x, y)
    real(r_typ), intent(in) :: x, y
    real(r_typ) :: p
    if (x.eq.0._r_typ) then
      if (y.ge.0._r_typ) then
        p = 2
      else
        p = -2
      end if
    end if
  end subroutine f2
end module

subroutine run
  use iso_fortran_env
  use stubs
  implicit none
  integer :: i, m, n
  real(r_typ) :: a, b
  real(r_typ), allocatable :: arr(:)
  real(r_typ) :: out

  m = 4
  n = 1
  out = 0._r_typ
  allocate(arr(m))
  arr(1) = 1._r_typ

!$omp target enter data map(to: arr)
!$omp target map(tofrom: out)
  do concurrent (i = 1: n) reduce(+:out)
    call f1(m, arr)
    a = 1._r_typ
    b = 1._r_typ
    call f2(a, b)
    out = out + a + b
  enddo
!$omp end target

  write(*,'(a,es12.4)') 'out (from device) = ', out
  deallocate(arr)
end subroutine run

program main
  call run
end program main

Error

NVFORTRAN-S-0155-Compiler failed to translate accelerator region (see -Minfo messages): Unknown variable reference (small_reproducer.F90: 41)
run:
41, Generating NVIDIA GPU code
41, Loop parallelized across CUDA thread blocks, CUDA threads(32) blockidx%x threadidx%x
Generating reduction(+:out)
NVFORTRAN-F-0704-Compilation aborted due to previous errors. (small_reproducer.F90)
NVFORTRAN/x86-64 Linux 26.1-0: compilation aborted

Workarounds I found:

Add an extra “use” of the mapped array inside the region (e.g., create a dummy tmp and touch arr), then it compiles:

module stubs
  use iso_fortran_env
  integer, parameter :: r_typ = REAL64
contains
  pure subroutine f1(n, x)
    integer, intent(in) :: n
    real(r_typ), dimension(n), intent(in) :: x
    real(r_typ) :: a
    a = 0._r_typ
  end subroutine f1
  pure subroutine f2(x, y)
    real(r_typ), intent(in) :: x, y
    real(r_typ) :: p
    if (x.eq.0._r_typ) then
      if (y.ge.0._r_typ) then
        p = 2
      else
        p = -2
      end if
    end if
  end subroutine f2
end module

subroutine run
  use iso_fortran_env
  use stubs
  implicit none
  integer :: i, m, n
  real(r_typ) :: a, b
  real(r_typ), allocatable :: arr(:), tmp(:)
  real(r_typ) :: out

  m = 4
  n = 1
  out = 0._r_typ
  allocate(arr(m))
  allocate(tmp(m))
  arr(1) = 1._r_typ

!$omp target enter data map(to: arr)
!$omp target map(tofrom: out)
  do concurrent (i = 1: n) reduce(+:out)
    tmp = arr(m)
    call f1(m, arr)
    a = 1._r_typ
    b = 1._r_typ
    call f2(a, b)
    out = out + a + b
  enddo
!$omp end target

  write(*,'(a,es12.4)') 'out (from device) = ', out
  deallocate(tmp, arr)
end subroutine run

program main
  call run
end program main

Or remove the tofrom map, then it compiles.

module stubs
  use iso_fortran_env
  integer, parameter :: r_typ = REAL64
contains
  pure subroutine f1(n, x)
    integer, intent(in) :: n
    real(r_typ), dimension(n), intent(in) :: x
    real(r_typ) :: a
    a = 0._r_typ
  end subroutine f1
  pure subroutine f2(x, y)
    real(r_typ), intent(in) :: x, y
    real(r_typ) :: p
    if (x.eq.0._r_typ) then
      if (y.ge.0._r_typ) then
        p = 2
      else
        p = -2
      end if
    end if
  end subroutine f2
end module

subroutine run
  use iso_fortran_env
  use stubs
  implicit none
  integer :: i, m, n
  real(r_typ) :: a, b
  real(r_typ), allocatable :: arr(:)

  m = 4
  n = 1
  allocate(arr(m))
  arr(1) = 1._r_typ

!$omp target enter data map(to: arr)
  do concurrent (i = 1: n)
    call f1(m, arr)
    a = 1._r_typ
    b = 1._r_typ
    call f2(a, b)
  enddo

  deallocate(arr)
end subroutine run

program main
  call run
end program main

Reproducer 2: OpenMP target teams distribute parallel do + nested if in pure routine fails

If I replace the do concurrent with OpenMP teams/distribute/parallel-do, I see a different failure:

module stubs
  use iso_fortran_env
  integer, parameter :: r_typ = REAL64
contains
  pure subroutine f1(n, x)
    integer, intent(in) :: n
    real(r_typ), dimension(n), intent(in) :: x
    real(r_typ) :: a
    a = 0._r_typ
  end subroutine f1
  pure subroutine f2(x, y)
    real(r_typ), intent(in) :: x, y
    real(r_typ) :: p
    if (x.eq.0._r_typ) then
      if (y.ge.0._r_typ) then
        p = 2
      else
        p = -2
      end if
    end if
  end subroutine f2
end module

subroutine run
  use iso_fortran_env
  use stubs
  implicit none
  integer :: i, m, n
  real(r_typ) :: a, b
  real(r_typ), allocatable :: arr(:)

  m = 4
  n = 1
  allocate(arr(m))
  arr(1) = 1._r_typ

!$omp target teams distribute parallel do &
!$omp        map(to: arr, m, n) &
!$omp        private(i, a, b)
  do i = 1, n
    call f1(m, arr)
    a = 1._r_typ
    b = 1._r_typ
    call f2(a, b)
  enddo

  deallocate(arr)
end subroutine run

program main
  call run
end program main

Error:

f1:
5, Generating acc routine seq
Generating NVIDIA GPU code
NVFORTRAN-W-0155-Compiler failed to translate accelerator region (see -Minfo messages): Missing branch target block (small_reproducer.F90: 11)
f2:
11, Generating acc routine seq
Generating NVIDIA GPU code
NVFORTRAN-F-0704-Compilation aborted due to previous errors. (small_reproducer.F90)
NVFORTRAN/x86-64 Linux 26.1-0: compilation aborted

However, rewriting f2 to a branchless form using merge() makes it compile:

module stubs
  use iso_fortran_env
  integer, parameter :: r_typ = REAL64
contains
  pure subroutine f1(n, x)
    integer, intent(in) :: n
    real(r_typ), dimension(n), intent(in) :: x
    real(r_typ) :: a
    a = 0._r_typ
  end subroutine f1
  pure subroutine f2(x, y)
    real(r_typ), intent(in) :: x, y
    real(r_typ) :: p
    p = merge(merge(2._r_typ, -2._r_typ, y.ge.0._r_typ), 0._r_typ, x.eq.0._r_typ)
  end subroutine f2
end module

subroutine run
  use iso_fortran_env
  use stubs
  implicit none
  integer :: i, m, n
  real(r_typ) :: a, b
  real(r_typ), allocatable :: arr(:)

  m = 4
  n = 1
  allocate(arr(m))
  arr(1) = 1._r_typ

!$omp target teams distribute parallel do &
!$omp        map(to: arr, m, n) &
!$omp        private(i, a, b)
  do i = 1, n
    call f1(m, arr)
    a = 1._r_typ
    b = 1._r_typ
    call f2(a, b)
  enddo

  deallocate(arr)
end subroutine run

program main
  call run
end program main