HDF5 develop testing on GitHub with nvhpc 26.3.0 fails to compile test/dt_arith.c, crashing LLVM

Accelerated Computing HPC Compilers nvc, nvc++ and nvfortran
1

  • This does not happen with nvhpc 25.9 and 25.11 versions.

  • We’re running the workflow on a github runner ubuntu-latest (Ubuntu 24.04.4).

  • MPI_C: /opt/nvidia/hpc_sdk/Linux_x86_64/26.3/comm_libs/hpcx/bin/mpicc (found version “3.1”)

  • Adding CMAKE_C_FLAGS=“-Mnovect” allows the compile to succeed.

    Compile output for hdf5/test/dt_arith.c:

    [1230/4123] Building C object test/CMakeFiles/dt_arith.dir/dt_arith.c.o

    FAILED: [code=2] test/CMakeFiles/dt_arith.dir/dt_arith.c.o

    /opt/nvidia/hpc_sdk/Linux_x86_64/26.3/comm_libs/hpcx/bin/mpicc -DH5_BUILT_AS_DYNAMIC_LIB -DH5_HAVE_TEST_API -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_POSIX_C_SOURCE=200809L -I/home/runner/work/hdf5/hdf5/src -I/home/runner/work/hdf5/hdf5/src/H5FDsubfiling -I/home/runner/work/hdf5/build/src -I/home/runner/work/hdf5/build/test -I/home/runner/work/hdf5/hdf5/test -c11 -Minform=warn -s -fast -O3 -DNDEBUG -c11 -fPIE -MD -MT test/CMakeFiles/dt_arith.dir/dt_arith.c.o -MF test/CMakeFiles/dt_arith.dir/dt_arith.c.o.d -o test/CMakeFiles/dt_arith.dir/dt_arith.c.o -c /home/runner/work/hdf5/hdf5/test/dt_arith.c

    “/home/runner/work/hdf5/hdf5/test/dt_arith.c”, line 4043: warning: conversion may change the value [lossy_conversion]

    *((H5__Float16 *)hw_dst) = (H5__Float16)aligned;

    ^

    Remark: individual warnings can be suppressed with “–diag_suppress ”

    “/home/runner/work/hdf5/hdf5/test/dt_arith.c”, line 4158: warning: conversion may change the value [lossy_conversion]

    *((H5__Float16 *)hw_dst) = (H5__Float16)aligned;

    ^

    “/home/runner/work/hdf5/hdf5/test/dt_arith.c”, line 4284: warning: conversion may change the value [lossy_conversion]

    *((H5__Float16 *)hw_dst) = (H5__Float16)aligned;

    ^

    “/home/runner/work/hdf5/hdf5/test/dt_arith.c”, line 4420: warning: conversion may change the value [lossy_conversion]

    *((H5__Float16 *)hw_dst) = (H5__Float16)aligned;

    ^

    “/home/runner/work/hdf5/hdf5/test/dt_arith.c”, line 4539: warning: conversion may change the value [lossy_conversion]

    *((H5__Float16 *)hw_dst) = (H5__Float16)aligned;

    ^

    “/home/runner/work/hdf5/hdf5/test/dt_arith.c”, line 4706: warning: conversion may change the value [lossy_conversion]

    *((H5__Float16 *)hw_dst) = (H5__Float16)aligned;

    ^

    LLVM ERROR: Cannot select: 0x55c3d35c73e0: v2f64 = X86ISD::SCALEFS 0x55c3d35b67f0, 0x55c3d3976390, /home/runner/work/hdf5/hdf5/test/dt_arith.c:5695:1

    0x55c3d35b67f0: v2f64 = scalar_to_vector 0x55c3d36ea3a0, /home/runner/work/hdf5/hdf5/test/dt_arith.c:5695:1

    0x55c3d36ea3a0: f64,ch = CopyFromReg 0x55c3d361e948, Register:f64 %2786, /home/runner/work/hdf5/hdf5/test/dt_arith.c:5695:1

    0x55c3d36e9d10: f64 = Register %2786

    0x55c3d3976390: v2f64 = scalar_to_vector 0x55c3d35ca170, /home/runner/work/hdf5/hdf5/test/dt_arith.c:5695:1

    0x55c3d35ca170: f64 = sint_to_fp 0x55c3d31588b0, /home/runner/work/hdf5/hdf5/test/dt_arith.c:5695:1

    0x55c3d31588b0: i32 = sub 0x55c3d38ece30, 0x55c3d388b560, /home/runner/work/hdf5/hdf5/test/dt_arith.c:5688:1

    0x55c3d38ece30: i32,ch = CopyFromReg 0x55c3d361e948, Register:i32 %2787, /home/runner/work/hdf5/hdf5/test/dt_arith.c:5688:1

    0x55c3d3158ae0: i32 = Register %2787

    0x55c3d388b560: i32,ch = CopyFromReg 0x55c3d361e948, Register:i32 %2788, /home/runner/work/hdf5/hdf5/test/dt_arith.c:5688:1

    0x55c3d35c9fb0: i32 = Register %2788

    In function: test_conv_flt_1

    PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.

    Stack dump:

    0. Program arguments: /opt/nvidia/hpc_sdk/Linux_x86_64/26.3/compilers/share/llvm/bin/llc /tmp/nvcB1LjrV3IvwKp.llvm -march=x86-64 -mcpu=native --mattr=+mmx --mattr=+sse --mattr=+sse2 --mattr=+sse3 --mattr=+ssse3 --mattr=+sse4a --mattr=+sse4.1 --mattr=+sse4.2 --mattr=+avx --mattr=+avx2 --mattr=+f16c --mattr=+fma --mattr=+xsave --mattr=+xsaveopt --mattr=+xsavec --mattr=+xsaves --mattr=+popcnt --mattr=+sha --mattr=+aes --mattr=+pclmul --mattr=+clflushopt --mattr=+fsgsbase --mattr=+rdrnd --mattr=+bmi --mattr=+bmi2 --mattr=+lzcnt --mattr=+fxsr --mattr=+clzero --mattr=+vaes --mattr=+vpclmulqdq --mattr=+adx --mattr=+clwb --mattr=+cx16 --mattr=+movbe --mattr=+prfchw --mattr=+rdpid --mattr=+rdseed --mattr=+sahf --mattr=+shstk -O3 --x86-cmov-converter=0 --dwarf-directory=false --non-global-value-max-name-size=-1 --align-all-functions=6 --nvhpc-select-transform=true --nvhpc-block-freq=true --nvhpc-use-text-section-for-constants=true --denormal-fp-math=positive-zero --nvhpc-memzero --nvhpc-optimize-select --nvhpc-dso-opt -relocation-model=pic -filetype=obj --frame-pointer=none -o test/CMakeFiles/dt_arith.dir/dt_arith.c.o

    1. Running pass ‘Function Pass Manager’ on module ‘/tmp/nvcB1LjrV3IvwKp.llvm’.

    2. Running pass ‘X86 DAG->DAG Instruction Selection’ on function ‘@test_conv_flt_1’

    Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):

    0 llc 0x000055c39d71c688 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 40

    1 llc 0x000055c39d71a3de llvm::sys::RunSignalHandlers() + 238

    2 llc 0x000055c39d71cd5d

    3 libc.so.6 0x00007ff993045330

    4 libc.so.6 0x00007ff99309eb2c pthread_kill + 284

    5 libc.so.6 0x00007ff99304527e gsignal + 30

    6 libc.so.6 0x00007ff9930288ff abort + 223

    7 llc 0x000055c39d69e2f7 llvm::report_fatal_error(llvm::Twine const&, bool) + 471

    8 llc 0x000055c39d561c9f

    9 llc 0x000055c39d5611b5

    10 llc 0x000055c39c4faea4

    11 llc 0x000055c39d55891c llvm::SelectionDAGISel::DoInstructionSelection() + 396

    12 llc 0x000055c39d5580c4 llvm::SelectionDAGISel::CodeGenAndEmitDAG() + 1316

    13 llc 0x000055c39d556c91 llvm::SelectionDAGISel::SelectAllBasicBlocks(llvm::Function const&) + 3041

    14 llc 0x000055c39d555582 llvm::SelectionDAGISel::runOnMachineFunction(llvm::MachineFunction&) + 322

    15 llc 0x000055c39d5540ce llvm::SelectionDAGISelLegacy::runOnMachineFunction(llvm::MachineFunction&) + 158

    16 llc 0x000055c39cb3858d llvm::MachineFunctionPass::runOnFunction(llvm::Function&) + 589

    17 llc 0x000055c39cf62a3a llvm::FPPassManager::runOnFunction(llvm::Function&) + 714

    18 llc 0x000055c39cf69803 llvm::FPPassManager::runOnModule(llvm::Module&) + 51

    19 llc 0x000055c39cf63719 llvm::legacy::PassManagerImpl::run(llvm::Module&) + 2553

    20 llc 0x000055c39c164da4 main + 10180

    21 libc.so.6 0x00007ff99302a1ca

    22 libc.so.6 0x00007ff99302a28b __libc_start_main + 139

    23 llc 0x000055c39c15f2ee _start + 46

    nvc-Fatal-/opt/nvidia/hpc_sdk/Linux_x86_64/26.3/compilers/share/llvm/bin/llc TERMINATED by signal 6

2

Hi lrknow,

Welcome and thanks for the report!

Which version of HDF5 are you using?

I just downloaded hdf5-2.1.1 and for good or bad, the compilation of dt_arith.c succeeds. Nor am I seeing any warning messages.

I ran it though the cmake build but also compiled by hand using the command you posted.

Could be a difference in the HDF5 version, or possibly a system difference. I’ll try again on a Ubuntu 24.04 just in case.

Running the test sees some failures, but I can look at those after I can recreate the LLC crash.

-Mat

3

We are testing the develop branch. It should behave similarly to the 2.1.1 version. I ran the workflow with the 2.1.1 tag. The build was successful the first time, so I ran it 2 more times, and the compile of dt_arith.c crashed llvm both of those times.

When dt_arith is built successfully, it will always fail certain subtests generally involving conversion to or from long double types. For nvhpc the dt_arith test is run as a test expected to fail. It is intended to fix that at some point, but is not a high priority. The source of that problem is believed to be in the dt_arith code, but the compile issue seems to be unique to nvhpc 26.3.0.

We intend to use the -Mnovect flag workaround for our GitHub CI for now.

4

I’ll keep trying to recreate.

Though I’m wondering since it seems to be related to vectorization, if it could be architecture specific.

What is the CPU being used for the test?

5

According to Dasv5 size series - Azure Virtual Machines | Microsoft Learn

Processor 2 - 96 vCPUs AMD EPYC 7763v (Milan) [x86-64]
AMD EPYC 9004 (Genoa) [x86-64]

After several more rounds with Claude, it came up with
-DCMAKE_C_FLAGS=“-tp=haswell”
-DCMAKE_Fortran_FLAGS=“-tp=haswell”

In a dozen test runs those seem to be consistently effective to prevent the crash. Is this flag correct for those processors?

C++ was not enabled; perhaps it would need the same flag

Some updates and corrections:

-Mnovect did not fix the problem. The LLVM crash happened sometimes with or without -Mnovect.

Looking closely at the history of the test runs, the first instance of the LLVM crash was 15 days (more than 15 test runs) after the nvhpc version was updated from 25.9 to 26.3. Running tests again with versions 25.9 and 25.11 now show the occasional crash with the earlier versions as well, so it is not a specific version 26.3 issue.

Thanks!

Larry

6

Hi Larry,

Is this flag correct for those processors?

No, haswell is an older (circa 2013) Intel processor. For Milan and Genoa, the target processors would be “znver3” or “znver4” respectively. i.e. AMD Zen Processors versions 3 or 4.

By selecting an older processor, it does point to some type of issue when targeting these newer ones. Though when I was testing earlier, I was on a Milan.

I was away last week mentoring at a hackathon so wasn’t able to get back to this, but will try on a Genoa in a few.

-Mat