NVHPC 26.3: `thrust::reduce` fails with `cudaErrorInvalidValue` in multi-arch fat binaries containing `cc100` (Blackwell); 26.1 is correct

Summary

Under NVIDIA HPC SDK 26.3, a thrust::reduce over a thrust::device_vector
fails at runtime on a GB200 (sm_100) when nvc++ -cuda builds a multi-arch
fat binary that includes cc100:

terminate called after throwing an instance of
  'thrust::_V_300201_SM_80_90_100_NVHPC::system::system_error'
  what():  after reduce invocation: cudaErrorInvalidValue: invalid argument

The same source, same flags, same GPU:

  • works under 26.1 for every arch list tested,
  • works under 26.3 when built single-arch (-gpu=cc100),
  • fails under 26.3 for -gpu=cc90,cc100 and -gpu=cc80,cc90,cc100.

So it appears to be a multi-arch dispatch problem introduced between 26.1 and
26.3, not a general Thrust or Blackwell failure. The mangled namespace in the
exception type — _V_300201_SM_80_90_100_NVHPC — encodes the arch list, which is
consistent with arch-list-dependent dispatch being the affected machinery.

Reproducer

18 lines, no MPI, no third-party libraries, single GPU:

// minimal.cu
#include <thrust/device_vector.h>
#include <thrust/reduce.h>
#include <thrust/execution_policy.h>
#include <cstdio>

int main() {
    const int n = 1024;
    thrust::device_vector<double> d(n, 1.0);
    double s = thrust::reduce(thrust::device, d.begin(), d.end(), 0.0);
    printf("sum = %.1f (expected %d)  %s\n", s, n, s == double(n) ? "OK" : "WRONG");
    return s == double(n) ? 0 : 1;
}

Compiler commands and results

nvc++ -cuda -gpu=<ARCH_LIST> -std=c++17 -O2 minimal.cu -o m && ./m
-gpu= NVHPC 26.1 NVHPC 26.3
cc100 sum = 1024.0 OK (rc 0) sum = 1024.0 OK (rc 0)
cc90,cc100 sum = 1024.0 OK (rc 0) throws, rc 134
cc80,cc90,cc100 sum = 1024.0 OK (rc 0) throws, rc 134

The exception’s namespace tracks the requested arch list exactly:

-gpu=cc90,cc100       ->  thrust::_V_300201_SM_90_100_NVHPC::system::system_error
-gpu=cc80,cc90,cc100  ->  thrust::_V_300201_SM_80_90_100_NVHPC::system::system_error

Both report what(): after reduce invocation: cudaErrorInvalidValue: invalid argument. The single-arch build (cc100) is the only cc100-capable
configuration that succeeds under 26.3, which is why the multi-arch code path
looks like the affected machinery.

cuobjdump --list-elf confirms all requested architectures are present in every
binary (e.g. sm_80 sm_90 sm_100 for the three-arch case), so this is not a
missing-SASS or JIT-fallback problem.

Arch lists that exclude cc100 (e.g. -gpu=cc90) fail on this GPU under both
SDKs with cudaErrorUnsupportedPtxVersion, which is simply the expected result of
running a binary built for a different architecture. Those rows are not part of
the report.

Environment

GPU 4x NVIDIA GB200, compute capability 10.0
Driver 580.126.20
Host arch aarch64 (Grace), Ubuntu 24.04
SDKs nvhpc_2026_261_Linux_aarch64_cuda_13.1, nvhpc_2026_263_Linux_aarch64_cuda_13.1
CUDA 13.1 (bundled, identical in both SDKs)
Thrust 3.2.1 (per the exception namespace _V_300201_...)
Compilers nvc++ 26.1-0 and nvc++ 26.3-0, both linuxarm64 ... -tp grace

Both SDKs were installed from the official tarballs to separate prefixes on the
same machine and exercised against the same GPU, so the compiler version is the
only variable between the two columns.

How this was found (why it matters beyond the throw)

The reduced case above fails loudly. It was found while tracking down a silent
failure in a real application: a quantum-chemistry eigensolver whose CUDA/Thrust
backend, built -gpu=cc80,cc90,cc100 with 26.3, returned wrong numerical
results with exit code 0
— a different wrong value on each run (a converged
energy of approximately -72 to -73 Ha instead of -76.2359466308, spread ~1.4 Ha
over 10 runs). The same source built with 26.1 produced the correct value
bit-identically across runs, and the application’s CPU and OpenMP-target-offload
backends were correct under both SDKs.

Both symptoms occur only with 26.3 + multi-arch + cc100, so they are plausibly
the same underlying defect — but that is an observation, not a proven identity,
and the reduced case is the one we can demonstrate cleanly.

Limitations of this report

  • Tested only on aarch64/GB200. We do not have an x86_64 Blackwell host
    available, so we cannot say whether the host architecture matters.
  • Not attempted with nvcc against upstream CCCL, so we cannot say whether this
    originates in the SDK’s Thrust packaging/codegen or in CCCL itself.
  • 26.5 could not be tested: there is no 26.5-*-cuda13.1-ubuntu24.04
    container tag, so we could not check whether it is already fixed.

Workarounds in use

  • Pin NVHPC 26.1, which is correct for every arch list.
  • Or build single-arch per GPU generation (-gpu=cc100 alone is correct under
    26.3), at the cost of one binary/image per generation.

Thanks for the report h_f_wen and welcome!

I was able to reproduce the issue here and have filed a report, TPR#38953. We’ll have engineering investigate.

-Mat