cuBLASLt sm_120 (Blackwell): TF32 split-K nvjet kernel raises "Warp Barrier Arrival Mismatch" — intermittent illegal access / GPU hang (RTX 5090)

Accelerated Computing GPU-Accelerated Libraries
,
1

Environment

  • GPU: NVIDIA GeForce RTX 5090 Laptop GPU (compute capability 12.0, sm_120, Blackwell)
  • Driver: 595.71.05
  • OS: Linux 6.17.0 (x86_64)
  • CUDA toolkit: 13.3 (nvcc V13.3.33)
  • cuBLAS / cuBLASLt: 13.5.1.27
  • Application: llama.cpp (CUDA backend), FP32 GEMMs issued via cuBLASLt

Summary
Under sustained inference, an internal cuBLASLt JIT kernel raises a Warp Barrier Arrival Mismatch device exception. It surfaces intermittently as either cudaErrorIllegalAddress (“an illegal memory access was encountered”) or a GPU hang (100% SM utilization, ~0% memory throughput — warps spinning at a named barrier that never completes). The failure is timing-dependent (stochastic threshold, ~15–180 GEMM-heavy iterations) and vanishes under any serialization (compute-sanitizer, or a cudaStreamSynchronize after each op) — a classic async heisenbug.

Faulting kernel (from CUDA device coredump)

CUDA Exception: Warp Barrier Arrival Mismatch
kernel: nvjet_sm120_sss_tf32_mma_32x32x64_6_16x16x64_tmaAB_alignCD4_splitK_TNNN
GridDim (8,1,16)  BlockDim (64,2,1)

This is the TF32 tensor-core, split-K GEMM kernel cuBLASLt selects for certain FP32 GEMM shapes on sm_120.

Kernel-name provenance: this kernel name and grid/block were captured by cuda-gdb from a device coredump during initial isolation on 2026-06-08. The original core was not retained, but the crash is reproducible — a fresh core can be regenerated in ~20 min with the recipe below (note: experts must be on the GPU, -ngl 99, not CPU-offloaded — the bug does not reproduce when expert matmuls run on CPU, since the cuBLAS FP32 path is then not exercised).

How it was captured
The bug vanishes under compute-sanitizer, so the faulting kernel was named via a CUDA device coredump (a host ulimit -c core does not capture device state):

CUDA_ENABLE_COREDUMP_ON_EXCEPTION=1 CUDA_ENABLE_LIGHTWEIGHT_COREDUMP=1 \
CUDA_COREDUMP_FILE=core_%p.nvcudmp  <run FP32-GEMM workload until it faults>
cuda-gdb -q -ex "target cudacore core_<pid>.nvcudmp" -ex "info cuda kernels" -ex "bt"

Reproduction

  1. Run an FP32 GEMM workload that cuBLASLt routes to the TF32 split-K kernel above on an RTX 5090, with experts on GPU — e.g. llama.cpp decoding google/gemma-4-26B-A4B-it MXFP4_MOE GGUF, -ngl 99 (TF32 enabled, i.e. default).
  2. Apply sustained load (hundreds of GEMMs). The crash/hang appears stochastically, typically before ~180 iterations.

Proven workaround
Disabling TF32 (NVIDIA_TF32_OVERRIDE=0) makes cuBLASLt select a non-TF32 FP32 kernel and the failure disappears: 3 independent runs of 300 generations each completed cleanly (vs systematic failure < ~180 with TF32 enabled), and the fix has since run cleanly for 960+ generations in a workload whose FP32 path is exercised even more heavily. Throughput impact is negligible (token-generation unchanged; prompt-GEMM −1.6%, within noise).

Questions for NVIDIA

  • Is the barrier-arrival mismatch a known issue in the nvjet_sm120_..._tf32_..._splitK_TNNN kernel’s intra-block named-barrier logic on sm_120 (Blackwell)? It looks like divergent / under-counted barrier participation in the split-K path for this tile configuration.
  • Is there a supported per-handle / per-operation way to force the non-TF32 FP32 kernel on sm_120 short of the global NVIDIA_TF32_OVERRIDE=0 — e.g. does cuBLASLt honor a CUBLAS_COMPUTE_32F_PEDANTIC compute type or an algo/heuristic flag to avoid this split-K TF32 kernel on Blackwell?

This is one of three distinct sm_120 GPU bugs isolated on the same workload; the other two are ggml/llama.cpp device-kernel issues: CUDA sm_120 (Blackwell, RTX 5090): `flash_attn_ext_f16<256,256,32,2>` raises "Warp Misaligned Register" device exception — intermittent crash; `-fa off` works around it · Issue #24398 · ggml-org/llama.cpp · GitHub (flash_attn_ext_f16, Warp Misaligned Register) and CUDA sm_120 (Blackwell, RTX 5090): out-of-range shared-memory store in `mul_mat_q<Q8_0,128>` MMA write-back epilogue — intermittent crash; eliminated by removing Q8_0 weights · Issue #24399 · ggml-org/llama.cpp · GitHub (mul_mat_q Q8_0, out-of-range shared store).

2

Thanks for reporting the issue @willowonevision We are currently working on reproducing the issue under the conditions you reported. Once we have confirmation of the erroneous behavior, we will provide update on any potential work-around (beyond what you currently use in the form of NVIDIA_TF32_OVERRIDE=0) and timeline for a fix

3

@willowonevision Since the issue is non-deterministic and happens intermittently, we would appreciate if you could please share your reproducer along with cuBLASLt logs? To collect logs, please set, export CUBLASLT_LOG_LEVEL=5 && export CUBLASLT_LOG_FILE=cublasLt.log . It appears that you are also collecting the coredump as well, so it would be great if you can share that as well from the failed run.