# Title
TensorRT 10.6: fused FP16 xmma GEMM kernel hangs under CUDA Graph replay – CTAs spin forever on a semaphore counter that reads 0 after their own RED.ADD (RTX 5000 Ada, sm_89)
# Summary
Our inference process hangs intermittently (about once every few hours) when the TensorRT enqueue is captured into a CUDA Graph and replayed. The host thread is stuck in `cuStreamSynchronize`, the GPU shows 100% SM utilization at low power, and there is no Xid in dmesg. Attaching cuda-gdb shows 34 CTAs of a TensorRT-internal fused FP16 GEMM kernel spinning in a wait loop on a global semaphore counter that never reaches the expected value. Disassembly suggests the kernel zeroes its own semaphore buffer in its prologue with no grid-level synchronization, so an increment can be lost. We would like to know whether this is a known issue and whether it is fixed in a later release.
# Environment
- GPU: NVIDIA RTX 5000 Ada Generation (AD103GL, sm_89, 76 SMs), 16 GB
- Driver 550.163.01, CUDA 12.4 (runtime 12.4.99), cuDNN 9.1.0.70
- TensorRT 10.6.0.26 (tar install, Python bindings 10.6.0), Python 3.8, pycuda 2022.2.2
- OS: Ubuntu 20.04, kernel 5.4.0-216
- Environment variables: `CUDA_SYNC_FLAGS=1`, `CUDA_DEVICE_MAX_CONNECTIONS=1`, `CUDA_DISABLE_PTX_JIT=1`, `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`. We have already tried changing these, and it did not help.
- Engine: FP16 detection + segmentation model, about 708 MB, input 1024x576. Engine build TensorRT version: 10.6.0.26 (same as the runtime)
- Single GPU, one inference process, no MPS, no other process using the GPU at hang time.
# Workload
Per frame we do: H2D copy → a custom fused preprocessing kernel → `execute_async_v3` → D2H copies, all captured into one CUDA Graph with the driver API (`cuStreamBeginCapture_v2`, thread-local capture mode, `cuGraphInstantiateWithFlags`, `cuGraphLaunch`). The graph is re-captured about every 41 s (about 900 frames) and runs at about 22-30 fps. The same workload with the CUDA Graph disabled has not hung for us.
# Observed hang
It happened twice on 2026-11-05: at 12:18:22 (frame 49591) and at 16:17:10 (frame 17719, 11 minutes after a reboot).
- Host: main thread in `pycuda::stream::synchronize()` spinning inside libcuda. No other thread is in a CUDA call.
- GPU: utilization 100%, memory utilization 0%, about 44 W. A kernel launched from a separate process on the same GPU runs normally, so the GPU is not globally wedged.
- dmesg: no Xid at the time of the hang. Earlier in the same boot we saw 9 Xid 31 (MMU fault) events, but none correlate with the hang times, and we do not know whether they are related.
# cuda-gdb findings (attached to the live hung process, then detached)
The stuck kernel is:
```
sm80_xmma_gemm_f16f16_f16f16_f16_nn_n_tilesize256x128x32_stage3_warpsize4x2x1_tensor16x8x16_fused
grid (1,48,1), block (256,1,1)
```
- 34 CTAs, (0,14,0) to (0,47,0), are resident and running, each on its own SM. The other 14 CTAs, (0,0,0) to (0,13,0), have finished.
- In each stuck CTA, warp 0 lane 0 loops in the following SASS (offsets from the kernel start). Warps 1-7 wait at the following `BAR.SYNC`.
```
+17680 MEMBAR.SC.GPU
+17808 RED.E.ADD.STRONG.GPU [R34.64], R14 ; R14 = 1
+17856 YIELD
+17920 @P0 ATOMG.E.ADD.STRONG.GPU PT, R13, [R34.64], RZ
+17952 ISETP.NE.AND P0, PT, R13, c[0x0][0x3fc], PT
+17968 @P0 BRA 0x45c0 ; back to YIELD
```
- Kernel parameters (read as `@parameter N` = `c[0x0][N]`; validated because `c[0x5b8]` equals the counter array base): `c[0x0][0x3fc] = 1`, `c[0x0][0x5c0] = 48`, `c[0x0][0x5b8] = 0x7f9c7cd3a600`.
- The counter array has one 32-bit counter per CTA at `c[0x5b8] + 4*y`. Counters for y = 0..13 are 1. Counters for y = 14..47 are **0**, even though those CTAs are already past their own `RED.ADD +1` and are waiting for the counter to equal 1.
- In the kernel prologue (SASS +416 and +768), threads zero this buffer with `STG.E [c[0x5b8] + 4*i], RZ` in a grid-stride loop. We found no grid-wide synchronization between that zeroing and the later `RED.ADD`.
- Hypothesis (not verified): if the CTAs responsible for zeroing run later than other CTAs’ `RED.ADD`, the increments are overwritten and the waiting CTAs hang forever. Another possibility is that a different kernel sharing the same workspace zeroed these counters. We could not distinguish these from the live state.
# Reproduction
- We have no deterministic reproducer. It happens about once every few hours with the CUDA Graph enabled.
- An internal stress script (different, older engine, pycuda-based graph code, no real camera pipeline) ran 6 times, up to 10.8 hours, without reproducing it.
- We have not tested whether this reproduces with `trtexec --useCudaGraph`.
# Questions
1. Is this a known issue? We found the 10.0.1/10.1.0 note about an H100 + r550 + CUDA graph hang, and the 10.7.0 note about a Hopper multi-stream warp-synchronization hang, but neither names this kernel.
2. Does this xmma kernel expect its semaphore workspace to be zeroed externally (for example by a memset), or does it zero it itself? Could the zeroing be lost or reordered under CUDA graph capture/replay?
3. Is there a fix in TensorRT 10.7-10.10 that covers this? We plan to test those versions.
4. Is there a supported way to avoid this tactic (for example excluding it at engine build time) while we investigate?
# Attachments (available on request)
- Full SASS disassembly of the stuck kernel (2455 lines), via `cuda-gdb` `disassemble`
- cuda-gdb outputs: warp/lane state, `info cuda blocks/kernels`, counter array dump, kernel parameter dump
- Host thread backtraces (gdb) at hang time, `nvidia-smi` and dmesg snapshots
forum_attach_trt_hang.tar.gz (40.3 KB)