cuBLASLt returns silently wrong FP16/BF16/FP8 GEMM results on Hopper under CUDA_MPS_ACTIVE_THREAD_PERCENTAGE (algoId 66, cluster kernels)

Accelerated Computing GPU-Accelerated Libraries
, ,
1

n an H200(or H100), half-precision GEMM returns wrong results with no error when the process runs as an MPS client with CUDA_MPS_ACTIVE_THREAD_PERCENTAGE below 100. cublasLtMatmul returns CUBLAS_STATUS_SUCCESS, there is no Xid, and part of the output matrix is simply never written.

Environment: H200 (132 SMs), driver 580.178.04, cuBLAS 12.6.4.1 / 12.8.3.14 / 13.1.1.3 / 13.4.1.3 / 13.8.1.7. cuBLAS 12.4.5.8 is not affected.

Minimal reproduction (stock PyTorch image, one script):

nvidia-cuda-mps-control -d
docker run --rm --gpus all --ipc=host -v /tmp/nvidia-mps:/tmp/nvidia-mps -v $PWD:/w \
  -e CUDA_MPS_ACTIVE_THREAD_PERCENTAGE=14 \
  pytorch/pytorch:2.14.1-cuda13.0-cudnn9-runtime python /w/repro_min.py

SM=18 pct=14 torch.float32  MxKxN=16x2048x1000 bad=0/16000
SM=18 pct=14 torch.float16  MxKxN=16x2048x1000 bad=11597/16000 bad_cols=[256..999]
SM=18 pct=14 torch.bfloat16 MxKxN=16x2048x1000 bad=11569/16000 bad_cols=[256..999]

At 12 % (14 SMs) the same script reports bad=0. A C program that calls cublasLtMatmul directly reproduces it without PyTorch or Docker.

Scope

  • 66 of the 100 percentages give wrong results (14–22, 28–46, 52–89 %, i.e. 18–28, 36–60, 68–116 SMs). 100 % and no MPS are always correct.
  • FP16, BF16 and FP8 (E4M3) are affected; FP32 is not.
  • cuDNN is affected too, because it calls cublasLtMatmul for half-precision 1×1 convolutions. ResNet and Qwen3-8B produce garbage or non-finite outputs.
  • Only algoId=66 configurations with 4 or 8 CTAs per cluster fail. Cluster shapes with 1 or 2 CTAs never did.
  • MIG instances are correct, including the 26-SM and 60-SM profiles whose SM counts fail under MPS. B200 and RTX PRO 6000 Blackwell are correct.

What differs between a correct and a wrong run

For the same algorithm and the same kernel launch, one kernel argument differs: a per-cluster table. In every correct run it is a permutation of the cluster indices (e.g. 0 2 1 3 4 5); in every wrong run it has trailing zeros (e.g. 0 1 2 3 4 0, or 0 0 0 0), so several clusters compute the same part and the rest of the output is left unwritten. cuOccupancyMaxActiveClusters returns different values for an MPS share than for a MIG instance with the same SM count. A plain cluster kernel with the same launch geometry runs correctly under the same MPS share, so the driver’s cluster scheduling itself looks fine.

Questions

  1. Is this table derived from cuOccupancyMaxActiveClusters, and are the values returned under CUDA_MPS_ACTIVE_THREAD_PERCENTAGE meant to be a valid input for it?
  2. Is there a supported way to keep cuBLASLt (and therefore cuDNN) from selecting cluster shapes with more than 2 CTAs? CUBLASLT_MATMUL_DESC_SM_COUNT_TARGET does not help.
  3. Is a fix available or planned?

Full reproduction, per-percentage results, kernel-argument dumps and an LD_PRELOAD workaround that rewrites only the cluster shape:

2

Excellent detective work on this — the trailing-zeros observation in the per-cluster table really narrows it down to a cluster-sizing mismatch. Since CUBLASLT_MATMUL_DESC_SM_COUNT_TARGET didn’t help, you might have better luck restricting the cluster shape directly: cublasLtMatmulAlgoConfigSetAttribute with CUBLASLT_ALGO_CONFIG_CLUSTER_SHAPE lets you limit heuristic selection to shapes with 1-2 CTAs per cluster, which your testing shows are always correct. And given the silent corruption also hits cuDNN’s half-precision 1x1 convs, it’s worth filing this as a bug report so the library team can fix the occupancy query mismatch under MPS shares.

3

Thanks for reporting the issue and providing the reproducer.

We have reproduced the bug on our internal system. A fix is ready and will be released in an upcoming cuBLAS version. In the meantime, the workaround mentioned above should work as well.