Hi,
We use cuDSS’s uniform batch mode (CUDSS_CONFIG_UBATCH_SIZE) to solve many Newton-Raphson systems with the same sparsity pattern at once (power-flow security analysis). With batch sizes above ~160, part of the solutions come back as NaN, and whether they do depends on the content of device memory that was never given to cuDSS.
Minimal reproduction (attached, ~80 lines, CuPy + SciPy + nvmath.bindings.cudss):
- a batch of identical, well-conditioned tridiagonal systems (n = 100, diagonal 5, off-diagonals 1), CSR, general, full view, default configuration except CUDSS_CONFIG_UBATCH_SIZE;
- one CSR descriptor and dense descriptors for a single system (n x 1, ld n); values, rhs and x hold the UBATCH_SIZE systems back to back;
- ANALYSIS, FACTORIZATION, SOLVE;
- before creating the cuDSS objects, the script allocates an unrelated 64 MiB buffer with cudaMalloc, fills it with a byte pattern, keeps it allocated and never passes it to cuDSS.
- every solution is compared with the exact one.
python cudss_ubatch_repro.py [--batch 512] [--fill 255|0|none] [--mb 64]
cudss_ubatch_repro.txt (4.1 KB)
Results (RTX PRO 6000 Blackwell Server Edition, driver 610.43.02, CUDA runtime 13.2, cuDSS 0.8.0, CuPy 14.2.0):
| UBATCH_SIZE | unrelated buffer filled with 0xFF | filled with 0x00 | no buffer |
|---|---|---|---|
| 512 | 155-181 of 512 solutions contain NaN | 0 | 0 |
| 256 | 55 / 256 | 0 | 0 |
| 192 / 224 / 240 / 255 | 4 / 24 / 46 / 52 | - | - |
| 64, 128, 129, 160 | 0 | 0 | 0 |
The other solutions are exact (max error 9e-16). The number of NaN solutions grows with the size of the 0xFF region, and no single 4 MiB part of it triggers the problem on its own.
What we checked:
- compute-sanitizer --tool memcheck and --tool initcheck: 0 errors, while the NaN still appear;
- NaN guard zones before and after each of our buffers (values, rhs, x, row offsets, column indices): never read;
- a device memory handler (cudssSetDeviceMemHandler) that zero-fills every cuDSS allocation, plus padding, and never frees: the NaN still appear with the unrelated 0xFF buffer;
- CUDA_MODULE_LOADING=EAGER and warm-up solves before filling the buffer: no change;
- a random sparsity pattern of the same size: no NaN; our power-flow Jacobians: NaN in the same conditions.
So it looks like a kernel in the uniform batch path reads from addresses outside the buffers it was given or allocated, for batch sizes above ~160.
Questions:
- Is this a known issue, and is it fixed in a later cuDSS release?
- Is UBATCH_SIZE <= 128 a safe workaround in the meantime, or is something wrong in how we describe the batch?
Thanks!
PS: We also have a pure C version of the same reproduction (CUDA runtime + cuDSS only, no Python, ~150 lines, built with gcc). It gives the same results, prints every buffer’s address range and checks that none of the cuDSS inputs / outputs overlaps the unrelated buffer. Happy to share it if that is more convenient for you.