Sparse matrix solve fails with cuda-sparse with cuDSS enabled

Accelerated Computing GPU-Accelerated Libraries
1

sparse matrix solve fails with cuda-sparse with cuDSS enabled. The same solves fine with traditional Eigen SparseLU. I compared the inputs and RHS they match exactly. But after one iteration.

catastrophic mismatch at same index/order:

  • variable (24: bus 8, e)

  • default: -2.726349437177e-02

  • cuDSS: 3.522308383149e+11

Could you advise:

  1. Any known cuDSS limitations/instabilities on Pascal SM 6.1 for this class of unsymmetric sparse systems?

  2. Recommended robust config (reordering/factorization/solve algorithm, pivot settings) for this case?

  3. Whether extra scaling / matrix type settings are required for stability here?

  4. Which cudssDataGet diagnostics are best to capture root cause (CUDSS_DATA_INFO, permutation/pivot-related outputs, etc.)?

2

Hi @vrkanduri!

Which version of cuDSS are you using? You can see the release notes (incl. the known issues) for different versions here: Release Notes — NVIDIA cuDSS

As the very first thing to check, please set CUDSS_LOG_LEVEL=5 in the environment and see if there are any errors reported in the log.

For investigating accuracy issues, please have a look at cuDSS Tips and tricks — NVIDIA cuDSS.

Assuming there are no hard failures in the verbose log, as a blind guess, I’d assume that cuDSS has perturbed pivots (which you can check via cudssDataGet() after factorization) and as there is iterative refinement by default, there is no way to mitigate the that. It is very likely that Eigen uses different default options than cuDSS and this either changes the number of perturbed pivots, or their magnitude.

If I am right, then there are multiple ways you can fix the issue. First, you can enable matching. Second, you can experiment with adding iterative refinement steps [but for ill-conditioned problems IR often does not help]. Third, you can play with the pivoting epsilon parameters. Third, you can use a more robust (but also slower) factorization/reordering via setting CUDSS_ALG_1 for reordering algorithm (factorization will be switched correspondingly automatically).

If none of this helps, could you share your matrix (as an mtx or format of your choice + reader) and cuDSS settings you use (+ the log with CUDSS_LOG_LEVEL=5)?

Thanks,
Kirill