Feature request: support many distributed small systems in cuDSS

Accelerated Computing GPU-Accelerated Libraries
,
1

I would like to request support for solving many independent small systems where each system is distributed along the axis being solved.

My use case is a structured-grid PDE code. The domain is decomposed across MPI ranks in the y direction. Each rank owns data shaped logically like:

nx * nz * ny_local

I need to solve many independent pentadiagonal systems in y, one for each (x, z) line.

At the moment, it is not clear that cuDSS supports this use case. The documentation appears inconsistent:

  • The release notes for cuDSS 0.6.0 say: “Disabled support for batched matrices in the MGMN mode.”
  • The current MGMN limitations say: “MGMN mode does not support matrix batches.”
  • cudssMatrixCreateBatchDn() also says: “MGMN mode does not support matrix batches.”
  • However, the cudssExecute() documentation still describes requirements “in the batch case” for MGMN, including that all processes must have a valid batchCount .
  • The Examples for MGMN say that it is for large matrices.

What is the current support status of batches for MGMN?

It would be very useful if cuDSS supported this class of problems directly: many independent tridiagonal/pentadiagonal/banded systems, with each system distributed along the solve direction.

My current approaches to the problem are

  • global transpose before/after the solve, and use cusparse
  • custom distributed Schur-like pentadiagonal solver (cusparse does not expose the partially eliminated matrix, so I have do to this myself)

Thanks in advance!

2

Hi @jonathan.neuhauser !

Sorry for the confusion caused by the note in the docs for cudssExecute(). Currently, neither uniform batches nor non-uniform batches are supported by MGMN mode (but ofc we would like to enable this case in the future).

However, note that cuDSS is a general sparse direct solver and as a such, it is not targeting solving banded matrices (especially with a narrow band), there are special algorithms which can likely do it faster.

This is one of the reasons why, e.g. solving banded linear systems is exposed in certain dense linear algebra libraries, not sparse “general” ones.

So if this is the case, I would actually look into making it a feature request for cuSolver rather than cuDSS.

Thanks,
Kirill