I am getting this error from cusparseSpGEMM_workEstimation while trying to multiply two not-so-large sparse matrices (both 900x900, one with 900 nonzero, the other 1699 nonzero).
I read here of a similar situation, where this error comes from cusparseSpGEMM_compute. A developer says:
It is not a bug but a limitation of the algorithm itself. CUSPARSE_STATUS_INSUFFICIENT_RESOURCES refers to all conditions that prevent computing the results. In your specific case, there is a segment of the intermediate products that cannot be codified in a 32-bit integer. Anyway, we are planning a new algorithm that uses less memory and does not have these issues.
Waiting for the mentioned new algorithm, in the meanwhile it would be really useful to have some indication on what can trigger the error (matrix sparsity pattern? values?) and possible workarounds.
Hi. Error in cusparseSpGEMM_workEstimation often occurs when the total number of intermediate products A[i,k]*B[k,j] is higher than 2^31. The common pattern for this case is when A has a long kth column and B has a long kth row.
However, the matrices are small in your case, this shouldn’t happen. We have to check the structure of your matrices to understand the reason for the issue. You can check if the total number of products is high in your case.
The workarounds can be:
use legacy API csrgemm2. Note that this legacy API will be removed in the next release. Its performance is lower than the current cusparseSpGEMM API.
If possible, break the AxB computation into smaller ones: Ax(B1 + B2 + B3 + …). If B is divided by columns, we shouldn’t get errors. I understand that this comes with high cost of breaking B and combining results.
Counting the number of intermediate products A[i,k]*B[k,j] i find that it is at most 1 for all i (row of A) and j (column of B) (the total number of nonzero results is 13470). So this should not be the problem.
(By the way, just to make sure I understand: when you say “… when A has a long k-th column and B has a long k-th row”, do you mean “A has a row with many nonzero elements and B has a column with many nonzero elements”?)
Could it be of help if I upload the two matrices in CSR format?
Of course I am going to try the legacy csrgemm2. I cannot find any example for this call, could you please point out one for me?
I meant total number of intermediate products of the entire AxB computation (for all i, k and j of A[i,k]*B[k,j]). Sorry for not being clear. Is that number 13470 as you said?
No, it’s actually reversed, i.e. A has a column with many nonzeros and B has a row with many nonzeros and both row and column are at the same index k (kth row and kth column).
It’s better if you can have the plot of the matrix’s structures.
You can find a small example in the cuSPARSE documentation. Let me know if that’s enough.
While making the plots I found a mistake in my preparation of the CSR: I correctly allocated M+1 elements for rowOffset, but then forgot to set the last one.
Now that cusparseSpGemm knows where the matrices end, it works fine :) No need to try csrgemm2.
Sorry for the false alarm and thank you for your time.