Linking issues with cuSPARSE functions when using new versions of CUDA Fortran compilers

Hi,
I am getting a number of undefined reference messages when trying to compile with the new versions of the PGI compilers. The compiler is pgi/20.11 and I am compiling on XSEDE’s Bridges2 and getting the following messages (I checked in a few other clusters with the most recent NVIDIA SDK package and get the same messages):

/jet/home/asaadat/github/BDpack/src/semidilute_bs/cuda/hi_cumod.cuf:328: undefined reference to `cusparseCreateSolveAnalysisInfo'

./libsmdlt_bs.a(conv_cumod.o): In function `conv_cumod_init_conv_d_':

/jet/home/asaadat/github/BDpack/src/semidilute_bs/cuda/conv_cumod.cuf:127: undefined reference to `cusparseDcsr2csc'

./libsmdlt_bs.a(conv_cumod.o): In function `conv_cumod_rbtoq_':

/jet/home/asaadat/github/BDpack/src/semidilute_bs/cuda/conv_cumod.cuf:225: undefined reference to `cusparsedcsrmv_sethpm_'

./libsmdlt_bs.a(sprforce_cumod.o): In function `sprforce_cumod_update_force_':

/jet/home/asaadat/github/BDpack/src/semidilute_bs/cuda/sprforce_cumod.cuf:114: undefined reference to `cusparsedcsrmv_sethpm_'

./libsmdlt_bs.a(diffcalc_cumod.o): In function `diffcalc_cumod_pme_d_calcdf_recip':

/jet/home/asaadat/github/BDpack/src/semidilute_bs/cuda/diffcalc_cumod.cuf:1260: undefined reference to `cusparsedcsrmv_sethpm_'

/jet/home/asaadat/github/BDpack/src/semidilute_bs/cuda/diffcalc_cumod.cuf:1265: undefined reference to `cusparsedcsrmv_sethpm_'

/jet/home/asaadat/github/BDpack/src/semidilute_bs/cuda/diffcalc_cumod.cuf:1270: undefined reference to `cusparsedcsrmv_sethpm_'

./libsmdlt_bs.a(diffcalc_cumod.o):/jet/home/asaadat/github/BDpack/src/semidilute_bs/cuda/diffcalc_cumod.cuf:1382: more undefined references to `cusparsedcsrmv_sethpm_' follow

./libsmdlt_bs.a(diffcalc_mod.o): In function `diffcalc_mod_calcdiff_recip_dev_':

/jet/home/asaadat/github/BDpack/src/semidilute_bs/diffcalc_mod.f90:1381: undefined reference to `cusparseDcsr2csc'

./libsmdlt_bs.a(diffcalc_mod.o): In function `diffcalc_mod_pme_dev_calcdf_recip':

/jet/home/asaadat/github/BDpack/src/semidilute_bs/diffcalc_mod.f90:1906: undefined reference to `cusparsedcsrmv_sethpm_'

/jet/home/asaadat/github/BDpack/src/semidilute_bs/diffcalc_mod.f90:1910: undefined reference to `cusparsedcsrmv_sethpm_'

/jet/home/asaadat/github/BDpack/src/semidilute_bs/diffcalc_mod.f90:1914: undefined reference to `cusparsedcsrmv_sethpm_'

/jet/home/asaadat/github/BDpack/src/semidilute_bs/diffcalc_mod.f90:2376: undefined reference to `cusparsedcsrmv_sethpm_'

/jet/home/asaadat/github/BDpack/src/semidilute_bs/diffcalc_mod.f90:2380: undefined reference to `cusparsedcsrmv_sethpm_'

./libsmdlt_bs.a(diffcalc_mod.o):/jet/home/asaadat/github/BDpack/src/semidilute_bs/diffcalc_mod.f90:2384: more undefined references to `cusparsedcsrmv_sethpm_' follow

pgacclnk: child process exit status 1: /usr/bin/ld

My code has been working well with previous versions of 2019 versions of PGI, and @MatColgrove previously helped me with a few questions in regards to my package development (like here).

The package is on GitHub and the makefile for different clusters and Bridges2 is available.

I would very much appreciate it if you have any thoughts on where I might need to make any changes. I navigated through the cusparse.mod and can see that the functions like cusparseDcsr2csc still exist.

Thank you,
Amir

Hi Amir,

Sorry but the cuSparse routines you’re using were deprecated in CUDA 10.1 and have since been removed. See: https://docs.nvidia.com/cuda/archive/10.1/pdf/CUSPARSE_Library.pdf

cusparseDcsr2csc is shown as deprecated in Sec. 13.11 page 261 and cusparsedcsrmv is section 8.4 page 70.

-Mat

1 Like

Thanks Mat,

I realized that someone reported this a while back here. My apologies for the duplicated question.

Thanks for the reply. I was guessing that they are deprecated, however, it is interesting that the function names still show up in the module file cusparse.mod and also is mentioned in some documentation pages.

So, the user is supposed to replace cusparseDcsr2csc with cusparseCsr2cscEx2, and cusparsedcsrmv with cusparsedcsrmvEx? (given some additional arguments are required), And cusparseCreateSolveAnalysisInfo is no longer required for initializing sparse operations?

Thank you,
Amir

Hi Amir,

Yes, the routines were deprecated for a few CUDA releases, and now have been removed. I will update our Fortran interfaces document accordingly. The Fortran document is not meant to be a definitive guide, but a supplement to the regular cuSparse (and other CUDA library) documents. Also, we support many CUDA versions with the same doc. Still, now that 10.2 is our lowest release, those interfaces should now be deleted from the docs and from the Fortran module.

cusparseDcsrmv has been replaced by the generic cusparseSpMV. And you are correct, cusparseDcsr2csc is now replaced with cusparseCsr2cscEx2. The generic functions take a data type argument. I believe our package examples under CUDA-Libraries have some examples using the newer cuSolver API.

1 Like

Thanks @bleback

In addition to the deprecated functions being mentioned in the interface documentation, I don’t see the new functions in the documentation (cusparseCsr2cscEx2 and cusparseCsr2cscEx2_buffersize). In fact, I get the following errors when trying to compile:

NVFORTRAN-S-0038-Symbol, cusparsecsr2cscex2_buffersize, has not been explicitly declared (cuda/conv_cumod.cuf)

NVFORTRAN-S-0038-Symbol, cusparsecsr2cscex2, has not been explicitly declared (cuda/conv_cumod.cuf)

Am I missing anything here?

Thank you,
Amir

I think that is an error on our part. I will take a closer look.

1 Like

Yes, these are inadvertently missing. If you need the interface source to these, let me know. Otherwise, I will get these into an upcoming release.

1 Like

Thanks @bleback ,

I am familiar with how to add a wrapper for a CUDA C library function, however, I’d like to take your version if you have it already.

Thank you,
Amir

I wrote these last night. They compile, but I haven’t had time to test them yet. You might need to use the current cusparse module for the types, “use cusparse” somewhere either in the declarations or surrounding module, wherever you put these.

! cusparseCsr2cscEx2
interface
integer(c_int) function cusparseCsr2cscEx2_bufferSize(handle, m, n, nnz, csrVal, csrRowPtr, csrColInd, &
cscVal, cscColPtr, cscRowInd, valType, copyValues, idxBase, alg, bufferSize) &
bind(C,name=‘cusparseCsr2cscEx2’)
use iso_c_binding
import cusparseHandle
type(cusparseHandle), value :: handle
integer(c_int), value :: m, n, nnz, valType, copyValues, idxBase, alg
!pgi$ ignore_tkr csrVal, cscVal
real(4), device :: csrVal(), cscVal()
integer(c_int), device :: csrRowPtr(), csrColInd(), cscRowInd(), cscColPtr()
integer(8) :: bufferSize
end function cusparseCsr2cscEx2_bufferSize
end interface

interface
integer(c_int) function cusparseCsr2cscEx2(handle, m, n, nnz, csrVal, csrRowPtr, csrColInd, &
cscVal, cscColPtr, cscRowInd, valType, copyValues, idxBase, alg, buffer) &
bind(C,name=‘cusparseCsr2cscEx2’)
use iso_c_binding
import cusparseHandle
type(cusparseHandle), value :: handle
integer(c_int), value :: m, n, nnz, valType, copyValues, idxBase, alg
!pgi$ ignore_tkr csrVal, cscVal
real(4), device :: csrVal(), cscVal()
integer(c_int), device :: csrRowPtr(), csrColInd(), cscRowInd(), cscColPtr()
!pgi$ ignore_tkr buffer
character, device :: buffer(*)
end function cusparseCsr2cscEx2
end interface