Proper CUDA library for huge Hermitian eigenvalue tasks

Accelerated Computing GPU-Accelerated Libraries
1

I am trying to update some fortran code from using CPU to using GPU. The task is to diagonalize a massive (10^6x10^6) sparse, complex, Hermitian matrix. Since I’m working with a Tesla K80 CUDA seems the best choice. Of the libraries that are listed on nVidia’s boast page:

  • cuSolver has sparse functions but the only sparse eigenvalue function is a shift-inverse algorithm which can only find the nearest eigenvalue to an initial guess.
  • CULA doesn't seem to have a sparse eigenvalue solver.
  • cuSparse doesn't seem to have a sparse eigenvalue solver.
  • ArrayFire doesn't seem to have a sparse eigenvalue solver.
  • MAGMA has a symmetric/Hermitian sparse eigenvalue solver but would require me to build an interface.

So it seems that MAGMA is my only choice but it requires some interfacing to work with fortran. Also, I’m not entirely convinced the standard CUDA toolkit (which I’ve installed) comes with MAGMA so I’ll have to install it.

Am I basically right in my characterizations?

2

I am curious as to which library you are currently using in your host code for this particular task.

3

I’ve been using cudaFortran for moving stuff between host and device as well as the general stuff. So far I’ve experimented with cuSolver but I underestimated (or rather, neglected to estimate at all) the memory requirement. Luckily I haven’t done much so starting over with a different library is fine.

4

So if I understand correctly, your computational task is so specialized that there are few, if any, host libraries available for it. If so, it stands to reason that the same level of scarcity would apply to corresponding GPU libraries and MAGMA is your best bet.

5

That’s basically my thinking - I was holding out hope for something hidden (maybe filed under ‘Shibboleth’). Do you happen to know of any good documents on constructing interfaces in Fortran?

6

I don’t have any personal experience with the ISO_C_BINDING facility of modern versions of Fortran. All interfacing between C and Fortran code I have done used knowledge of the calling conventions of compilers in the same family, e.g. gcc plus g77 or gfortran, or the Intel C/C++ and Fortran compilers.

I assume that use of ISO_C_BINDING requires a C-style interface, as the name implies. Since CUDA is in the C++ family, you may need to use ‘extern “C”’ wrappers on the CUDA side.

You mentioned that you are using CUDA Fortran. Surely the documentation for that should have a section that explains how to properly invoke third-party CUDA code. Have you checked the docs?

7

Yeah I’ve checked the docs, they have decent coverage but are mainly aimed at programmers (understandably). I am more at the “For Dummies” level but I can figure it out given enough time.

8

We all start out as “dummies” every time we start in a new knowledge area, then learn by reading in combination with doing (me personally: mostly doing). So my advice would be to get in there and get your hands dirty, so to speak.

If you are a scientist, consider that these days, computing is an indispensable tool for just about any kind science. And computing involves programming as a hands-on activity, just like an archaeologist will need to learn how to handle shovels, trowels, and brushes expertly.