CUDA Blas libraries not installed?

I have installed drivers, toolkit, etc for Cuda_10.1.243_418.87.00.
All seemed to go according to plan. However, the cublas libs are
not found. I believe they are supposed to be here:

/usr/local/cuda-10.0/lib64/libcublas.so

Aren’t those installed automatically when the Toolkit is installed?

CMake-Gui (for building OpenCV 4) seems to find all other CUDA libs,
and there are other libs in the normal CUDA folder:

/usr/local/cuda-10.0/lib64/

In fact, OpenCV builds fine, even with the WITH_CUDA flag turned on,
as long as WITH_CUBLAS is turned off. Other libraries, are auto-detected
(such as CUDA_cufft_LIBRARY /usr/local/cuda-10.1/lib64/libcufft.so)

However, there are two CUBLAS libs that are not auto-detected, incl:
CUDA_cublas_LIBRARY-CUDA, and_cublas_device_LIBRARY-NOTFOUND

As a result, enabling the WITH_CUBLAS flag triggers a cascade of
errors.

Given past experience with tricky CUDA installs, I would like to make
sure of the correct method for resolving the CUBLAS problems.

not anymore. CUDA 10.1 changed the location of the cublas libraries, specifically. The cublas libraries are now installed in a “system standard” library location. For example on my RHEL/CentOS setup they are in /usr/lib64.

https://docs.nvidia.com/cuda/archive/10.1/cuda-toolkit-release-notes/index.html#cublas-new-features

https://devtalk.nvidia.com/default/topic/1047981/cuda-setup-and-installation/cublas-for-10-1-is-missing/

Oh, great. Cmake apparently needs to be updated then too. It appears to have found all the other CUDA-related libraries except for CuBlas.

There are several libs in the /usr/lib/x86_64-linux-gnu folder, including “libcublas.so.10.2.1.243” and “libcublasLt.so.10.2.1.243”. I presume that the correct one is the former? (not the Lt version)

Also, do you know the correct name and location for “CUDA_cublas_device_LIBRARY” ?

Thanks, Robert.

You don’t indicate which version of CMake you are using. it might be updated already (I’ll bet it is.) This change to CUDA has been public for close to a year.

If you study the cublas documentation you’ll figure out what the Lt library is for. With respect to what is the “correct one” that depends on what you are doing. I really don’t know much about building OpenCV from scratch, but I imagine it may only need the libcublas library, because the libcublasLt library is “relatively new”. Really no guarantees that this couldn’t/wouldn’t change in the future. Not sure why it should matter anyway, if your question is about library search paths.

The cublas device library was deprecated some time ago and has not been available since CUDA 10. This was also covered in release notes and also various questions on these forums about it. There is no “correct name and location” for it in CUDA 10 or CUDA 10.1 or CUDA 10.2; it does not exist.

It basically sounds to me like you are using an old CMake/OpenCV setup and attempting to use a recent version of CUDA with it. It might be easier to use recent version of CMake and OpenCV with recent vesion of CUDA, so as to pick up harmonious changes, and avoid a lot of hacking work on your own.

OK, I did try selecting by navigating to the files above. Attempt to rebuild OpenCV failed with the same error messages, and afterward, Cmake’s path to the CudaBlas lib was reset to Not-Found again.

Not sure why the device library entry appeared, since Cmake was only a couple months old, and I’m trying to build the latest version of OpenCV 4.

The Cmake version that I was running was the stock ‘apt-get’ version. Reported as latest, though the cmake.org site had a version that is a couple months newer. So I forced the update to the latest, v3.15.5. No change, except the entry for CUDA_cublas_device_LIBRARY has now disappeared. I was hoping that the new cmake version would automatically find the CuBlas lib, but it did not.

The cmake-gui menu is looking for a specific file to be selected. It will not accept a folder. So I tried each of the libcublas* files, one at a time. They all fail with the same message.

It seemed likely that the correct file was ‘libcublas.so.10.2.1.243’ (which also appears to be sym-linked by ‘libcublas.so.10’) but neither worked. Nor did ‘libcublasLt.so.10.2.1.243’

So everything is indeed the latest version, except CUDA may have had a minor rev since 10.2 (?)

Does the lib file listed above appear to be correct? If so, are there other possible causes for this? Kind of odd that Cmake resets the CuBlas entry.

After forcing updates to CMake and CMake-gui to the very newest version (3.15.5), adding the folder containing libcublas.10.so to the LD_LIBRARY path, and attempting complete rebuild, the problem remains exactly the same.

Robert, can you confirm that libcublas.10 is likely to be the correct library? Even after hand-selecting the exact path (/usr/lib/x86_64-linux-gnu/libcublas.10.so), the cmake-gui generate process appears to reject the library’s validity once the ‘generate’ button is pressed.

All other CUDA library locations are auto-detected correctly.

Before pressing Generate:
CUDA_cublas_LIBRARY /usr/lib/x86_64-linux-gnu/libcublas.10.so

After pressing Generate, Cmake-gui resets that entry to:
CUDA_cublas_LIBRARY CUDA_cublas_LIBRARY-NOTFOUND

I haven’t built OpenCV recently so really can’t help much at this point.

Perhaps it is a bug in CMake. CMake is not developed or supported by NVIDIA (and, speaking generally, neither is OpenCV)

Yeah, that seems to be the case at this point. I posted an inquiry on the CMake site, and of course they pointed to OpenCV as the source of the problem. Perhaps some of the detection algorithms are from the OpenCV crew, I don’t know. But the problem seems to straddle all three platforms somehow–OpenCV, CMake, and CUDA. Also got nothing from inquiries on the OpenCV forum, so I’m not sure how to pursue it.

Robert, can you confirm that libcublas.so.10.2.1.243 is the correct library?

It is the library that contains the bulk of the CUBLAS library code (other than the Lt api). I can’t confirm its correct for any particular use case, because I don’t know what you are trying to link exactly, and this is basically a linking question.

I can certainly write a CUBLAS code that links correctly against that library.

Thanks for confirming the library, Robert. It seemed like the obvious candidate for the OpenCV build, but I wanted to make sure. I’ve even tried hand-selecting that file in the CMake-gui build menu, but pressing the ‘configure’ button erases it and replaces the entry with --NOT-FOUND. I thought maybe there was a problem with the file itself.

I’ll keep looking, but I’ve got to get a project underway soon. For better or worse, I may have to build without the cublas library. Not sure how that will impact execution speed, but it may not be possible to solve this with current versions of CMake and OpenCV.

Robert, can you confirm that libcublas.so.10.2.1.243 is the correct library?

It is the library that contains the bulk of the CUBLAS library code

Well, it appears that that was not the correct name for the library file. Or at least it was not understood by CMake. Cmake appears to look for a library that ends with “.so”, so I created a symlink with the .so ending, and Cmake ran without complaints.

I guess the question now is… why does the toolkit install lib files with ‘.so’ in the -middle- of the file name? The regular installed symlink file was also misnamed to ‘libcublas.so.10’. All the other required CUDA files had the correct .so or .a extensions.

I would have guessed this right away, but I’m from Windows platform. I thought perhaps some latitude was afforded for file ‘extensions’ under Linux. Apparently not.

The usual linux linking target is indeed a .so file.

it will be something like libcublas.so

That libcublas.so will actually just be a symlink, however. It will symlink to a libcublas.so.10 and then that will symlink to libcublas.so.10.xxx where xxx is the full version. This file that ends in .xxxx is the actual library (you can tell by the size, and it is not a symlink).

Sorry for the confusion.

The naming scheme Robert Crovella describes for CUBLAS has not changed since CUBLAS first shipped in 2007. Tiered symlinks were a common way of how programmers dealt with versioned dynamic libraries on Linux back then (not that this has changed as far as I am aware).

That makes sense then. It appears that the libcublas.so symlink was not created, for some reason. The .so.10 symlink is there, so it would not have been a permissions problem. In fact, all other CUDA libs appear to be linked correctly.

CMake was not accepting any file that did not end in .so, even when ‘hard-coded’ to libcublas.so.10.

It seems very simple in retrospect, but that was time-consuming!

It looks like I’m not out of the woods yet. After creating the libcublas.so symlink, CMake completes the configure and generate phases. But now errors occur when running make to do the final build:

cublas.h and cublas_api.h are not found. This has to be related. Incredibly weird!

If you cannot update cmake:
from /usr/local/cuda-10.2/lib64
$ sudo ln -s -T /usr/lib/x86_64-linux-gnu/libcublas.so libcublas.so

$ cd /usr/local/cuda-10.2/lib64
$ sudo ln -s -T libcublas.so /usr/lib/x86_64-linux-gnu/libcublas_device.so

This can work in my side for CUDA_cublas_device_LIBRARY-NOTFOUND issue.

1 Like

I tried all the mentioned solution but none works for me . I have ubuntu 16.04 and cuda 8 and i installed cuda 10.1. I get the same error under kubuntu 20.04 with cuda 11.3 and cuda 8. what should I do?