TensorRT was linked against cuBLAS/cuBLAS LT 10.2.3 but loaded cuBLAS/cuBLAS LT 10.2.2

Description

I’m trying to build docker container with tensorrt. But when run my code there is the error:

 [TRT] [E] 2: [ltWrapper.cpp::setupHeuristic::327] Error Code 2: Internal Error (Assertion cublasStatus == CUBLAS_STATUS_SUCCESS failed. 

As I understand there issue is due to my tensorrt version 8.2.1.8 is linked to cublas version 10.2.3. but it is installed v. 10.2.2.
As I work in container I start from this image:

FROM nvidia/cuda:10.2-cudnn8-devel-ubuntu18.04

were cuda 10.2 is installed.
As I understand I should update version of libcublas but it is usually installed with cuda.
Is there any chance to fix it?

I have read the installation guide provided here:

And I choose to install from tar package.

Another suggestion to use ngc containers

But the container with tensorrt 8.2.1. 22.01-py3 is linked to cuda 11.6 but not cuda 10.2 and I can not switch the version of cuda.

Environment

TensorRT Version: 8.2.1.8
GPU Type: RTX 2080 TI
Nvidia Driver Version: 470.103.01
CUDA Version: 10.2
CUDNN Version: 8.2
Operating System + Version: Ubuntu 18.04
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

I am facing the exact same issue and I’ve got the exact same environment as you. Have you find any solution yet?

Unfortunately no(

Hi,

As the support matrix doc indicates, you need to update the cuBlas version. Or you may need to use a lower version of the TensorRT to match your environment.

Thank you.

Thanks, I understand that I need to upgrade cublas, but it’s not clear how to do this in docker. I can not find smth like apt install clibcublas or so on. As I understand cublas is installed along with cuda. What is the easiest way to upgrade ?

We are moving this post to a CUDA setup related forum to get better help on updating CuBlas if possible.

Hi, any suggestions?

I also faced the same error. I don’t use docker but still couldn’t find how to upgrade cublas version.

After installing cuda 10.2 patch 1 and patch 2, I had cublas 10.2.3. I think only patch 2 contains cublas 10.2.3, patch 1 includes only cublas 10.2.2.214.
After patch 2 libcublas.so.10.2.3.254 and other .so files are installed.
But so far I haven’t found any official cuda image containing 10.2 with patch 1 and patch 2, and I don’t know if you can install a runfile inside docker.

As a workaround for docker I managed to upgrade cublas like this:

RUN apt-get update && apt-get install -y --allow-change-held-packages libcublas10=10.2.3.254-1
RUN cd /usr/lib/x86_64-linux-gnu && \
mv /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcublas.so.10 . && \
mv /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcublas.so.10.2.3.254 . && \
mv /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcublasLt.so.10 . && \
mv /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcublasLt.so.10.2.3.254 . && \
mv /usr/local/cuda-10.2/targets/x86_64-linux/lib/libnvblas.so.10 . && \
mv /usr/local/cuda-10.2/targets/x86_64-linux/lib/libnvblas.so.10.2.3.254 .

This is not pretty, and I think nvidia should instead update their images to contain the patches for a proper solution. Shouldn’t be too much of an effort on their part. Please? :)