TensorRT was linked against cuBLAS/cuBLAS LT 10.2.3 but loaded cuBLAS/cuBLAS LT 10.2.2

Accelerated Computing CUDA CUDA Setup and Installation
1

Description

I’m trying to build docker container with tensorrt. But when run my code there is the error:

 [TRT] [E] 2: [ltWrapper.cpp::setupHeuristic::327] Error Code 2: Internal Error (Assertion cublasStatus == CUBLAS_STATUS_SUCCESS failed.

As I understand there issue is due to my tensorrt version 8.2.1.8 is linked to cublas version 10.2.3. but it is installed v. 10.2.2.
As I work in container I start from this image:

FROM nvidia/cuda:10.2-cudnn8-devel-ubuntu18.04

were cuda 10.2 is installed.
As I understand I should update version of libcublas but it is usually installed with cuda.
Is there any chance to fix it?

I have read the installation guide provided here:

And I choose to install from tar package.

Another suggestion to use ngc containers

But the container with tensorrt 8.2.1. 22.01-py3 is linked to cuda 11.6 but not cuda 10.2 and I can not switch the version of cuda.

Environment

TensorRT Version: 8.2.1.8
GPU Type: RTX 2080 TI
Nvidia Driver Version: 470.103.01
CUDA Version: 10.2
CUDNN Version: 8.2
Operating System + Version: Ubuntu 18.04
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

2

I am facing the exact same issue and I’ve got the exact same environment as you. Have you find any solution yet?

3

Unfortunately no(

4

Hi,

As the support matrix doc indicates, you need to update the cuBlas version. Or you may need to use a lower version of the TensorRT to match your environment.

Thank you.

5

Thanks, I understand that I need to upgrade cublas, but it’s not clear how to do this in docker. I can not find smth like apt install clibcublas or so on. As I understand cublas is installed along with cuda. What is the easiest way to upgrade ?

6

We are moving this post to a CUDA setup related forum to get better help on updating CuBlas if possible.

7

Hi, any suggestions?

8

I also faced the same error. I don’t use docker but still couldn’t find how to upgrade cublas version.

9

After installing cuda 10.2 patch 1 and patch 2, I had cublas 10.2.3. I think only patch 2 contains cublas 10.2.3, patch 1 includes only cublas 10.2.2.214.
After patch 2 libcublas.so.10.2.3.254 and other .so files are installed.
But so far I haven’t found any official cuda image containing 10.2 with patch 1 and patch 2, and I don’t know if you can install a runfile inside docker.

10

As a workaround for docker I managed to upgrade cublas like this:

RUN apt-get update && apt-get install -y --allow-change-held-packages libcublas10=10.2.3.254-1
RUN cd /usr/lib/x86_64-linux-gnu && \
mv /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcublas.so.10 . && \
mv /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcublas.so.10.2.3.254 . && \
mv /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcublasLt.so.10 . && \
mv /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcublasLt.so.10.2.3.254 . && \
mv /usr/local/cuda-10.2/targets/x86_64-linux/lib/libnvblas.so.10 . && \
mv /usr/local/cuda-10.2/targets/x86_64-linux/lib/libnvblas.so.10.2.3.254 .

This is not pretty, and I think nvidia should instead update their images to contain the patches for a proper solution. Shouldn’t be too much of an effort on their part. Please? :)

1 Like