ImportError: libcublas.so.10 on nvidia docker xavier nx

[fischer4]

Hello, we have some xavier nx (carrier board from auvidia) in use. We successfully used nvidia docker. One of the xavier nx has a problem with an ImportError: libcublas.so.10

Package: nvidia-jetpack
Version: 4.6-b197
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-cuda (= 4.6-b197), nvidia-opencv (= 4.6-b197), nvidia-cudnn8 (= 4.6-b197), nvidia-tensorrt (= 4.6-b197), nvidia-visionworks (= 4.6-b197), nvidia-container (= 4.6-b197), nvidia-vpi (= 4.6-b197), nvidia-l4t-jetson-multimedia-api (>> 32.6-0), nvidia-l4t-jetson-multimedia-api (<< 32.7-0)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_4.6-b197_arm64.deb

cat /etc/docker/daemon.json

{
    "runtimes": {
        "nvidia": {
            "path": { "nvidia-container-runtime",
            "runtimeArgs": []
        }
    },

    "default-runtime": "nvidia"
}

baseimage: nvcr.io/nvidia/l4t-ml:r32.6.1-py3

inside the container:

root@34a549ff0f0b:/app# find / -name "libcublas*"
/usr/local/cuda-10.2/targets/aarch64-linux/lib/stubs/libcublasLt.so
/usr/local/cuda-10.2/targets/aarch64-linux/lib/stubs/libcublas.so

and then with python3

root@cc4428d9191d:/app# python3
Python 3.6.9 (default, Jan 26 2021, 15:33:00)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/cv2/__init__.py", line 96, in <module>
    bootstrap()
  File "/usr/local/lib/python3.6/dist-packages/cv2/__init__.py", line 86, in bootstrap
    import cv2
ImportError: libcublas.so.10: cannot open shared object file: No such file or directory

any idea how to fix this ?
Best regards

[AastaLLL]

Hi,

Could you check cuBLAS outside of the container as well?

$ sudo find / -name "libcublas*"

Thanks.

[fischer4]

yes i do

sudo find / -name “libcublas*”

find: ‘/proc/3429/task/3429/net’: Invalid argument
find: ‘/proc/3429/net’: Invalid argument
/usr/share/doc/libcublas-dev
/usr/share/doc/libcublas10
/usr/local/cuda-10.2/targets/aarch64-linux/lib/libcublas.so.10.2.3.300
/usr/local/cuda-10.2/targets/aarch64-linux/lib/libcublas_static.a
/usr/local/cuda-10.2/targets/aarch64-linux/lib/stubs/libcublasLt.so
/usr/local/cuda-10.2/targets/aarch64-linux/lib/stubs/libcublas.so
/usr/local/cuda-10.2/targets/aarch64-linux/lib/libcublasLt.so
/usr/local/cuda-10.2/targets/aarch64-linux/lib/libcublas.so
/usr/local/cuda-10.2/targets/aarch64-linux/lib/libcublasLt_static.a
/usr/local/cuda-10.2/targets/aarch64-linux/lib/libcublas.so.10
/usr/local/cuda-10.2/targets/aarch64-linux/lib/libcublasLt.so.10
/usr/local/cuda-10.2/targets/aarch64-linux/lib/libcublasLt.so.10.2.3.300
/usr/local/cuda-10.2/doc/man/man7/libcublas.so.7
/usr/local/cuda-10.2/doc/man/man7/libcublas.7
    /var/lib/docker/overlay2/d01d44d7fb92dd69219150f71088a83489cc019c6e72c442d8dd64f6241cb978/diff/usr/local/cuda-10.2/targets/aarch64-linux/lib/stubs/libcublasLt.so
    /var/lib/docker/overlay2/d01d44d7fb92dd69219150f71088a83489cc019c6e72c442d8dd64f6241cb978/diff/usr/local/cuda-10.2/targets/aarch64-linux/lib/stubs/libcublas.so
    /var/lib/docker/overlay2/56470aa2ae7ec698de061213449c4b961b6a820a52d39504a9c91a7f67ee1e24/merged/usr/local/cuda-10.2/targets/aarch64-linux/lib/stubs/libcublasLt.so
    /var/lib/docker/overlay2/56470aa2ae7ec698de061213449c4b961b6a820a52d39504a9c91a7f67ee1e24/merged/usr/local/cuda-10.2/targets/aarch64-linux/lib/stubs/libcublas.so
    /var/lib/dpkg/info/libcublas-dev.md5sums
    /var/lib/dpkg/info/libcublas10.list
    /var/lib/dpkg/info/libcublas-dev.list
    /var/lib/dpkg/info/libcublas10.md5sums

[AastaLLL]

Hi,

We test the l4t-ml:r32.7.1-py3 container on a JetPack 4.6.2 environment and it works well.
Could you also give it a try?

$ sudo docker run -it --rm --runtime nvidia --network host nvcr.io/nvidia/l4t-ml:r32.7.1-py3
allow 10 sec for JupyterLab to start @ http://10.19.107.79:8888 (password nvidia)
JupterLab logging location:  /var/log/jupyter.log  (inside the container)
root@nvidia-desktop:/# python3
Python 3.6.9 (default, Dec  8 2021, 21:08:43)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> cv2.__version__
'4.5.0'
>>>

Thanks.