Libcublas.so.11 not found when working with PyTorch

edgarvardanyan1999 · February 10, 2023, 3:01pm

Hi, I have installed pytorch on my Xavier NX from https://developer.download.nvidia.cn/compute/redist/jp/v51/pytorch/torch-1.14.0a0+44dac51c.nv23.01-cp38-cp38-linux_aarch64.whl. However, when I import torch, I receive the following exception

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/cascade-st/venv/lib/python3.8/site-packages/torch/__init__.py", line 192, in <module>
    _load_global_deps()
  File "/data/cascade-st/venv/lib/python3.8/site-packages/torch/__init__.py", line 154, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/home/dev/.pyenv/versions/3.8.15/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libcublas.so.11: cannot open shared object file: No such file or directory

When I try to find the version of Jetpack with apt-cache show I find the following.

Package: nvidia-jetpack
Version: 4.6.3-b17
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-l4t-jetson-multimedia-api (>> 32.7-0), nvidia-l4t-jetson-multimedia-api (<< 32.8-0), nvidia-cuda (= 4.6.3-b17), nvidia-tensorrt (= 4.6.3-b17), nvidia-nsight-sys (= 4.6.3-b17), nvidia-cudnn8 (= 4.6.3-b17), nvidia-opencv (= 4.6.3-b17), nvidia-container (= 4.6.3-b17), nvidia-vpi (= 4.6.3-b17)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_4.6.3-b17_arm64.deb
Size: 29376
SHA256: bc486787ebc35228cd51b647f581e52f53e6238c0444a7654d3dcd43c6c8f6aa
SHA1: 5db526f319cae2dca64d7722cdb72bc9eaccb6c1
MD5sum: 5e44667d6e5b9d6d7ea8f0830fb83d66
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8

Package: nvidia-jetpack
Version: 4.6.2-b5
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-cuda (= 4.6.2-b5), nvidia-opencv (= 4.6.2-b5), nvidia-cudnn8 (= 4.6.2-b5), nvidia-tensorrt (= 4.6.2-b5), nvidia-visionworks (= 4.6.2-b5), nvidia-container (= 4.6.2-b5), nvidia-vpi (= 4.6.2-b5), nvidia-l4t-jetson-multimedia-api (>> 32.7-0), nvidia-l4t-jetson-multimedia-api (<< 32.8-0)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_4.6.2-b5_arm64.deb
Size: 29358
SHA256: 3b173ce179df79348fc68254a1fa4b7c9fc1d65ff2a065de7bf70459223ac39f
SHA1: eb5f9357dffa31c6945b39d6d28f7ef31b10fda2
MD5sum: 93001822fdf68db3a47cc832a17cbce8
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8

Package: nvidia-jetpack
Version: 4.6.1-b110
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-cuda (= 4.6.1-b110), nvidia-opencv (= 4.6.1-b110), nvidia-cudnn8 (= 4.6.1-b110), nvidia-tensorrt (= 4.6.1-b110), nvidia-visionworks (= 4.6.1-b110), nvidia-container (= 4.6.1-b110), nvidia-vpi (= 4.6.1-b110), nvidia-l4t-jetson-multimedia-api (>> 32.7-0), nvidia-l4t-jetson-multimedia-api (<< 32.8-0)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_4.6.1-b110_arm64.deb
Size: 29374
SHA256: 312e75d89d0837472668714c43590fd204a33b9c7727c542e1b5df367ec239c4
SHA1: 769cbf1c7acb9e39a4539879884a570bde12292c
MD5sum: a074f5225dfe8d7ea24560b15eef08d9
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8

Can you please help mi with the reason I am seeing multiple Jetpack versions and how can I solve the issue with Pytorch.

dusty_nv · February 10, 2023, 3:36pm

Hi @edgarvardanyan1999, it appears that you’ve installed a PyTorch wheel build for JetPack 5.1 when you are running JetPack 4.6. Please uninstall the previous wheel with pip3 uninstall torch and then try installing this wheel for JetPack 4.6 instead:

https://developer.download.nvidia.cn/compute/redist/jp/v461/pytorch/torch-1.11.0a0+17540c5+nv22.01-cp36-cp36m-linux_aarch64.whl

Alternatively, you could re-flash or upgrade your Xavier to JetPack 5.x

edgarvardanyan1999 · February 11, 2023, 5:54pm

Thanks a lot for the response. However, I was unable to install the wheel that you specified.

dusty_nv · February 11, 2023, 11:56pm

Can you run python3 --version ? It should be Python 3.6 on JetPack 4.x to run these wheels

edgarvardanyan1999 · February 12, 2023, 11:56am

@dusty_nv I checked, it was actually python3.8. However, after I downgraded to python3.6 and Installed the wheel you sent, I got the same error I had at the beginning.

Is it possible that something is messed up with the Jetpack installation? I see 3 versions of jetpack when I check with apt-cache

Package: nvidia-jetpack
Version: 4.6.3-b17
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-l4t-jetson-multimedia-api (>> 32.7-0), nvidia-l4t-jetson-multimedia-api (<< 32.8-0), nvidia-cuda (= 4.6.3-b17), nvidia-tensorrt (= 4.6.3-b17), nvidia-nsight-sys (= 4.6.3-b17), nvidia-cudnn8 (= 4.6.3-b17), nvidia-opencv (= 4.6.3-b17), nvidia-container (= 4.6.3-b17), nvidia-vpi (= 4.6.3-b17)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_4.6.3-b17_arm64.deb
Size: 29376
SHA256: bc486787ebc35228cd51b647f581e52f53e6238c0444a7654d3dcd43c6c8f6aa
SHA1: 5db526f319cae2dca64d7722cdb72bc9eaccb6c1
MD5sum: 5e44667d6e5b9d6d7ea8f0830fb83d66
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8

Package: nvidia-jetpack
Version: 4.6.2-b5
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-cuda (= 4.6.2-b5), nvidia-opencv (= 4.6.2-b5), nvidia-cudnn8 (= 4.6.2-b5), nvidia-tensorrt (= 4.6.2-b5), nvidia-visionworks (= 4.6.2-b5), nvidia-container (= 4.6.2-b5), nvidia-vpi (= 4.6.2-b5), nvidia-l4t-jetson-multimedia-api (>> 32.7-0), nvidia-l4t-jetson-multimedia-api (<< 32.8-0)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_4.6.2-b5_arm64.deb
Size: 29358
SHA256: 3b173ce179df79348fc68254a1fa4b7c9fc1d65ff2a065de7bf70459223ac39f
SHA1: eb5f9357dffa31c6945b39d6d28f7ef31b10fda2
MD5sum: 93001822fdf68db3a47cc832a17cbce8
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8

Package: nvidia-jetpack
Version: 4.6.1-b110
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-cuda (= 4.6.1-b110), nvidia-opencv (= 4.6.1-b110), nvidia-cudnn8 (= 4.6.1-b110), nvidia-tensorrt (= 4.6.1-b110), nvidia-visionworks (= 4.6.1-b110), nvidia-container (= 4.6.1-b110), nvidia-vpi (= 4.6.1-b110), nvidia-l4t-jetson-multimedia-api (>> 32.7-0), nvidia-l4t-jetson-multimedia-api (<< 32.8-0)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_4.6.1-b110_arm64.deb
Size: 29374
SHA256: 312e75d89d0837472668714c43590fd204a33b9c7727c542e1b5df367ec239c4
SHA1: 769cbf1c7acb9e39a4539879884a570bde12292c
MD5sum: a074f5225dfe8d7ea24560b15eef08d9
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8

dusty_nv · February 12, 2023, 11:21pm

Hmm okay, can you check that it exists under /usr/local/cuda/lib64 ?

ls /usr/local/cuda/lib64
libcudadevrt.a                libculibos.a              libmetis_static.a        libnppidei.so            libnppist.so.10         libnvgraph.so.10.2.89
libcudart.so                  libcupti.so               libnppc.so               libnppidei.so.10         libnppist.so.10.2.1.89  libnvgraph_static.a
libcudart.so.10.2             libcupti.so.10.2          libnppc.so.10            libnppidei.so.10.2.1.89  libnppist_static.a      libnvperf_host.so
libcudart.so.10.2.89          libcupti.so.10.2.75       libnppc.so.10.2.1.89     libnppidei_static.a      libnppisu.so            libnvperf_target.so
libcudart_static.a            libcurand.so              libnppc_static.a         libnppif.so              libnppisu.so.10         libnvrtc-builtins.so
libcufft.so                   libcurand.so.10           libnppial.so             libnppif.so.10           libnppisu.so.10.2.1.89  libnvrtc-builtins.so.10.2
libcufft.so.10                libcurand.so.10.1.2.89    libnppial.so.10          libnppif.so.10.2.1.89    libnppisu_static.a      libnvrtc-builtins.so.10.2.89
libcufft.so.10.1.2.89         libcurand_static.a        libnppial.so.10.2.1.89   libnppif_static.a        libnppitc.so            libnvrtc.so
libcufft_static.a             libcusolver.so            libnppial_static.a       libnppig.so              libnppitc.so.10         libnvrtc.so.10.2
libcufft_static_nocallback.a  libcusolver.so.10         libnppicc.so             libnppig.so.10           libnppitc.so.10.2.1.89  libnvrtc.so.10.2.89
libcufftw.so                  libcusolver.so.10.3.0.89  libnppicc.so.10          libnppig.so.10.2.1.89    libnppitc_static.a      libnvToolsExt.so
libcufftw.so.10               libcusolver_static.a      libnppicc.so.10.2.1.89   libnppig_static.a        libnpps.so              libnvToolsExt.so.1
libcufftw.so.10.1.2.89        libcusparse.so            libnppicc_static.a       libnppim.so              libnpps.so.10           libnvToolsExt.so.1.0.0
libcufftw_static.a            libcusparse.so.10         libnppicom.so            libnppim.so.10           libnpps.so.10.2.1.89    stubs
libcuinj64.so                 libcusparse.so.10.3.1.89  libnppicom.so.10         libnppim.so.10.2.1.89    libnpps_static.a
libcuinj64.so.10.2            libcusparse_static.a      libnppicom.so.10.2.1.89  libnppim_static.a        libnvgraph.so
libcuinj64.so.10.2.89         liblapack_static.a        libnppicom_static.a      libnppist.so             libnvgraph.so.10

If so, you may need to add /usr/local/cuda/lib64 to your LD_LIBRARY_PATH if PyTorch still isn’t finding it.

edgarvardanyan1999 · February 13, 2023, 12:02pm

Hmm, cuda is not found in /usr/local

dusty_nv · February 13, 2023, 6:58pm

Are you running that in a container, or on the device itself?

It doesn’t appear that JetPack was installed when you flashed the device - you could try installing it with sudo apt-get install nvidia-jetpack

system · March 8, 2023, 12:48am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.