Orin Pytorch/CUDA issue

s.k · August 11, 2023, 7:22am

We have trouble getting Pytorch to work on Orin. We have Jetpack 5.1.1-b56 installed, and downloaded Pytorch 2.0.0 from this website: PyTorch for Jetson

The output of the installation:

pip3 install numpy torch-2.0.0+nv23.05-cp38-cp38-linux_aarch64.whl 
Defaulting to user installation because normal site-packages is not writeable
Processing ./torch-2.0.0+nv23.05-cp38-cp38-linux_aarch64.whl
Requirement already satisfied: numpy in /usr/lib/python3/dist-packages (1.17.4)
Requirement already satisfied: filelock in /home/nvidia/.local/lib/python3.8/site-packages (from torch==2.0.0+nv23.05) (3.12.2)
Requirement already satisfied: jinja2 in /home/nvidia/.local/lib/python3.8/site-packages (from torch==2.0.0+nv23.05) (3.1.2)
Requirement already satisfied: networkx in /home/nvidia/.local/lib/python3.8/site-packages (from torch==2.0.0+nv23.05) (3.1)
Requirement already satisfied: sympy in /home/nvidia/.local/lib/python3.8/site-packages (from torch==2.0.0+nv23.05) (1.12)
Requirement already satisfied: typing-extensions in /home/nvidia/.local/lib/python3.8/site-packages (from torch==2.0.0+nv23.05) (4.7.1)
Requirement already satisfied: MarkupSafe>=2.0 in /home/nvidia/.local/lib/python3.8/site-packages (from jinja2->torch==2.0.0+nv23.05) (2.1.3)
Requirement already satisfied: mpmath>=0.19 in /home/nvidia/.local/lib/python3.8/site-packages (from sympy->torch==2.0.0+nv23.05) (1.3.0)
DEPRECATION: distro-info 0.23ubuntu1 has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of distro-info or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
DEPRECATION: python-debian 0.1.36ubuntu1 has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of python-debian or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
Installing collected packages: torch
Successfully installed torch-2.0.0+nv23.5

When we type this in the terminal: Python3 → import torch, the following error message is shown:

>>> import torch
Traceback (most recent call last):
  File "/home/nvidia/.local/lib/python3.8/site-packages/torch/__init__.py", line 168, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/usr/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libcurand.so.10: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/nvidia/.local/lib/python3.8/site-packages/torch/__init__.py", line 228, in <module>
    _load_global_deps()
  File "/home/nvidia/.local/lib/python3.8/site-packages/torch/__init__.py", line 189, in _load_global_deps
    _preload_cuda_deps(lib_folder, lib_name)
  File "/home/nvidia/.local/lib/python3.8/site-packages/torch/__init__.py", line 154, in _preload_cuda_deps
    raise ValueError(f"{lib_name} not found in the system path {sys.path}")
ValueError: libcublas.so.*[0-9] not found in the system path ['', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/nvidia/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.8/dist-packages']

We believe this might have something to due with conflicting/missing CUDA versions. We currently have CUDA 11.4 installed, but not 10.

AastaLLL · August 11, 2023, 7:58am

Hi,

It looks like some CUDA libraries are missing in your environment.
Do you also install the component from JetPack 5.1.1?

Thanks.

s.k · August 11, 2023, 8:37am

Thanks for the quick answer. We already have CUDA 11.4 installed.

Can you please clarify what we are missing?
Could it perhaps be related to an environment variable?

dusty_nv · August 11, 2023, 2:11pm

@s.k can you confirm that you have these libraries installed in the CUDA Toolkit? they should have been installed by SDK Manager when you flashed the board

ls -ll /usr/local/cuda/lib64/libcurand*
lrwxrwxrwx 1 root root       15 Sep 14  2022 /usr/local/cuda/lib64/libcurand.so -> libcurand.so.10
lrwxrwxrwx 1 root root       23 Sep 14  2022 /usr/local/cuda/lib64/libcurand.so.10 -> libcurand.so.10.2.5.297
-rw-r--r-- 1 root root 77999248 Sep 14  2022 /usr/local/cuda/lib64/libcurand.so.10.2.5.297
-rw-r--r-- 1 root root 77968142 Sep 14  2022 /usr/local/cuda/lib64/libcurand_static.a

You could also try the l4t-pytorch container which comes with it all pre-installed for you:

s.k · August 11, 2023, 3:37pm

This is the output of ls --ll in that directory:

nvidia@orin:/usr/local/cuda/lib64$ ls -ll
total 2672
-rw-r--r-- 1 root root  801732 sep.  14  2022 libcudadevrt.a
lrwxrwxrwx 1 root root      17 sep.  14  2022 libcudart.so -> libcudart.so.11.0
lrwxrwxrwx 1 root root      21 sep.  14  2022 libcudart.so.11.0 -> libcudart.so.11.4.298
-rw-r--r-- 1 root root  699488 sep.  14  2022 libcudart.so.11.4.298
-rw-r--r-- 1 root root 1189054 sep.  14  2022 libcudart_static.a
-rw-r--r-- 1 root root   36394 sep.  14  2022 libculibos.a
drwxr-xr-x 2 root root    4096 aug.   2 13:13 stubs

We will try to do a fresh install of the CUDA components with the SDK manager. If it doesn’t work we will try the container. Thanks a lot!

dusty_nv · August 11, 2023, 5:53pm

@s.k you could also try sudo apt-get install cuda-toolkit-11-4

s.k · August 14, 2023, 12:52pm

Problem resolved by installing the necessary CUDA components from the SDK manager. Thanks!

hieprooney16 · August 17, 2023, 2:28am

Help me this .
I also face missing cuda problem after install jetpack 5.1 on Nvidia Jetson Orin NX.

I have tried install cuda on jetson with this commands :

sudo apt-get install nvidia-cuda
But it does not work. I also tried with the above command you suggested but i get the error :

image727×131 14.3 KB

Please help me solve this problem .
Thanks you !

dusty_nv · August 17, 2023, 2:56am

@hieprooney16 can you do cat /etc/apt/sources.list.d/nvidia-l4t-apt-source.list ? Did you do sudo apt-get update before?

When you flashed the board, did you have SDK Manager install CUDA/cuDNN/TensorRT? You can run SDK Manager again and just de-select the flashing part and it will just install CUDA/ect on your Jetson for you.

hieprooney16 · August 17, 2023, 4:30am

I install Jetpack base on this link : JETSON-ORIN-NX-16G-DEV-KIT - Waveshare Wiki .
And i do as these 4 steps below on the image

Do i need to run this command after 4 steps above :

Thanks

dusty_nv · August 17, 2023, 1:13pm

Yes, I’m not entirely sure what state that Waveshare’s flashing sequence puts the board in, but presumably sudo apt-get update && sudo apt-get install nvidia-jetpack would install CUDA/ect for you

system · September 11, 2023, 7:51am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
PyTorch for Jetson Orin Jetpack 5.1 installation issue Jetson AGX Orin pytorch	12	903	January 23, 2024
Cannot install Pytorch 2.x with CUDA support Jetson Orin Nano cuda , pytorch	9	114	September 2, 2024
Cannot get Torch met CUDA to work on Jetson Orin Jetson Orin NX pytorch	10	1393	February 26, 2024
Jetson orin nano Cuda Cudnn torch torchauido torchvision Jetson Orin Nano cuda	11	643	May 27, 2024
Issue to install pytorch on Jetson Orin platform Jetson Orin NX jetson-inference , pytorch	10	4500	June 7, 2023
Pytorch for jetson nano orin does not work Jetson Orin Nano pytorch	5	1291	August 22, 2023
Pytorch installed on l4t-jetpack:r35.4.1 container on Jetson Orin Nano (JetPack 6.0 Developer Kit) fails to recognize CUDA Jetson Orin Nano cuda , docker , pytorch , python , containers	2	45	October 22, 2024
Error flashing Jetson Orin including Cuda Jetson AGX Orin reflash , cuda	9	1094	July 11, 2022
Error While Running in Jetson Orin NX board Jetson Orin NX	31	331	June 21, 2024
Install Pytorch with cuda on Jetson Orin nano Devloper Kit Jetson Orin Nano cuda , pytorch	13	1146	July 30, 2024

Orin Pytorch/CUDA issue

Related Topics