we’re migrating a project to the Jetson Orin Nano and struggle with getting hardware acceleration for ONNX inference in Python. I’ve been searching around and found that I have to install the onnxruntime-gpu package, but this is not available on PyPI for aarch64. Searching the web, I found a similar/same thread here: Onnx runtime GPU
The answer is to just download the package from https://pypi.jetson-ai-lab.dev/ but something appears to be broken with this site:
$ curl https://pypi.jetson-ai-lab.dev/jp6/cu126/
curl: (7) Failed to connect to pypi.jetson-ai-lab.dev port 443 after 1326 ms: No route to host
$ ping pypi.jetson-ai-lab.dev
PING pypi.jetson-ai-lab.dev (108.32.95.36) 56(84) bytes of data.
64 bytes from pool-108-32-95-36.pitbpa.fios.verizon.net (108.32.95.36): icmp_seq=1 ttl=45 time=106 ms
64 bytes from pool-108-32-95-36.pitbpa.fios.verizon.net (108.32.95.36): icmp_seq=2 ttl=45 time=106 ms
64 bytes from pool-108-32-95-36.pitbpa.fios.verizon.net (108.32.95.36): icmp_seq=3 ttl=45 time=106 ms
64 bytes from pool-108-32-95-36.pitbpa.fios.verizon.net (108.32.95.36): icmp_seq=4 ttl=45 time=105 ms
^C
--- pypi.jetson-ai-lab.dev ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 104.785/105.548/105.905/0.448 ms
Am I just unlucky and the site is temporarily down, or is there now another way to get onnxruntime-gpu on the Jetson Orin Nano with the latest JetPack 6.2?
I also tried building it myself, but this appears not be so easy either:
Ubuntu 22.04’s CMake 3.22 is not supported, so I had to manually install a newer version (3.31 in my case) → relatively straightforward
The libeigen hash changed due to an imprecise download URL (points to a branch that changes, instead of a commit/tag that is consistent), see also https://github.com/microsoft/onnxruntime/issues/18286 → Fixed by updating the expected hash in cmake/deps.txt
Next was Could not find CUDNN_INCLUDE_DIR using the following files: cudnn.h, but how to install it? → That’s the point where I’m now; I’ve expected all those CUDA libraries to be installed during the initial setup of the board using SDK Manager, but something still seems to be missing. Isn’t it compatible with the Python 3.12 that I’m using? Or some other issue?
In the end, it boils down to the question: How are we supposed to get GPU acceleration for ONNX inference on a Jetson Orin Nano in 2025 with the current components as of today?
This means that Python 3.10 must be used, which is the version preinstalled on the Ubuntu 22.04 system, so I guess this makes sense. Anyway, no way to use a newer Python release then.
Not directly related to this topic, but in case somebody wonders: To get GPU acceleration for TensorFlow on the Jetson, the image must be downloaded from another URL:
It is listed as JetPack v61, but appears to work fine on 6.2. However, it caused a downgrade of the ml_dtypes package, which in turn causes an incompatibility with the onnxruntime. I’ve “fixed” that by force-upgrading ml_dtypes again using
pip install ml_dtypes==0.5.3
This works for me, but might cause a failure for others, depending on what features of TensorFlow are actually used. Well, most people would probably not use ONNX and TensorFlow in parallel, but I still wanted to mention it.