Jetson AGX Orin recent update has broken CUDNN

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
1

I noticed an update came though yesterday and CUDNN was updated on a reboot. Today I am getting “RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED” errors.

I am running Python 3.10.12, Ubuntu 22.04, CUDA 12.6, cuDNN 9.13.1, jetpack 6.2.1 Torch 2.8.0.

according to JTOP CUDA is CUDA 12.9.86.

Can anyone tell me what is incompatible and how to get back to a working system?

Thanks

2

*** Please note that this reply is generated by LLM automatically ***

The issue you’re experiencing is likely due to the cuDNN update that occurred during the JetPack update. The error message CUDNNSTATUSNOTINITIALIZED typically indicates that the cuDNN library is not properly initialized.

To resolve this issue, you’ll need to ensure that the cuDNN version is compatible with the CUDA version and the PyTorch version you’re using. Here’s a step-by-step guide to help you troubleshoot and potentially resolve the issue:

  1. Verify the cuDNN version: You’ve mentioned that cuDNN was updated to version 9.1.31. However, JTOP reports CUDA 12.9.86, which might indicate a mismatch. You can verify the cuDNN version by running the following command:
nvcc --version

This should display the CUDA version, which should match the cuDNN version.

  1. Check PyTorch compatibility: PyTorch 2.8.0 is compatible with CUDA 11.6 and cuDNN 8.2.4. However, since you’ve updated to CUDA 12.9.86 and cuDNN 9.1.31, you might need to update PyTorch to a version that supports these newer versions.

You can check the PyTorch compatibility matrix to determine the compatible version:

import torch
print(torch.__version__)
print(torch.version.cuda)
print(torch.version.cudnn)

This will display the PyTorch version, CUDA version, and cuDNN version used by PyTorch.

  1. Update PyTorch: If the PyTorch version is not compatible with the updated CUDA and cuDNN versions, you’ll need to update PyTorch. You can do this by running the following command:
pip install torch --upgrade

This will update PyTorch to the latest version that supports the updated CUDA and cuDNN versions.

  1. Verify the CUDA version: Although JTOP reports CUDA 12.9.86, you’ve mentioned that you’re using CUDA 12.6. Ensure that the CUDA version is correctly set by running:
nvcc --version

This should display the correct CUDA version.

  1. Reinstall cuDNN: If the issue persists, you can try reinstalling cuDNN to ensure that it’s properly initialized. You can do this by running:
sudo apt-get install --reinstall libcudnn8=8.2.4-1+cuda11.6

Replace the version numbers with the ones that match your CUDA and cuDNN versions.

After completing these steps, restart your system and try running your PyTorch application again. If the issue persists, you may want to consider reverting to a previous version of cuDNN or seeking further assistance from the PyTorch community or NVIDIA support.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

4

None of that AI answer was any use. Some of it was just plain wrong.

SolutionAn automatic upgrade had installed CUDA 13. There is no compatible version of Pytorch for CUDA 13 on the Jetson AGX Orin the solution was to downgrade to Cuda version 12.

6

Hi,

Could you try to reinstall the JetPack to see if it will downgrade automatically?

$ sudo apt install nvidia-jetpack*

Thanks.

7

Hi,

We want to test this issue internally.
Could you also share which command will lead to the upgrade?

Try to re-install cuDNN, but the latest available is 9.3.0:

$ sudo apt install cudnn --upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
cudnn is already the newest version (9.3.0-1).
...

Thanks.

8

Hi AastaLL,

I have got the system stable again by downgrading to CUDA 12.6. Somehow CUDA 13 had got installed. I don’t know how as I only noticed when I powered powered down by using the power menu shutdown command on the AGX Orin and I noticed it automatically restarted, updated to CUDA 13 and then shut down. I fixed it by uninstalling CUDA 13 and reinstalling CUDA 12.6

I don’t want to mess with it any more as I am in the middle of an exiting project and about to apply to the Nvidia Inception Program. But here is the stable diagnostics thats working OK now.

Package: nvidia-jetpack
Source: nvidia-jetpack (6.2.1)
Version: 6.2.1+b38
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-jetpack-runtime (= 6.2.1+b38), nvidia-jetpack-dev (= 6.2.1+b38)
Homepage: Jetson - Embedded AI Computing Platform | NVIDIA Developer
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_6.2.1+b38_arm64.deb
Size: 29300
SHA256: dd9cb893fbe7f80d2c2348b268f17c8140b18b9dbb674fa8d79facfaa2050c53
SHA1: dc630f213f9afcb6f67c65234df7ad5c019edb9c
MD5sum: 9c8dc61bdab2b816dcc7cd253bcf6482
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8

Package: nvidia-jetpack
Source: nvidia-jetpack (6.1)
Version: 6.1+b123
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-jetpack-runtime (= 6.1+b123), nvidia-jetpack-dev (= 6.1+b123)
Homepage: Jetson - Embedded AI Computing Platform | NVIDIA Developer
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_6.1+b123_arm64.deb
Size: 29312
SHA256: b6475a6108aeabc5b16af7c102162b7c46c36361239fef6293535d05ee2c2929
SHA1: f0984a6272c8f3a70ae14cb2ca6716b8c1a09543
MD5sum: a167745e1d88a8d7597454c8003fa9a4
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8

PyTorch: 2.8.0
CUDA: 12.6
cuDNN: 91301
CUDA available: True

9

Hi,

Good to know you get it working now.

Just for your reference.
The latest CUDA version for JetPack 6.2 is v12.9.

CUDA 13 is now released for r38 users only.
Thanks.

10

HI Aastalll, what happened was on about the 29th of September there was a release of CUDA13 via the Nvidia Ubuntu repository update and I accepted the download by accident not realising it was not compatible with my system. It did not complete the install until I re-powered the machine on the 30th. Unfortunately there was no corresponding release of PyTorch and so the existing Torch 2.8..0 was incompatible and the software stack broke just before I was due to demonstrate the machine. :(

11

Hi,

CUDA 13 is not compatible with JetPack 6 due to the library expects the SBSA driver.
For Orin, please stay with CUDA 12.

Sorry for the inconvenience, and thanks for your update.

12

Hi AastaLLL, A whole load of new updates have come through on the Nvidia Jammy repository channel. The descriptions for each package are unusually terse. I assume it’s safer to refuse them all?

13

Hi,

The packages come from the same JetPack that has been tested and verified together before the release, and it’s expected to be stable.

Thanks.