Encountered an issue while building l4t-pytorch using Jetson containers on JetPack 6.2

kerokerokero · May 21, 2025, 5:31am

I encountered an issue while trying to build l4t-pytorch using jetson-containers, which caused the build to fail. Below is the error message I received:

testing torchvision...
Traceback (most recent call last):
  File "/test/test.py", line 12, in <module>
    import torchvision
  ...
line 198, in _register_fake
    handle = entry.fake_impl.register(func_to_register, source)
  File "/usr/local/lib/python3.10/dist-packages/torch/_library/fake_impl.py", line 31, in register
    if torch._C._dispatch_has_kernel_for_dispatch_key(self.qualname, "Meta"):
RuntimeError: operator torchvision::nms does not exist
[13:22:54] Failed building:  l4t-pytorch

May I ask how I can resolve this and complete the build successfully?

By the way, I’m aware that I can use images from Docker Hub, but I’d like to try building one myself using Jetson containers, as I might need different combinations of packages or specific versions of CUDA and Python.

For reference, my setup is as follows:

Device: Jetson AGX Orin 32GB
JetPack version: 6.2
Build command: jetson-containers build l4t-pytorch

AastaLLL · May 21, 2025, 8:26am

Hi,

Confirm that we can reproduce the same issue.
We will check it and provide more info to you later.

Thanks.

AastaLLL · May 21, 2025, 9:32am

Hi,

Please try the below two commands instead:

$ PYTORCH_VERSION=2.7 jetson-containers build pytorch
...
PyTorch 2.7.0

  * CUDA device     Orin
  * CUDA version    12.6
  * CUDA cuDNN      90300
  * CUDA BLAS       _BlasBackend.Cublas
  * CUDA linalg     _BlasBackend.Cublas
  * CUDA flash_attn True
  * CUDA flash_sdp  True
  * CUDA cudnn_sdp  True
  * CUDA math_sdp   True
  * CUDA mem_efficient_sdp_enabled    True
  * CUDA fp16_bf16_reduction_math_sdp False

torch.distributed: True
  * NCCL backend:  True
  * GLOO backend:  True
  * MPI backend:   False

PACKAGING_VERSION=2.7.0
TORCH_CUDA_ARCH_LIST=8.7

/test/test.py:45: UserWarning: The torch.cuda.*DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=*, device='cuda') to create tensors. (Triggered internally at /opt/pytorch/torch/csrc/tensor/python_tensor.cpp:78.)
  a = torch.cuda.FloatTensor(2).zero_()
Tensor a = tensor([0., 0.], device='cuda:0')
Tensor b = tensor([ 0.2636, -1.0142], device='cuda:0')
Tensor c = tensor([ 0.2636, -1.0142], device='cuda:0')
testing LAPACK (OpenBLAS)...
done testing LAPACK (OpenBLAS)
testing torch.nn (cuDNN)...
done testing torch.nn (cuDNN)
testing CPU tensor vector operations...
/test/test.py:84: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
  cpu_y = F.softmax(cpu_x)
Tensor cpu_x = tensor([12.3450])
Tensor softmax = tensor([1.])
Tensor exp (float32) = tensor([[2.7183, 2.7183, 2.7183],
        [2.7183, 2.7183, 2.7183],
        [2.7183, 2.7183, 2.7183]])
Tensor exp (float64) = tensor([[2.7183, 2.7183, 2.7183],
        [2.7183, 2.7183, 2.7183],
        [2.7183, 2.7183, 2.7183]], dtype=torch.float64)
Tensor exp (diff) = 7.429356050359104e-07
PyTorch OK


[09:22:16] ✅ Built pytorch (pytorch:r36.4-cu126-22.04)

$ PYTORCH_VERSION=2.7 jetson-containers build torchvision
...
---------------------------------------------
-- Summary
---------------------------------------------
 
resnet18
   * Acc@1 69.740  Expected 69.760   Delta -0.020
   * Acc@5 88.760  Expected 89.080   Delta -0.320
   * Images/sec  294.496
   * PASS

Model tests passing:  1 / 1
torchvision OK


[09:30:20] ✅ Built torchvision (torchvision:r36.4-cu126-22.04)

Thanks.

kerokerokero · May 21, 2025, 10:19am

Hi @AastaLLL

First of all, thank you for your reply. The command PYTORCH_VERSION=2.7 jetson-containers build pytorch does work as expected. However, I was wondering—is this the same image as l4t-pytorch? I suspect the main issue might be related to torchvision. It seems that torchvision isn’t included in the PyTorch installation by default, which might explain why the build completes successfully.

Hi @AastaLLL

Let me provide a quick update — building transformers using jetson-containers results in the same error: RuntimeError: operator torchvision::nms does not exist , which causes the build to fail. I suspect this might also be due to a compatibility issue between torchvision and pytorch .

The command PYTORCH_VERSION=2.7 jetson-containers build torchvision runs successfully. However, if I change the PYTORCH_VERSION to a version other than 2.7 (for example, PYTORCH_VERSION=2.5 jetson-containers build torchvision), I encounter some errors. Here’s an excerpt from the error message:

[6402/7067] Building CUDA object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/sparse/cuda/SparseCsrTensorMath.cu.o
[6403/7067] Building CUDA object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/attention_backward.cu.o
[6404/7067] Building CUDA object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_bwd_hdim224_bf16_sm80.cu.o
[6405/7067] Building CUDA object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_bwd_hdim160_bf16_sm80.cu.o
[6406/7067] Building CUDA object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_bwd_hdim160_fp16_sm80.cu.o
[6407/7067] Building CUDA object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_bwd_hdim224_fp16_sm80.cu.o
[6408/7067] Building CUDA object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_bwd_hdim192_fp16_sm80.cu.o
[6409/7067] Building CUDA object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_bwd_hdim192_bf16_sm80.cu.o
[6410/7067] Building CUDA object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_bwd_hdim128_bf16_sm80.cu.o
[6411/7067] Building CUDA object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/sparse/cuda/SparseSemiStructuredLinear.cu.o
...
ptxas /tmp/tmpxft_0000a77a_00000000-6_SparseSemiStructuredLinear.ptx, line 256643; info    : Advisory: Modifier '.sp::ordered_metadata' should be used on instruction 'mma' instead of modifier '.sp' as it is expected to have substantially reduced performance on some future architectures
[6412/7067] Building CUDA object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_bwd_hdim256_fp16_sm80.cu.o
[6413/7067] Building CUDA object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/transformers/cuda/flash_attn/kernels/flash_bwd_hdim256_bf16_sm80.cu.o
ninja: build stopped: subcommand failed.
The command '/bin/sh -c /tmp/pytorch/install.sh || /tmp/pytorch/build.sh' returned a non-zero code: 1
[11:31:07] Failed building:  torchvision

Does this mean that JetPack 6.2 only supports PyTorch version 2.7 for now?
Or is there a known workaround to make other versions work?

Any insights or suggestions would be greatly appreciated. Thanks in advance!

AastaLLL · May 22, 2025, 6:04am

Hi,

You can find more info in the jetson-container source directly.

The l4t-pytorch has some dependencies to build so the procedures are much more complicated:

github.com/dusty-nv/jetson-containers

packages/ml/l4t/l4t-pytorch/l4t-pytorch.json

master


      
          {

              "l4t-pytorch": {

                  "group": "ml",

                  "depends": ["pytorch", "torchvision", "torchaudio", "torch2trt", "opencv", "pycuda"]

              }

          }

Pytorch 2.5 is expected to work and the container should download from the server instead of building it from the source.

Pytorch 2.5.0 can be found in below link:
jp6/cu126/: torch-2.5.0 metadata and description
From the torchvision config, the corresponding package should be 0.20.0:
jetson-containers/packages/pytorch/torchvision/config.py at master · dusty-nv/jetson-containers · GitHub
TorchVision 0.20.0 is also available on our server:
jp6/cu126/: torchvision-0.20.0 metadata and description

Thanks.

kerokerokero · May 22, 2025, 7:45am

Hi @AastaLLL
Based on the first message you provided, it seems that l4t-pytorch requires additional dependencies to be installed. Therefore, I’m currently trying to build it using the command:

PYTORCH_VERSION=2.7 jetson-containers build l4t-pytorch

This builds successfully. However, modifying PYTORCH_VERSION to another value or leaving it unspecified results in build failure.

According to the second message, PyTorch 2.5 is expected to work normally. But when I try building the image with that version, I noticed the following message during installation:

Collecting torch==2.5
  Downloading https://pypi.jetson-ai-lab.dev/jp6/cu126/%2Bf/5cf/9ed17e35cb752/torch-2.5.0-cp310-cp310-linux_aarch64.whl (230.6 MB)

This clearly shows that PyTorch 2.5 is being downloaded from the server. However, in the subsequent steps, it appears to fall back to building from source. Here is part of the output:

PyTorch version: 2.5.0
CUDA available:  True
Traceback (most recent call last):
...
RuntimeError: cuDNN version incompatibility: PyTorch was compiled against (9, 4, 0) but found runtime version (9, 3, 0). PyTorch already comes bundled with cuDNN. One option to resolving this error is to ensure PyTorch can find the bundled cuDNN. Looks like your LD_LIBRARY_PATH contains incompatible version of cudnn. Please either remove it from the path or install cudnn (9, 4, 0)
+ echo 'Building PyTorch 2.5.0'
+ git clone --branch v2.5.0 --depth=1 --recursive https://github.com/pytorch/pytorch /opt/pytorch

This part might be where the issue lies. You might be able to reproduce the same behavior on your end by running the same command:

PYTORCH_VERSION=2.5 jetson-containers build l4t-pytorch

I took a quick look at the source code and here’s my assumption:

In version.py, I noticed the following logic:

elif CUDA_VERSION == Version('12.6'):  # JetPack 6.2 (CUDA 12.6)
    PYTORCH_VERSION = Version('2.6')

So, when no specific version is provided, it defaults to PyTorch 2.6. This explains why omitting the version (e.g., not specifying 2.7) causes a failure.

Also, according to the Dockerfile, the build instruction:

RUN /tmp/pytorch/install.sh || /tmp/pytorch/build.sh

will attempt to install PyTorch first, and if that fails, it will proceed to build from source.

It might be that the installation step fails due to a mismatch between the expected and actual cuDNN versions.

kerokerokero · May 26, 2025, 1:55am

Hi @AastaLLL

May I ask for your thoughts on my assumption? Going back to my original question — I’d like to use Jetson Containers to build a Docker image with a specific version of PyTorch and CUDA. However, it seems that both commands:

CUDA_VERSION=12.2 jetson-containers build pytorch, and
PYTORCH_VERSION=2.5 jetson-containers build l4t-pytorch

result in errors and fail to build successfully. So far, the only setup that seems to work is using the default CUDA version along with PyTorch 2.7.

Does this mean that, at the moment, it’s not possible to build a Docker image with custom versions of both CUDA and PyTorch using Jetson Containers?

AastaLLL · May 26, 2025, 5:37am

Hi,

RuntimeError: cuDNN version incompatibility: PyTorch was compiled against (9, 4, 0) but found runtime version (9, 3, 0). PyTorch already comes bundled with cuDNN. One option to resolving this error is to ensure PyTorch can find the bundled cuDNN. Looks like your LD_LIBRARY_PATH contains incompatible version of cudnn. Please either remove it from the path or install cudnn (9, 4, 0)

Based on the above log, you can force the container to install cuDNN 9.4.0 to fix the issue.

Thanks.

kerokerokero · May 26, 2025, 7:32am

Hi @AastaLLL ,

I think there might have been a misunderstanding, so I’d like to clarify my question.

What I was trying to ask is: is it possible to directly build with a specific version of PyTorch or CUDA using jetson-containers, without having to manually enter the container and run additional commands to install or switch versions?

According to the GitHub documentation, it seems like we’re supposed to be able to specify the desired version via environment variables during the build process—such as:

CUDA_VERSION=12.2 jetson-containers build pytorch

or

PYTORCH_VERSION=2.5 jetson-containers build l4t-pytorch

From what I understand, this should allow us to build the desired version directly, without needing to do anything further inside the container. However, in my case, the builds are currently failing even when I follow the instructions as described.

That’s why I’m raising this question—just to confirm:
Is it actually possible to select the PyTorch or CUDA version directly using jetson-containers as intended?
Or is some manual intervention still required inside the container?

Thanks in advance for your help!

AastaLLL · May 28, 2025, 10:05am

Hi,

The Pytorch 2.5 should work on JetPack 6.2 as there is prebuilt available.

The jetson-container first tries to download the package before building it from the source.
However, you will need to use the same environment (like CUDA or cuDNN) as the package had been built.

Alternatively, you can always force the container to build from the source so it will recreate a package based on the environment you have.

Thanks.

kerokerokero · May 28, 2025, 10:30am

Hi @AastaLLL

First of all, I noticed that it initially tried to download the prebuilt version. Then, after the installation failed, it proceeded to build from source. Finally, the build itself failed. This is the situation I observed during my testing.

Given this, it seems that neither of the two methods you provided currently work with jetson-containers on JetPack 6.2.

That’s why I’ve been repeatedly asking: Is it actually possible to select the PyTorch or CUDA version directly using jetson-containers as intended?

Perhaps it might help if you could try to reproduce the issue on your own setup.

AastaLLL · June 2, 2025, 8:48am

Hi,

To auto-select the PyTorch version, you can disable the install script and force the jetson-container to build from the source every time.
The compatibility issue usually comes from the prebuilt package.

Thanks.

kerokerokero · June 3, 2025, 5:58am

Hi @AastaLLL

Hmm… I did notice that after the installation failed, jetson-containers proceeded to download the source and attempted to build it, but it still ended up failing. I’m wondering—how is this different from what you mentioned about disabling the install script and forcing jetson-containers to build? If it’s essentially the same process, wouldn’t the result still be a failure?

AastaLLL · June 19, 2025, 6:18am

Hi,

Do you have the logs of the building process?

Since you may still meet the compatibility issue if not all the dependencies are built from the source.
(Ex. Downloading PyTorch from the server but building TorchVision from the source).

Thanks.

kerokerokero · June 20, 2025, 6:49am

HI @AastaLLL

May I ask what kind of logs you need? Would you prefer the full build log, or just the initial part of it? I believe the information I shared above should be enough to reproduce the issue — or is it perhaps not reproducible on your end?

AastaLLL · July 17, 2025, 5:13am

Hi,

We changed the download link to https://pypi.jetson-ai-lab.io recently.
And only PyTorch 2.6 and 2.8 are supported for JetPack 6.2.

Thanks.

system · August 13, 2025, 1:42am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
PyTorch container build failing Jetson AGX Orin pytorch , containers	5	374	June 8, 2023
Error in Building Machine Learning Container for Jetson and Jetpack 4.4 Jetson AGX Xavier docker	3	638	October 18, 2021
Jetson container package torch2trt on nano orin 8gb Developement kit Jetson Orin Nano containers	2	682	September 6, 2023
Docker container example that can use CUDA and Pytorch 2.5.0 with the host running Jetpack 6.1 Jetson AGX Orin docker , pytorch	8	542	December 26, 2024
Installing torchvision inside docker image Jetson TX2 docker	7	3591	October 18, 2021
Build Hello AI World Jetson Inference in docker container Jetson AGX Xavier docker	2	1238	October 18, 2021
Torchaudio fails to build in jetson containers Jetson Orin Nano pytorch	3	879	September 20, 2023
Install older version of pyotrch on jetpack 4.5.1 Jetson AGX Xavier pytorch	4	1498	December 8, 2021
Pytorch on Jetson nano Jetpack 4.4 R32.4.4 Jetson Nano pytorch	4	1467	October 18, 2021
Jetpack 4.6 L4T 32.6.1 allows pytorch cuda support Jetson Nano pytorch	2	3410	October 15, 2021

Encountered an issue while building l4t-pytorch using Jetson containers on JetPack 6.2

Related topics