Jetson-containers Error: Unexpected error from cudaGetDeviceCount()

Hi there Jetson Community,

I am trying to build an image from jetson-containers with the following command:

jetson-containers build --name=test_combo_image realsense nanoowl nanosam

During the testing phase of the build, I get the following error:

───────────────────────────────────────────────────────────────────────┐
│ > TESTING  test_combo_image:r36.4.tegra-aarch64-cu126-22.04-torch2trt │
└───────────────────────────────────────────────────────────────────────┘

docker run -t --rm --network=host --privileged --runtime=nvidia \
  --volume /ssd/repos/jetson-containers/packages/pytorch/torch2trt:/test \
  --volume /ssd/repos/jetson-containers/data:/data \
  test_combo_image:r36.4.tegra-aarch64-cu126-22.04-torch2trt \
    /bin/bash -c 'python3 /test/test.py


testing torch2trt...
/usr/local/lib/python3.10/dist-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
/usr/local/lib/python3.10/dist-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=AlexNet_Weights.IMAGENET1K_V1`. You can also use `weights=AlexNet_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
Traceback (most recent call last):
  File "/test/test.py", line 9, in <module>
    model = alexnet(pretrained=True).eval().cuda()
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1082, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 928, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 928, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 955, in _apply
    param_applied = fn(param)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1082, in <lambda>
    return self._apply(lambda t: t.cuda(device))
  File "/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py", line 412, in _lazy_init
    torch._C._cuda_init()
RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 34: CUDA driver is a stub library
[12:25:51] ===================================================================================== 
[12:25:51] ===================================================================================== 
[12:25:51] 💣 `jetson-containers build` failed after 75.1 seconds (1.3 minutes) 
[12:25:51] Error: Command 'docker run -t --rm --network=host --privileged --runtime=nvidia   --volume /ssd/repos/jetson-containers/packages/pytorch/torch2trt:/test   --volume /ssd/repos/jetson-containers/data:/data   test_combo_image:r36.4.tegra-aarch64-cu126-22.04-torch2trt     /bin/bash -c 'python3 /test/test.py' 2>&1 | tee /ssd/repos/jetson-containers/logs/20250906_122435/test/19-1_test_combo_image_r36.4.tegra-aarch64-cu126-22.04-torch2trt_test.py.txt; exit ${PIPESTATUS[0]}' returned non-zero exit status 1. 
[12:25:51] ===================================================================================== 
[12:25:51] ===================================================================================== 
[12:25:51] Failed building:  realsense, nanoowl, nanosam

Traceback (most recent call last):
  File "/ssd/repos/jetson-containers/jetson_containers/build.py", line 129, in <module>
    build_container(**vars(args))
  File "/ssd/repos/jetson-containers/jetson_containers/container.py", line 246, in build_container
    test_container(container_name, pkg, simulate, build_idx=idx)
  File "/ssd/repos/jetson-containers/jetson_containers/container.py", line 456, in test_container
    status = subprocess.run(cmd.replace(_NEWLINE_, ' '), executable='/bin/bash', shell=True, check=True)
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'docker run -t --rm --network=host --privileged --runtime=nvidia   --volume /ssd/repos/jetson-containers/packages/pytorch/torch2trt:/test   --volume /ssd/repos/jetson-containers/data:/data   test_combo_image:r36.4.tegra-aarch64-cu126-22.04-torch2trt     /bin/bash -c 'python3 /test/test.py' 2>&1 | tee /ssd/repos/jetson-containers/logs/20250906_122435/test/19-1_test_combo_image_r36.4.tegra-aarch64-cu126-22.04-torch2trt_test.py.txt; exit ${PIPESTATUS[0]}' returned non-zero exit status 1

I suspect it is due to some driver/library mismatch. Can anyone help me out? Thanks!

*** Please note that this reply is generated by LLM automatically ***

Based on the provided context, it appears that you are experiencing an error while building an image for the Jetson Orin Nano. The error message is not explicitly stated, but you suspect it might be due to a driver or library mismatch.

To help you troubleshoot the issue, I’ll provide some general guidance and potential solutions. Please note that without the exact error message, it’s challenging to provide a precise solution.

  1. Verify the JetPack version: Ensure that you are using the correct JetPack version for your Jetson Orin Nano. You can check the JetPack version by running the command dpkg -l | grep jetpack on your Jetson device.
  2. Check the kernel version: Verify that the kernel version matches the one expected by the JetPack version. You can check the kernel version by running the command uname -r on your Jetson device.
  3. Update the kernel and JetPack: If you are using an older kernel or JetPack version, try updating to the latest version. You can use the apt-get command to update the kernel and JetPack.
  4. Verify the driver and library versions: Check the versions of the drivers and libraries used in your build process. Ensure that they are compatible with the JetPack version and kernel version you are using.
  5. Clean and rebuild: Try cleaning the build directory and rebuilding the image from scratch. This can help resolve any issues caused by incomplete or corrupted builds.
  6. Check for known issues: Search the NVIDIA forums and documentation for known issues related to the error you are experiencing. There might be a specific solution or workaround available.
  7. Provide more information: If none of the above steps help, please provide more information about the error message you are seeing, including the exact text and any relevant logs or output.

Some additional resources that might be helpful:

  • NVIDIA Jetson Linux Developer Guide: This guide provides detailed information on building and customizing images for Jetson devices.
  • NVIDIA JetPack Documentation: This documentation provides information on the JetPack SDK, including installation, configuration, and troubleshooting.
  • NVIDIA Forums: The NVIDIA forums are a great resource for finding solutions to common issues and getting help from the community.

I hope this helps you troubleshoot and resolve the issue. If you have any further questions or need more specific guidance, please feel free to ask.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

Hello,

  • dpkg -l | grep jetpack shows no output.
  • Output for uname -r
    • 5.15.148-tegra
  • Output for apt show nvidia-jetpack
    •  Package: nvidia-jetpack
       Version: 6.2.1+b38
       Priority: standard
       Section: metapackages
       Source: nvidia-jetpack (6.2.1)
       Maintainer: NVIDIA Corporation
       Installed-Size: 199 kB
       Depends: nvidia-jetpack-runtime (= 6.2.1+b38), nvidia-jetpack-dev (= 6.2.1+b38)
       Homepage: http://developer.nvidia.com/jetson
       Download-Size: 29.3 kB
       APT-Sources: https://repo.download.nvidia.com/jetson/common r36.4/main arm64 Packages
       Description: NVIDIA Jetpack Meta Package
      

Also note:

┌───────────────────────┬────────────────────────┐
│ L4T_VERSION   36.4.4  │ JETPACK_VERSION  6.2.1 │
│ CUDA_VERSION  12.6    │ PYTHON_VERSION   3.10  │
│ SYSTEM_ARCH   aarch64 │ LSB_RELEASE      22.04 │
└───────────────────────┴────────────────────────┘

Another thing to note here, I previous build the nanoowl container using this method, and it build and worked fine. However, when I tried to build it again just now, I ran into the same error. So something changed in between then, but I am not sure. Any help would be appreciated. Thanks!

Hi,
We have tried the steps for building NanoSAM:

Making sure you're not a bot!

Please give it a try and see if you can execute the steps seccesssfully. Would like to align the result on this setup.

Hi @DaneLLL,

I am unable to build nanosam. I am getting the following error:

triniphantom@firebolt:/ssd/repos/jetson-containers$ jetson-containers build $(autotag nanosam)
Namespace(packages=['nanosam'], prefer=['local', 'registry', 'build'], disable=[''], user='dustynv', output='/tmp/autotag', quiet=False, verbose=False)
-- L4T_VERSION=36.4.4  JETPACK_VERSION=6.2.1  CUDA_VERSION=12.6
-- Finding compatible container image for ['nanosam']
nanosam:r36.4.tegra-aarch64-cu126-22.04-tensorrt

Namespace(packages=['nanosam:r36.4.tegra-aarch64-cu126-22.04-tensorrt'], name='', base='', multiple=False, build_flags='', build_args='', use_proxy=False, package_dirs=[''], list_packages=False, show_pack
ages=False, skip_packages=[''], skip_errors=False, skip_tests=[''], test_only=[''], simulate=False, push='', no_github_api=False, log_dir=None, log_level=None, log_colors=None, log_status=None, debug=Fals
e, verbose=False, version=False)

┌───────────────────────┬────────────────────────┐
│ L4T_VERSION   36.4.4  │ JETPACK_VERSION  6.2.1 │
│ CUDA_VERSION  12.6    │ PYTHON_VERSION   3.10  │
│ SYSTEM_ARCH   aarch64 │ LSB_RELEASE      22.04 │
└───────────────────────┴────────────────────────┘

$ jetson-containers nanosam:r36.4.tegra-aarch64-cu126-22.04-tensorrt

[10:00:13] ===================================================================================== 
[10:00:13] ===================================================================================== 
[10:00:13] 💣 `jetson-containers build` failed after 1.8 seconds (0.0 minutes) 
[10:00:13] Error: "couldn't find package:  nanosam:r36.4.tegra-aarch64-cu126-22.04-tensorrt" 
[10:00:13] ===================================================================================== 
[10:00:13] ===================================================================================== 
[10:00:13] Failed building:  nanosam:r36.4.tegra-aarch64-cu126-22.04-tensorrt

Traceback (most recent call last):
  File "/ssd/repos/jetson-containers/jetson_containers/build.py", line 129, in <module>
    build_container(**vars(args))
  File "/ssd/repos/jetson-containers/jetson_containers/container.py", line 120, in build_container
    packages = resolve_dependencies(packages, skip_packages=skip_packages)
  File "/ssd/repos/jetson-containers/jetson_containers/packages.py", line 392, in resolve_dependencies
    packages, changed = add_depends(packages)
  File "/ssd/repos/jetson-containers/jetson_containers/packages.py", line 363, in add_depends
    for dependency in find_package(package).get('depends', []):
  File "/ssd/repos/jetson-containers/jetson_containers/packages.py", line 247, in find_package
    raise KeyError(f"couldn't find package:  {package}")
KeyError: "couldn't find package:  nanosam:r36.4.tegra-aarch64-cu126-22.04-tensorrt"

So I tried to install the tensorrt package with jetson-containers build $(autotag tensorrt) and I get the same error:

  --help, -h                  Print this message
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/tensorrt/__init__.py", line 76, in <module>
    from .tensorrt import *
ImportError: libnvdla_compiler.so: cannot open shared object file: No such file or directory
WARNING - failed to 'import tensorrt' under Python Python 3.10.12

[09:57:11] ===================================================================================== 
[09:57:11] ===================================================================================== 
[09:57:11] ✅ `jetson-containers build tensorrt` (tensorrt:r36.4.tegra-aarch64-cu126-22.04) 
[09:57:11] ⏱️  Total build time: 220.2 seconds (3.7 minutes) 
[09:57:11] ===================================================================================== 
[09:57:11] ===================================================================================== 
tensorrt:r36.4.tegra-aarch64-cu126-22.04

Namespace(packages=['tensorrt:r36.4.tegra-aarch64-cu126-22.04'], name='', base='', multiple=False, build_flags='', build_args='', use_proxy=False, package_dirs=[''], list_packages=False, show_packages=Fal
se, skip_packages=[''], skip_errors=False, skip_tests=[''], test_only=[''], simulate=False, push='', no_github_api=False, log_dir=None, log_level=None, log_colors=None, log_status=None, debug=False, verbo
se=False, version=False)

┌───────────────────────┬────────────────────────┐
│ L4T_VERSION   36.4.4  │ JETPACK_VERSION  6.2.1 │
│ CUDA_VERSION  12.6    │ PYTHON_VERSION   3.10  │
│ SYSTEM_ARCH   aarch64 │ LSB_RELEASE      22.04 │
└───────────────────────┴────────────────────────┘

$ jetson-containers tensorrt:r36.4.tegra-aarch64-cu126-22.04

[09:57:14] ===================================================================================== 
[09:57:14] ===================================================================================== 
[09:57:14] 💣 `jetson-containers build` failed after 1.9 seconds (0.0 minutes) 
[09:57:14] Error: "couldn't find package:  tensorrt:r36.4.tegra-aarch64-cu126-22.04" 
[09:57:14] ===================================================================================== 
[09:57:14] ===================================================================================== 
[09:57:14] Failed building:  tensorrt:r36.4.tegra-aarch64-cu126-22.04

Traceback (most recent call last):
  File "/ssd/repos/jetson-containers/jetson_containers/build.py", line 129, in <module>
    build_container(**vars(args))
  File "/ssd/repos/jetson-containers/jetson_containers/container.py", line 120, in build_container
    packages = resolve_dependencies(packages, skip_packages=skip_packages)
  File "/ssd/repos/jetson-containers/jetson_containers/packages.py", line 392, in resolve_dependencies
    packages, changed = add_depends(packages)
  File "/ssd/repos/jetson-containers/jetson_containers/packages.py", line 363, in add_depends
    for dependency in find_package(package).get('depends', []):
  File "/ssd/repos/jetson-containers/jetson_containers/packages.py", line 247, in find_package
    raise KeyError(f"couldn't find package:  {package}")
KeyError: "couldn't find package:  tensorrt:r36.4.tegra-aarch64-cu126-22.04"

The root cause of the error seems to be that it cannot find libnvdla_compiler.so

Hi @DaneLLL I was able to find the fix for the aforementioned using- Fixing the "libnvdla_compiler.so" on Jetson devices · GitHub

However I am still unable to build nanosam. I get the following error:

triniphantom@firebolt:/ssd/repos/jetson-containers$ jetson-containers build $(autotag nanosam)
Namespace(packages=['nanosam'], prefer=['local', 'registry', 'build'], disable=[''], user='dustynv', output='/tmp/autotag', quiet=False, verbose=False)
-- L4T_VERSION=36.4.4  JETPACK_VERSION=6.2.1  CUDA_VERSION=12.6
-- Finding compatible container image for ['nanosam']
nanosam:r36.4.tegra-aarch64-cu126-22.04-tensorrt

Namespace(packages=['nanosam:r36.4.tegra-aarch64-cu126-22.04-tensorrt'], name='', base='', multiple=False, build_flags='', build_args='', use_proxy=False, package_dirs=[''], list_packages=False, show_packages=False, skip_packages=[''], skip_errors=False, skip_tests=[''], test_only=[''], simulate=False, push='', no_github_api=False, log_dir=None, log_level=None, log_colors=None, log_status=None, debug=False, verbose=False, version=False)

┌───────────────────────┬────────────────────────┐
│ L4T_VERSION   36.4.4  │ JETPACK_VERSION  6.2.1 │
│ CUDA_VERSION  12.6    │ PYTHON_VERSION   3.10  │
│ SYSTEM_ARCH   aarch64 │ LSB_RELEASE      22.04 │
└───────────────────────┴────────────────────────┘

$ jetson-containers nanosam:r36.4.tegra-aarch64-cu126-22.04-tensorrt

[10:17:20] ===================================================================================== 
[10:17:20] ===================================================================================== 
[10:17:20] 💣 `jetson-containers build` failed after 1.7 seconds (0.0 minutes) 
[10:17:20] Error: "couldn't find package:  nanosam:r36.4.tegra-aarch64-cu126-22.04-tensorrt" 
[10:17:20] ===================================================================================== 
[10:17:20] ===================================================================================== 
[10:17:20] Failed building:  nanosam:r36.4.tegra-aarch64-cu126-22.04-tensorrt

Traceback (most recent call last):
  File "/ssd/repos/jetson-containers/jetson_containers/build.py", line 129, in <module>
    build_container(**vars(args))
  File "/ssd/repos/jetson-containers/jetson_containers/container.py", line 120, in build_container
    packages = resolve_dependencies(packages, skip_packages=skip_packages)
  File "/ssd/repos/jetson-containers/jetson_containers/packages.py", line 392, in resolve_dependencies
    packages, changed = add_depends(packages)
  File "/ssd/repos/jetson-containers/jetson_containers/packages.py", line 363, in add_depends
    for dependency in find_package(package).get('depends', []):
  File "/ssd/repos/jetson-containers/jetson_containers/packages.py", line 247, in find_package
    raise KeyError(f"couldn't find package:  {package}")
KeyError: "couldn't find package:  nanosam:r36.4.tegra-aarch64-cu126-22.04-tensorrt"

Which doesn’t make sense because the output of docker image ls shows:

triniphantom@firebolt:/ssd/repos/jetson-containers$ docker image ls
REPOSITORY                  TAG                                               IMAGE ID       CREATED          SIZE
tensorrt                    r36.4.tegra-aarch64-cu126-22.04                   0a715d78dda9   21 minutes ago   10.4GB
tensorrt                    r36.4.tegra-aarch64-cu126-22.04-tensorrt          0a715d78dda9   21 minutes ago   10.4GB
tensorrt                    r36.4.tegra-aarch64-cu126-22.04-python            87675231d0f4   23 minutes ago   7.02GB
<none>                      <none>                                            89252ef29d37   6 days ago       15.5GB
nanosam                     r36.4.tegra-aarch64-cu126-22.04-tensorrt          7773d0cc4b1f   6 days ago       15.5GB
nanosam                     r36.4.tegra-aarch64-cu126-22.04-torchvision       06030c5d4a41   6 days ago       12.1GB
nanosam                     r36.4.tegra-aarch64-cu126-22.04-pytorch_2.8       29258d62ef18   6 days ago       12.1GB
nanosam                     r36.4.tegra-aarch64-cu126-22.04-onnx              b11f38e16282   6 days ago       10.5GB
nanosam                     r36.4.tegra-aarch64-cu126-22.04-cmake             7009a6817a5b   6 days ago       10.4GB
nanosam                     r36.4.tegra-aarch64-cu126-22.04-numpy             8a1207c62e70   6 days ago       10.4GB
nanosam                     r36.4.tegra-aarch64-cu126-22.04-python            5738dbe7cad4   6 days ago       10.3GB
nanosam                     r36.4.tegra-aarch64-cu126-22.04-nvshmem           970580bf9809   6 days ago       10.2GB
nanosam                     r36.4.tegra-aarch64-cu126-22.04-cudss             08a0ccb2af8e   6 days ago       8.3GB
nanosam                     r36.4.tegra-aarch64-cu126-22.04-cusparselt        71b99321c4d8   6 days ago       8.09GB
nanosam                     r36.4.tegra-aarch64-cu126-22.04-nvpl              d65d2b013191   6 days ago       7.38GB
nanosam                     r36.4.tegra-aarch64-cu126-22.04-gdrcopy           441beae0c94e   6 days ago       7.31GB
nanosam                     r36.4.tegra-aarch64-cu126-22.04-nccl              d2f99a74bc8f   6 days ago       7.25GB
nanosam                     r36.4.tegra-aarch64-cu126-22.04-cudnn_9.3         4983e1602383   6 days ago       6.9GB
tensorrt                    r36.4.tegra-aarch64-cu126-22.04-cudnn_9.3         4983e1602383   6 days ago       6.9GB

So the package appears to be there…

I ran sudo apt install nvidia-cuda and that seemed to remove the above issue, but now when I try to build nanosam I get the original error back again:

[13:58:05] [18/22] Testing torch2trt (my_nanosam:r36.4.tegra-aarch64-cu126-22.04-torch2trt)                                                                                   17 stages completed in 01m09s 
at 13:58:05 

┌─────────────────────────────────────────────────────────────────┐
│ > TESTING  my_nanosam:r36.4.tegra-aarch64-cu126-22.04-torch2trt │
└─────────────────────────────────────────────────────────────────┘

docker run -t --rm --network=host --privileged --runtime=nvidia \
  --volume /ssd/repos/jetson-containers/packages/pytorch/torch2trt:/test \
  --volume /ssd/repos/jetson-containers/data:/data \
  my_nanosam:r36.4.tegra-aarch64-cu126-22.04-torch2trt \
    /bin/bash -c 'python3 /test/test.py


testing torch2trt...
/usr/local/lib/python3.10/dist-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
/usr/local/lib/python3.10/dist-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future
. The current behavior is equivalent to passing `weights=AlexNet_Weights.IMAGENET1K_V1`. You can also use `weights=AlexNet_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
Traceback (most recent call last):
  File "/test/test.py", line 9, in <module>
    model = alexnet(pretrained=True).eval().cuda()
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1082, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 928, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 928, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 955, in _apply
    param_applied = fn(param)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1082, in <lambda>
    return self._apply(lambda t: t.cuda(device))
  File "/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py", line 412, in _lazy_init
    torch._C._cuda_init()
RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 34: CUDA driver is a stub library
[13:58:13] ===================================================================================== 
[13:58:13] ===================================================================================== 
[13:58:13] 💣 `jetson-containers build` failed after 80.6 seconds (1.3 minutes) 
[13:58:13] Error: Command 'docker run -t --rm --network=host --privileged --runtime=nvidia   --volume /ssd/repos/jetson-containers/packages/pytorch/torch2trt:/test   --volume /ssd/repos/jetson-containers/
data:/data   my_nanosam:r36.4.tegra-aarch64-cu126-22.04-torch2trt     /bin/bash -c 'python3 /test/test.py' 2>&1 | tee /ssd/repos/jetson-containers/logs/20250913_135653/test/18-1_my_nanosam_r36.4.tegra-aar
ch64-cu126-22.04-torch2trt_test.py.txt; exit ${PIPESTATUS[0]}' returned non-zero exit status 1. 
[13:58:13] ===================================================================================== 
[13:58:13] ===================================================================================== 
[13:58:13] Failed building:  nanosam

Traceback (most recent call last):
  File "/ssd/repos/jetson-containers/jetson_containers/build.py", line 129, in <module>
    build_container(**vars(args))
  File "/ssd/repos/jetson-containers/jetson_containers/container.py", line 246, in build_container
    test_container(container_name, pkg, simulate, build_idx=idx)
  File "/ssd/repos/jetson-containers/jetson_containers/container.py", line 456, in test_container
    status = subprocess.run(cmd.replace(_NEWLINE_, ' '), executable='/bin/bash', shell=True, check=True)
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'docker run -t --rm --network=host --privileged --runtime=nvidia   --volume /ssd/repos/jetson-containers/packages/pytorch/torch2trt:/test   --volume /ssd/repos/jetso
n-containers/data:/data   my_nanosam:r36.4.tegra-aarch64-cu126-22.04-torch2trt     /bin/bash -c 'python3 /test/test.py' 2>&1 | tee /ssd/repos/jetson-containers/logs/20250913_135653/test/18-1_my_nanosam_r3
6.4.tegra-aarch64-cu126-22.04-torch2trt_test.py.txt; exit ${PIPESTATUS[0]}' returned non-zero exit status 1.

@DaneLLL @AastaLLL any assistance would be greatly appreciated.

I am at the point now where I am trying to build my own container using Pytorch from jp6/cu126/: torch versions. While this container did build, I got the following error when trying to run the nanoowl demo:


NotImplementedError: Could not run 'torchvision::roi_align' with arguments from the 'CUDA' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'torchvision::roi_align' is only available for these backends: [CPU, Meta, QuantizedCPU, BackendSelect, Python, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradHIP, AutogradXLA, AutogradMPS, AutogradIPU, AutogradXPU, AutogradHPU, AutogradVE, AutogradLazy, AutogradMTIA, AutogradMAIA, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, AutogradMeta, AutogradNestedTensor, Tracer, AutocastCPU, AutocastMTIA, AutocastMAIA, AutocastXPU, AutocastMPS, AutocastCUDA, FuncTorchBatched, BatchedNestedTensor, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, PreDispatch, PythonDispatcher].

This is similar to the error here-Error with Pytorch and Torchvision, however the wheels in the solution do not exist.

Any assistance would be greatly appreciated. Thank you!

I fixed it by adding:

Fix for CUDA driver is a stub library (from NVIDIA forums)

rm -f /usr/local/cuda/lib64/stubs/libcuda.so && \
ln -s /usr/local/cuda/lib64/libcuda.so /usr/local/cuda/lib64/stubs/libcuda.so

to the build script of torch2trt from [checkMacros.cpp::catchCudaError::272] Error Code 1: Cuda Runtime (CUDA driver is a stub library) - #11 by Robert_Crovella.

Hi @se.bader thanks for this information. I ended up building my own Docker image and I was able to solve the original issue with the cudaGetDeviceCount() with your suggestion. I addition I had to pass some .so files from /usr/lib/aarch64-linux-nvidia to the Docker image during the build process.

I was able to build the engine for nanoowl, but when I try to run tree_demo.py, I get the following issue when I request it to detect a an object:

NotImplementedError: Could not run 'torchvision::roi_align' with arguments from the 'CUDA' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'torchvision::roi_align' is only available for these backends: [CPU, Meta, QuantizedCPU, BackendSelect, Python, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradHIP, AutogradXLA, AutogradMPS, AutogradIPU, AutogradXPU, AutogradHPU, AutogradVE, AutogradLazy, AutogradMTIA, AutogradMAIA, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, AutogradMeta, AutogradNestedTensor, Tracer, AutocastCPU, AutocastMTIA, AutocastMAIA, AutocastXPU, AutocastMPS, AutocastCUDA, FuncTorchBatched, BatchedNestedTensor, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, PreDispatch, PythonDispatcher].

CPU: registered at /opt/torchvision/torchvision/csrc/ops/cpu/roi_align_kernel.cpp:390 [kernel]
Meta: registered at /dev/null:19 [kernel]
QuantizedCPU: registered at /opt/torchvision/torchvision/csrc/ops/quantized/cpu/qroi_align_kernel.cpp:283 [kernel]
BackendSelect: fallthrough registered at /opt/pytorch/aten/src/ATen/core/BackendSelectFallbackKernel.cpp:3 [backend fallback]
Python: registered at /opt/pytorch/aten/src/ATen/core/PythonFallbackKernel.cpp:194 [backend fallback]
FuncTorchDynamicLayerBackMode: registered at /opt/pytorch/aten/src/ATen/functorch/DynamicLayer.cpp:479 [backend fallback]
Functionalize: registered at /opt/pytorch/aten/src/ATen/FunctionalizeFallbackKernel.cpp:375 [backend fallback]
Named: registered at /opt/pytorch/aten/src/ATen/core/NamedRegistrations.cpp:7 [backend fallback]
Conjugate: registered at /opt/pytorch/aten/src/ATen/ConjugateFallback.cpp:17 [backend fallback]
Negative: registered at /opt/pytorch/aten/src/ATen/native/NegateFallback.cpp:18 [backend fallback]
ZeroTensor: registered at /opt/pytorch/aten/src/ATen/ZeroTensorFallback.cpp:86 [backend fallback]
ADInplaceOrView: fallthrough registered at /opt/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:104 [backend fallback]
AutogradOther: registered at /opt/torchvision/torchvision/csrc/ops/autograd/roi_align_kernel.cpp:157 [autograd kernel]
AutogradCPU: registered at /opt/torchvision/torchvision/csrc/ops/autograd/roi_align_kernel.cpp:157 [autograd kernel]
AutogradCUDA: registered at /opt/torchvision/torchvision/csrc/ops/autograd/roi_align_kernel.cpp:157 [autograd kernel]
AutogradHIP: registered at /opt/torchvision/torchvision/csrc/ops/autograd/roi_align_kernel.cpp:157 [autograd kernel]
AutogradXLA: registered at /opt/torchvision/torchvision/csrc/ops/autograd/roi_align_kernel.cpp:157 [autograd kernel]
AutogradMPS: registered at /opt/torchvision/torchvision/csrc/ops/autograd/roi_align_kernel.cpp:157 [autograd kernel]
AutogradIPU: registered at /opt/torchvision/torchvision/csrc/ops/autograd/roi_align_kernel.cpp:157 [autograd kernel]
AutogradXPU: registered at /opt/torchvision/torchvision/csrc/ops/autograd/roi_align_kernel.cpp:157 [autograd kernel]
AutogradHPU: registered at /opt/torchvision/torchvision/csrc/ops/autograd/roi_align_kernel.cpp:157 [autograd kernel]
AutogradVE: registered at /opt/torchvision/torchvision/csrc/ops/autograd/roi_align_kernel.cpp:157 [autograd kernel]
AutogradLazy: registered at /opt/torchvision/torchvision/csrc/ops/autograd/roi_align_kernel.cpp:157 [autograd kernel]
AutogradMTIA: registered at /opt/torchvision/torchvision/csrc/ops/autograd/roi_align_kernel.cpp:157 [autograd kernel]
AutogradMAIA: registered at /opt/torchvision/torchvision/csrc/ops/autograd/roi_align_kernel.cpp:157 [autograd kernel]
AutogradPrivateUse1: registered at /opt/torchvision/torchvision/csrc/ops/autograd/roi_align_kernel.cpp:157 [autograd kernel]
AutogradPrivateUse2: registered at /opt/torchvision/torchvision/csrc/ops/autograd/roi_align_kernel.cpp:157 [autograd kernel]
AutogradPrivateUse3: registered at /opt/torchvision/torchvision/csrc/ops/autograd/roi_align_kernel.cpp:157 [autograd kernel]
AutogradMeta: registered at /opt/torchvision/torchvision/csrc/ops/autograd/roi_align_kernel.cpp:157 [autograd kernel]
AutogradNestedTensor: registered at /opt/torchvision/torchvision/csrc/ops/autograd/roi_align_kernel.cpp:157 [autograd kernel]
Tracer: registered at /opt/pytorch/torch/csrc/autograd/TraceTypeManual.cpp:294 [backend fallback]
AutocastCPU: registered at /opt/torchvision/torchvision/csrc/ops/autocast/roi_align_kernel.cpp:43 [kernel]
AutocastMTIA: fallthrough registered at /opt/pytorch/aten/src/ATen/autocast_mode.cpp:466 [backend fallback]
AutocastMAIA: fallthrough registered at /opt/pytorch/aten/src/ATen/autocast_mode.cpp:504 [backend fallback]
AutocastXPU: registered at /opt/torchvision/torchvision/csrc/ops/autocast/roi_align_kernel.cpp:51 [kernel]
AutocastMPS: fallthrough registered at /opt/pytorch/aten/src/ATen/autocast_mode.cpp:209 [backend fallback]
AutocastCUDA: registered at /opt/torchvision/torchvision/csrc/ops/autocast/roi_align_kernel.cpp:35 [kernel]
FuncTorchBatched: registered at /opt/pytorch/aten/src/ATen/functorch/LegacyBatchingRegistrations.cpp:731 [backend fallback]
BatchedNestedTensor: registered at /opt/pytorch/aten/src/ATen/functorch/LegacyBatchingRegistrations.cpp:758 [backend fallback]
FuncTorchVmapMode: fallthrough registered at /opt/pytorch/aten/src/ATen/functorch/VmapModeRegistrations.cpp:27 [backend fallback]
Batched: registered at /opt/pytorch/aten/src/ATen/LegacyBatchingRegistrations.cpp:1075 [backend fallback]
VmapMode: fallthrough registered at /opt/pytorch/aten/src/ATen/VmapModeRegistrations.cpp:33 [backend fallback]
FuncTorchGradWrapper: registered at /opt/pytorch/aten/src/ATen/functorch/TensorWrapper.cpp:210 [backend fallback]
PythonTLSSnapshot: registered at /opt/pytorch/aten/src/ATen/core/PythonFallbackKernel.cpp:202 [backend fallback]
FuncTorchDynamicLayerFrontMode: registered at /opt/pytorch/aten/src/ATen/functorch/DynamicLayer.cpp:475 [backend fallback]
PreDispatch: registered at /opt/pytorch/aten/src/ATen/core/PythonFallbackKernel.cpp:206 [backend fallback]
PythonDispatcher: registered at /opt/pytorch/aten/src/ATen/core/PythonFallbackKernel.cpp:198 [backend fallback

Hi,

Sorry for the late update.
Could you try the command shared in the link below to see if it can solve the NMS error?

Thanks.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.