Jetson Orin Nano Developer Kit, Jetpack, Cuda, Tensorflow with GPU and TensorRT

[pavanrshetty361998]

Hi
I was trying to get Tensorflow working with GPU support and also TensorRT in my Jetson Orin Nano Developer Kit for my project.
I was able to get Tensorflow working with GPU, but TensorRT failed to build.
I tried to use Jetpack 5 series image, but this comes with CUDA 11.4 and there is no tensorRT support for this Cuda version.
I also tried the Jetson image with the Jetpack version in the 4 and below series, but Jetson does not even boot, apparently, it is not meant for the Jetson Orin Nano developer kit.

I needed some help to go about this such that I have my Jetson Orin Nano Developer Kit ready to work with TensorFlow GPU support along with TensorRT. Or suggest me any doable alternatives

[AastaLLL]

Hi,

Pleas use JetPack 5.x for Orin series.
There is a compatible TensorRT package in JetPack 5.1.1.

We can find TensorRT 8.5 in the JetPack 5.1.1 environment. Please check it again.

$ apt show nvidia-tensorrt
Package: nvidia-tensorrt
Version: 5.1.1-b56
Priority: standard
Section: metapackages
Maintainer: NVIDIA Corporation
Installed-Size: 205 kB
Depends: libnvinfer-bin (= 8.5.2-1+cuda11.4), libnvinfer8 (= 8.5.2-1+cuda11.4), libnvparsers8 (= 8.5.2-1+cuda11.4), libnvinfer-plugin8 (= 8.5.2-1+cuda11.4), libnvonnxparsers8 (= 8.5.2-1+cuda11.4)
Conflicts: libnvinfer-plugin6, libnvinfer-plugin7, libnvinfer6, libnvinfer7, libnvonnxparsers6, libnvonnxparsers7, libnvparsers6, libnvparsers7, python-libnvinfer
Homepage: http://developer.nvidia.com/jetson
Download-Size: 27.3 kB
APT-Manual-Installed: yes
APT-Sources: https://repo.download.nvidia.com/jetson/common r35.3/main arm64 Packages
Description: NVIDIA TensorRT Meta Package

Thanks.

[pavanrshetty361998]

Even though I see:

apt show nvidia-tensorrt
Package: nvidia-tensorrt
Version: 5.1.1-b56
Priority: standard
Section: metapackages
Maintainer: NVIDIA Corporation
Installed-Size: 205 kB
Depends: libnvinfer-bin (= 8.5.2-1+cuda11.4), libnvinfer8 (= 8.5.2-1+cuda11.4), libnvparsers8 (= 8.5.2-1+cuda11.4), libnvinfer-plugin8 (= 8.5.2-1+cuda11.4), libnvonnxparsers8 (= 8.5.2-1+cuda11.4)
Conflicts: libnvinfer-plugin6, libnvinfer-plugin7, libnvinfer6, libnvinfer7, libnvonnxparsers6, libnvonnxparsers7, libnvparsers6, libnvparsers7, python-libnvinfer
Homepage: Jetson - Embedded AI Computing Platform | NVIDIA Developer
Download-Size: 27.3 kB
APT-Sources: https://repo.download.nvidia.com/jetson/common r35.3/main arm64 Packages
Description: NVIDIA TensorRT Meta Package

I am unable to import tensorrt as described below irrespective of the environment I am in:

Python 3.8.17 (default, Jul 5 2023, 20:40:03)
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type “help”, “copyright”, “credits” or “license” for more information.

import tensorrt as trt
Traceback (most recent call last):
File “”, line 1, in
ModuleNotFoundError: No module named ‘tensorrt’

If I do a pip installation of Tensorrt, I fail to build the wheel and the error is described below:

pip install tensorrt
Collecting tensorrt
Downloading tensorrt-8.6.1.tar.gz (16 kB)
Preparing metadata (setup.py) … done
Building wheels for collected packages: tensorrt
Building wheel for tensorrt (setup.py) … error
error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [65 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build/lib
creating build/lib/tensorrt
copying tensorrt/init.py → build/lib/tensorrt
running egg_info
writing tensorrt.egg-info/PKG-INFO
writing dependency_links to tensorrt.egg-info/dependency_links.txt
writing requirements to tensorrt.egg-info/requires.txt
writing top-level names to tensorrt.egg-info/top_level.txt
reading manifest file ‘tensorrt.egg-info/SOURCES.txt’
adding license file ‘LICENSE.txt’
writing manifest file ‘tensorrt.egg-info/SOURCES.txt’
/home/shepherdlab/miniconda3/envs/tf_install/lib/python3.8/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

          ********************************************************************************
          Please avoid running ``setup.py`` directly.
          Instead, use pypa/build, pypa/installer, pypa/build or
          other standards-based tools.

          See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
          ********************************************************************************

  !!
    self.initialize_options()
  installing to build/bdist.linux-aarch64/wheel
  running install
  Looking in indexes: https://pypi.nvidia.com
  ERROR: Could not find a version that satisfies the requirement tensorrt_libs==8.6.1 (from versions: none)
  ERROR: No matching distribution found for tensorrt_libs==8.6.1
  Traceback (most recent call last):
    File "<string>", line 2, in <module>
    File "<pip-setuptools-caller>", line 34, in <module>
    File "/tmp/pip-install-cpfzofbz/tensorrt_e506330037c640aba1576df8e6923f5c/setup.py", line 49, in <module>
      setup(
    File "/home/shepherdlab/miniconda3/envs/tf_install/lib/python3.8/site-packages/setuptools/__init__.py", line 107, in setup
      return distutils.core.setup(**attrs)
    File "/home/shepherdlab/miniconda3/envs/tf_install/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
    File "/home/shepherdlab/miniconda3/envs/tf_install/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/home/shepherdlab/miniconda3/envs/tf_install/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/home/shepherdlab/miniconda3/envs/tf_install/lib/python3.8/site-packages/setuptools/dist.py", line 1244, in run_command
      super().run_command(command)
    File "/home/shepherdlab/miniconda3/envs/tf_install/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/home/shepherdlab/miniconda3/envs/tf_install/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 360, in run
      self.run_command("install")
    File "/home/shepherdlab/miniconda3/envs/tf_install/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/home/shepherdlab/miniconda3/envs/tf_install/lib/python3.8/site-packages/setuptools/dist.py", line 1244, in run_command
      super().run_command(command)
    File "/home/shepherdlab/miniconda3/envs/tf_install/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/tmp/pip-install-cpfzofbz/tensorrt_e506330037c640aba1576df8e6923f5c/setup.py", line 43, in run
      install_dep("{:}_libs".format(tensorrt_module))
    File "/tmp/pip-install-cpfzofbz/tensorrt_e506330037c640aba1576df8e6923f5c/setup.py", line 41, in install_dep
      status.check_returncode()
    File "/home/shepherdlab/miniconda3/envs/tf_install/lib/python3.8/subprocess.py", line 448, in check_returncode
      raise CalledProcessError(self.returncode, self.args, self.stdout,
  subprocess.CalledProcessError: Command '['/home/shepherdlab/miniconda3/envs/tf_install/bin/python', '-m', 'pip', 'install', 'tensorrt_libs==8.6.1', '--index-url', 'https://pypi.nvidia.com']' returned non-zero exit status 1.
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for tensorrt
Running setup.py clean for tensorrt
Failed to build tensorrt
ERROR: Could not build wheels for tensorrt, which is required to install pyproject.toml-based projects

[pavanrshetty361998]

Also please have a look here

TensorRT for CUDA 11.4 Requested

Description TensorRT 8.0 is ONLY for CUDA 11.3 so far. Looking forward to TensorRT for CUDA 11.4. Environment TensorRT Version: N/A (8.0 GA) GPU Type: Geforce 2080 Ti Nvidia Driver Version: 470.57.02 CUDA Version: 11.4.1 CUDNN Version: 8.2.2 Operating System + Version: Ubuntu 20.04.2 Python Version (if applicable): 3.8.10 TensorFlow Version (if applicable): 2.7.0 PyTorch Version (if applicable): 1.10 Baremetal or Container (if container which image + tag): N/A Relevant Files N/A …

[AastaLLL]

Hi,

Please try the following command to install the TensorRT python package.

$ sudo apt install python3-libnvinfer*

Thanks.

[pavanrshetty361998]

I did the ```
$ sudo apt install python3-libnvinfer*

**and I am able to import Tensort in Python, but none of the Tensorrt attributes seems to work such as trt.Logger, trt.Runtime, trt.volume, trt.version or trt.__version__**
**In short, I am unable to use tensorrt in any way**

There is an attribute error:
AttributeError: module 'tensorrt' has no attribute '__version__'
AttributeError: module 'tensorrt' has no attribute 'version
AttributeError: module 'tensorrt' has no attribute 'Runtime'

**Also, I don't see enough packages under usr/lib/python3.8/dist_pacakges/tensorrt. There are __init__.py and tensorrt.so files only as shown in the screenshot attached:**
![Screenshot 2023-07-27 132937|612x406](upload://2iAYEV3poM732rVujn5ItBhGWqF.png)

**The files under /usr/lib/python3.8/dist-packages/tensorrt-8.5.2.2.dist-info is shown in the screenshot attached:**
![image|609x400](upload://azyUWmnkfjyyHJQdgFEybj8A6I3.png)

[AastaLLL]

Hi,

Do you happen to have a script called “tensorrt”?

The screenshot is not attached.
Could you upload it again?

Thanks

[pavanrshetty361998]

No, there is no script called Tensorrt. I am trying to import TensorRT and use its attribute, but none of the attributes seems to work

Below, you can find the ss

[AastaLLL]

Hi,

Your library looks the same as our environment. So it should be good.

xxx:/usr/lib/python3.8/dist-packages/tensorrt-8.5.2.2.dist-info$ ll
total 76
drwxr-xr-x  2 root root  4096 Jun 29 10:44 ./
drwxr-xr-x 12 root root  4096 Jun 29 10:45 ../
-rw-r--r--  1 root root 46991 Dec  6  2022 LICENSE.txt
-rw-r--r--  1 root root   589 Dec  6  2022 METADATA
-rw-r--r--  1 root root   651 Dec  6  2022 RECORD
-rw-r--r--  1 root root     9 Dec  6  2022 top_level.txt
-rw-r--r--  1 root root   103 Dec  6  2022 WHEEL
-rw-r--r--  1 root root     1 Dec  6  2022 zip-safe
xxx:/usr/lib/python3.8/dist-packages/tensorrt$ ll
total 3392
drwxr-xr-x  2 root root    4096 Jun 29 10:44 ./
drwxr-xr-x 12 root root    4096 Jun 29 10:45 ../
-rw-r--r--  1 root root    5118 Dec  6  2022 __init__.py
-rw-r--r--  1 root root 3453024 Dec  6  2022 tensorrt.so

Do you import TensorRT with python3?
We can import it without issues:

$ python3
Python 3.8.10 (default, Mar 13 2023, 10:26:41)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorrt as trt
>>> trt.__version__
'8.5.2.2'

Thanks.

[pavanrshetty361998]

Yes, I tried with Python3 as well, but I am unable to utilize any attribute as earlier. Below is the screenshot of the same

[AastaLLL]

Hi,

Could you try it outside of the virtual environment?
Thanks.

[pavanrshetty361998]

yes, it works outside the virtual environment, but not within the environment, any suggestions to fix this?

[AastaLLL]

Hi,

Could you try to create the virtual environment with --system-site-packages?
Thanks.

[pavanrshetty361998]

While this thread is active, I have another question

My Jetson Orin Nano now would not go to the login screen. As soon as I power it up, the NVIDIA screen pops up, after that it is completely blank

Unable to reach the login screen

[kayccc]

@pavanrshetty361998

My Jetson Orin Nano now would not go to the login screen. As soon as I power it up, the NVIDIA screen pops up, after that it is completely blank

Unable to reach the login screen

Sorry for the late response, is this still an issue to support? Thanks

[pavanrshetty361998]

well, I reflashed my Jetson and started all over again. It’s not an issue anymore