TensorRT 8.4.2 for Ubuntu 18.04 arm64

Description

I was wondering if there were any .deb files or ways to install TensorRT 8.4.2 onto my Jetson TX2 device. Or if it is even possible. I am preferably looking for a .deb file because that’s what I used to get the latest version of TensorRT onto Google Colab, but tar files would work fine too, it’s just the hardware specifications that are tricky right now.

Environment

TensorRT Version: Currently 8.2 but looking to get it to 8.4.2
GPU Type: Jetson TX2
Nvidia Driver Version: unknown
CUDA Version: 10.2
CUDNN Version: 8.2
Operating System + Version: Ubuntu 18.04
Python Version (if applicable): Python 3.6.9
TensorFlow Version (if applicable): NA
PyTorch Version (if applicable): NA
Baremetal or Container (if container which image + tag): Flash (Jetpack 4.6.2)

This is a repost, please refer to:

Yes and no-this time I am wondering if it is possible to build from source or even possible to begin with?

The TensorRT libraries themselves aren’t open-source so you couldn’t just build them from source. Further, they have dependencies on specific versions of cuDNN, CUDA, and the GPU drivers in the L4T that come with JetPack. Hence the version of TensorRT is bundled with JetPack.

Then why the installation instructions on the github?

The GitHub repo (https://github.com/nvidia/tensorrt) contains open-source samples, plugins, and parsers, but not the actual TensorRT runtime libraries themselves:

This repository contains the Open Source Software (OSS) components of NVIDIA TensorRT. Included are the sources for TensorRT plugins and parsers (Caffe and ONNX), as well as sample applications demonstrating usage and capabilities of the TensorRT platform. These open source software components are a subset of the TensorRT General Availability (GA) release with some extensions and bug-fixes.

So will following their installation processes give me TensorRT 8.4.2?

@dusty_nv On the colab that I managed to get working I used this-
!wget -O nv-tensorrt-repo-ubuntu1804-cuda10.0-trt5.1.2.2-rc-20190227_1-1_amd64.deb Dropbox - nv-tensorrt-repo-ubuntu1804-cuda10.0-trt5.1.2.2-rc-20190227_1-1_amd64.deb - Simplify your life

This gave me TensorRT 8.4.2, is there anything like that for arm architectures?
edit: That command and a couple more but thats the thing I need most right now

No, it will build you the latest samples and such, but your actual TensorRT version will remain the same.

There aren’t public TensorRT debs for Jetson (aside from what is in the JetPack apt servers, which are the same debs that already come installed with JetPack). I believe there are ones for arm64 SBSA, but those are for ARM servers with discrete GPU cards and not the integrated GPU that Jetson uses.

Ok. With that, I created an engine on colab which was in TensorRT 8.4.2, I take it that I won’t be able to use it in my host then?

And I’m going to guess that NVIDIA won’t ever support those versions in their Jetpack for Jetson TX2?@dusty_nv

TensorRT engines aren’t portable across different GPU devices or across different versions of TensorRT. For example on TX2, the TensorRT engines need to be built on another TX2 running the same version of TensorRT.

Sorry, Jetson TX2 will remain on JetPack 4.6.x and won’t be receiving newer versions of TensorRT. For more information, please refer to the Jetson Software Roadmap.

Hmm, because then there is really a lack of understanding. I was told on the github that I needed to go onto a different PC, make the engine there and test it and whatnot then deploy it on the Jetson. Ofc I didn’t have another PC with GPU capabilities so I did it on colab successfully, but still. Really weird. Also as for the Jetson tx2 tensorrt versions thanks for letting me know

Why don’t you verify with detectron2 sample on your PC and check what evaluation and inference results are following Detectron2 readme? Inference in the sample is purely for checking purposes not real world inference. This python inference will significantly slow you down.

The main reason you don’t get proper results is your TRT version is old 8.2.1. In detectron2 readme is says that TRT must be >= 8.4.1, I also mentioned it to you in your initial issue.

Getting back to proper inference. Here is what you should do: convert detectron2 model to TensorRT on your PC, evaluate on your PC (using detectron2 code and TRT 8.4.1) and check if you are getting 40.2 mAP, you can also infer a couple images after that if you want. After that you should figure out how to install TRT 8.4.1 on Jetson device replacing the default one in JetPack. This was originally mentioned. After you have done that you should look into DeepStream SDK and build efficient inference pipeline (using converted on your PC ONNX and built from it TRT engine on a Jetson). Please don’t ask me how to use DeepStream, I have not done that yet, so I simply do not know. With DeepStream you will get best inference perf possible which is critical if you want to run on Jetson. In addition, I recommend using DeepStream C++ SDK to get best perf, you will loose some perf due to python interpreter if using python SDK.
from azhurkevich in issue -Inference accuracy · Issue #2233 · NVIDIA/TensorRT · GitHub

This gives me a little bit of confusion @dusty_nv and is very misleading.

Unfortunately I don’t think the other person understood that it’s not officially supported for individuals to independently upgrade the version of TensorRT in JetPack, outside of upgrading JetPack itself.

What about unofficially??? I get what you mean, but does that mean my project is impossible? Also it’d be useful if you communicated that to him

Unofficially, I haven’t tried it and doubt that it would work installing the ARM64 SBSA TensorRT debs that you can download from the TensorRT page, because they probably aren’t built with the Jetson integrated GPU architectures in mind and may have dependencies on newer versions of other libraries (like CUDA and the GPU drivers) that aren’t available in JetPack 4.x.

Alright - I tried some things that I found online on something NVIDIA but my browser doesnt save history (RIP) but it worked. I basically had to upgrade the libnvinfer plugin or smt (i totally forget what it was i think that) and it worked for creating the engine and what not. It also worked for inference (not that it was super accurate but mby a 35 MAP (detectron2). Thanks anyways for the help.