Please provide the following info (tick the boxes after creating this topic): Software Version
[*] DRIVE OS 6.0.8.1
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other
Target Operating System
[*] Linux
QNX
other
Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)
DRIVE AGX Orin Developer Kit (940-63710-0010-200)
DRIVE AGX Orin Developer Kit (940-63710-0010-100)
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
[*] other (Orin IGX)
SDK Manager Version
1.9.3.10904
other
Host Machine Version
[*] native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other
I use command "sudo dpkg -i ./nv-tensorrt-repo-ubuntu2004-cuda11.4-trt8.6.12.4-d6l-target-ga-20231014_1-1_arm64.deb " to install TRT on Drive Os.
After using command “sudo make TARGET=aarch64” in /usr/src/tensorrt/samples, I get the trtexec.
Howerver, when I use trtexec to convert onnx model, the following error occurs:
Hi
You need to generate TRT model on hardware where you want to use(may be it is x86 host or Drive platform). You can run inference only when there are available resources. If you want to run two inference models in parallel, you can either schedule them on different resources. For example, if you take DRIVE AGX Orin platform, it has a iGPU and DLA engines. So you generate GPU/DLA TRT model using trtexec or TRT APIs to run the model on the specific engine. In case, you want to run two inference tasks on same GPU, you can use different cudaStreams for each task to achieve it.
But, note that, launching two tasks in parallel does not mean they run in parallel. if the GPU is fully used by one task, other task can not run. You can actually verify if two GPU tasks are running in nsys trace.
I did not understand your test. what is the platform your are intended to run your model? Does the testing platform already has any other GPU/DLA workload?
Please note that this platform is exclusively for developers who are part of the NVIDIA DRIVE™ AGX SDK Developer Program.
Are your using DRIVE AGX Orin platform or Jetson platform?
Thanks for your reply.
The ONNX model was created on x86 machines and wants to deploy it on the Orin IGX platform (the operating system is drive os).
The testing platform does not have any other GPU/DLA workloads.
Please note that the device I’m using is the Orin IGX with Drive OS.
Dear @liangjz,
May I know the used platform? Is it DRIVE AGX Orin devkit?
If so, you don’t need to need to install TensorRT packages on target. It gets installed automatically when flashing DRIVE OS. trtexec can be found at /usr/src/tensorrt/bin. Could you please check using that?
Also, DRIVE OS 6.0.8.1 comes with TensorRT 8.6.11.4. How did you get /nv-tensorrt-repo-ubuntu2004-cuda11.4-trt8.6.12.4-d6l-target-ga-20231014_1-1_arm64.deb ?
nvidia@tegra-ubuntu:/usr/src/tensorrt/bin$ sudo dpkg -l | grep TensorRT
[sudo] password for nvidia:
ii libnvinfer-bin 8.6.11.4-1+cuda11.4 arm64 TensorRT binaries
ii libnvinfer-dispatch8 8.6.11.4-1+cuda11.4 arm64 TensorRT dispatch runtime library
ii libnvinfer-lean8 8.6.11.4-1+cuda11.4 arm64 TensorRT lean runtime library
ii libnvinfer-plugin8 8.6.11.4-1+cuda11.4 arm64 TensorRT plugin libraries
ii libnvinfer-vc-plugin8 8.6.11.4-1+cuda11.4 arm64 TensorRT vc-plugin library
ii libnvinfer8 8.6.11.4-1+cuda11.4 arm64 TensorRT runtime libraries
ii libnvonnxparsers8 8.6.11.4-1+cuda11.4 arm64 TensorRT ONNX libraries
ii libnvparsers8 8.6.11.4-1+cuda11.4 arm64 TensorRT parsers libraries
ii python3-libnvinfer 8.6.11.4-1+cuda11.4 arm64 Python 3 bindings for TensorRT standard runtime
