Is it possible to install TensorRT8 on JetPack 4.5.1?

SteveAtSWS · January 24, 2023, 8:47pm

See updated replies below. I’ve narrowed down the search while poking around for a solution.

Old initial post:

We have some Jetson Nano’s on Jetpack 4.5.1 in service that are running an older OpenCV application and we’ve been working on moving the application over to Deepstream. We’ve installed Deepstream 6.0 Early Access since the regular 6.0 was throwing errors during installation due to missing dependencies.

When we go to run the detections application, we get an error that libnvinfer_plugin.so.8 is missing. I can see in /usr/lib/aarch64-linux-gnu that libnvinfer_plugin.so.7 and its associated versions are running.

We have another older Nano that we’ve been doing development on that our developer didn’t document how they got the .8 version on there to work.

What’s the “right way” to get Deepstream 6.0 Early Access to work properly on JetPack 4.5.1?

The reason we want to stick with 4.5.1 on these existing machines is they’re currently in service at some of our customer’s locations and there’s no feasible way take them out of service to swap them over to 4.6.3 right now.

SteveAtSWS · January 25, 2023, 2:58pm

I did quite a bit more digging about the deepstream versions that are/aren’t working.
On the 4.5.1 Nano that we first got working, the version file in /opt/nvidia/deepstream/deepstream-6.0 shows:

Version: 6.0
GCID: 28816764
EABI:
DATE: Tue Oct 5 11:17:37 UTC 2021

But, the 6.0 Early Access (which only shows “Early Access” on the download page, nowhere else) has the version as:

Version: 6.0
GCID: 27423701
EABI:
DATE: Wed May 26 06:35:53 UTC 2021

So, we DO have a different version working on the original 4.5.1 nano. We need the GCID: 28816764 version and solid install instructions on how to get that version working on our existing 4.5.1 Nano’s in the field.

SteveAtSWS · January 26, 2023, 3:24pm

It looks like it’s just TensorRT 8 missing is the issue. Is it possible to install it on Jetpack 4.5.1? I still don’t know how our developer got it to install in the first place when we set up this test system in August of 2022.

SteveAtSWS · January 26, 2023, 6:55pm

Is there any command I can use to install this version of TensorRT 8 on my Nano JetPack 4.5.1?

AastaLLL · January 30, 2023, 3:31am

Hi,

Unfortunately, there are some dependencies between libraries and the driver.
To use Deepstream 6.0, please set up Nano with JetPack 4.6.2.

If JetPack 4.5.1 is essential for you, please use TensorRT 7 and Deepstream 5.0 instead.
You can find the supported software in detail below:

Thanks.

SteveAtSWS · January 30, 2023, 1:24pm

There is a version of Deepstream 6.0 Archived labelled “Early Access” that is able to work on JetPack 4.5.1. That is the version we’re using on the original Nano with TensorRT 8.0.1-1+cuda10.2. I don’t know how we got TensorRT 8.0.1-1+cuda10.2 on that machine since the original Developer didn’t document it. Deepstream isn’t the issue, we just need instructions on how to install TensorRT 8.0.1-1+cuda10.2 on our other machines in production. We can not set up our current machines in production with JetPack 4.6.2 since they’re already in service in remote locations.

Re-stating my questions for clarity:

The original test machine was set up August 16th 2022. Was there a change to the sdkmanager or jetpack repo’s around that time that changed TensorRT8 to TensorRT7 in Jetpack 4.5.1?
Since we already have one Nano on 4.5.1 with TensorRT 8.0.1-1+cuda10.2, is it possible to copy those currently working 8.0.1-1+cuda10.2 libraries over to other Nano’s on 4.5.1?

If I search for “8.0.1-1+cuda10.2” in the Jetson Repo I can see there are libnvinfer .deb’s for arm64. I believe these are what I’m looking for. Can someone confirm this?

AastaLLL · January 31, 2023, 3:45am

Hi,

Could you do some experiments for us?

1. Please help to confirm that the Nano with TensorRT 8.0 is really using BSP r32.5 (JetPack 4.5.1).
In case the BSP is already upgraded to the newer via OTA.

$ cat /etc/nv_tegra_release

2. If the BSP is r32.5, please try the below command to check if TensorRT 8 can work well.

$ /usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/resnet50/ResNet50.onnx

Thanks.

SteveAtSWS · January 31, 2023, 3:52am

output of cat /etc/nv_tegra_release:

# R32 (release), REVISION: 5.2, GCID: 27767740, BOARD: t210ref, EABI: aarch64, DATE: Fri Jul  9 16:01:52 UTC 2021

output of /usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/resnet50/ResNet50.onnx:

tess@tess-test:~$ /usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/resnet50/ResNet50.onnx
&&&& RUNNING TensorRT.trtexec [TensorRT v8001] # /usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/resnet50/ResNet50.onnx
[01/30/2023-22:49:03] [I] === Model Options ===
[01/30/2023-22:49:03] [I] Format: ONNX
[01/30/2023-22:49:03] [I] Model: /usr/src/tensorrt/data/resnet50/ResNet50.onnx
[01/30/2023-22:49:03] [I] Output:
[01/30/2023-22:49:03] [I] === Build Options ===
[01/30/2023-22:49:03] [I] Max batch: explicit
[01/30/2023-22:49:03] [I] Workspace: 16 MiB
[01/30/2023-22:49:03] [I] minTiming: 1
[01/30/2023-22:49:03] [I] avgTiming: 8
[01/30/2023-22:49:03] [I] Precision: FP32
[01/30/2023-22:49:03] [I] Calibration: 
[01/30/2023-22:49:03] [I] Refit: Disabled
[01/30/2023-22:49:03] [I] Sparsity: Disabled
[01/30/2023-22:49:03] [I] Safe mode: Disabled
[01/30/2023-22:49:03] [I] Restricted mode: Disabled
[01/30/2023-22:49:03] [I] Save engine: 
[01/30/2023-22:49:03] [I] Load engine: 
[01/30/2023-22:49:03] [I] NVTX verbosity: 0
[01/30/2023-22:49:03] [I] Tactic sources: Using default tactic sources
[01/30/2023-22:49:03] [I] timingCacheMode: local
[01/30/2023-22:49:03] [I] timingCacheFile: 
[01/30/2023-22:49:03] [I] Input(s)s format: fp32:CHW
[01/30/2023-22:49:03] [I] Output(s)s format: fp32:CHW
[01/30/2023-22:49:03] [I] Input build shapes: model
[01/30/2023-22:49:03] [I] Input calibration shapes: model
[01/30/2023-22:49:03] [I] === System Options ===
[01/30/2023-22:49:03] [I] Device: 0
[01/30/2023-22:49:03] [I] DLACore: 
[01/30/2023-22:49:03] [I] Plugins:
[01/30/2023-22:49:03] [I] === Inference Options ===
[01/30/2023-22:49:03] [I] Batch: Explicit
[01/30/2023-22:49:03] [I] Input inference shapes: model
[01/30/2023-22:49:03] [I] Iterations: 10
[01/30/2023-22:49:03] [I] Duration: 3s (+ 200ms warm up)
[01/30/2023-22:49:03] [I] Sleep time: 0ms
[01/30/2023-22:49:03] [I] Streams: 1
[01/30/2023-22:49:03] [I] ExposeDMA: Disabled
[01/30/2023-22:49:03] [I] Data transfers: Enabled
[01/30/2023-22:49:03] [I] Spin-wait: Disabled
[01/30/2023-22:49:03] [I] Multithreading: Disabled
[01/30/2023-22:49:03] [I] CUDA Graph: Disabled
[01/30/2023-22:49:03] [I] Separate profiling: Disabled
[01/30/2023-22:49:03] [I] Time Deserialize: Disabled
[01/30/2023-22:49:03] [I] Time Refit: Disabled
[01/30/2023-22:49:03] [I] Skip inference: Disabled
[01/30/2023-22:49:03] [I] Inputs:
[01/30/2023-22:49:03] [I] === Reporting Options ===
[01/30/2023-22:49:03] [I] Verbose: Disabled
[01/30/2023-22:49:03] [I] Averages: 10 inferences
[01/30/2023-22:49:03] [I] Percentile: 99
[01/30/2023-22:49:03] [I] Dump refittable layers:Disabled
[01/30/2023-22:49:03] [I] Dump output: Disabled
[01/30/2023-22:49:03] [I] Profile: Disabled
[01/30/2023-22:49:03] [I] Export timing to JSON file: 
[01/30/2023-22:49:03] [I] Export output to JSON file: 
[01/30/2023-22:49:03] [I] Export profile to JSON file: 
[01/30/2023-22:49:03] [I] 
[01/30/2023-22:49:03] [I] === Device Information ===
[01/30/2023-22:49:03] [I] Selected Device: NVIDIA Tegra X1
[01/30/2023-22:49:03] [I] Compute Capability: 5.3
[01/30/2023-22:49:03] [I] SMs: 1
[01/30/2023-22:49:03] [I] Compute Clock Rate: 0.9216 GHz
[01/30/2023-22:49:03] [I] Device Global Memory: 3956 MiB
[01/30/2023-22:49:03] [I] Shared Memory per SM: 64 KiB
[01/30/2023-22:49:03] [I] Memory Bus Width: 64 bits (ECC disabled)
[01/30/2023-22:49:03] [I] Memory Clock Rate: 0.01275 GHz
[01/30/2023-22:49:03] [I] 
[01/30/2023-22:49:03] [I] TensorRT version: 8001
[01/30/2023-22:49:05] [I] [TRT] [MemUsageChange] Init CUDA: CPU +202, GPU +0, now: CPU 205, GPU 3626 (MiB)
[01/30/2023-22:49:05] [I] Start parsing network model
[01/30/2023-22:49:05] [I] [TRT] ----------------------------------------------------------------
[01/30/2023-22:49:05] [I] [TRT] Input filename:   /usr/src/tensorrt/data/resnet50/ResNet50.onnx
[01/30/2023-22:49:05] [I] [TRT] ONNX IR version:  0.0.3
[01/30/2023-22:49:05] [I] [TRT] Opset version:    9
[01/30/2023-22:49:05] [I] [TRT] Producer name:    onnx-caffe2
[01/30/2023-22:49:05] [I] [TRT] Producer version: 
[01/30/2023-22:49:05] [I] [TRT] Domain:           
[01/30/2023-22:49:05] [I] [TRT] Model version:    0
[01/30/2023-22:49:05] [I] [TRT] Doc string:       
[01/30/2023-22:49:05] [I] [TRT] ----------------------------------------------------------------
[01/30/2023-22:49:06] [W] [TRT] onnx2trt_utils.cpp:364: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[01/30/2023-22:49:06] [I] Finish parsing network model
[01/30/2023-22:49:06] [I] [TRT] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 304, GPU 3521 (MiB)
[01/30/2023-22:49:06] [I] [TRT] [MemUsageSnapshot] Builder begin: CPU 304 MiB, GPU 3519 MiB
[01/30/2023-22:49:06] [I] [TRT] ---------- Layers Running on DLA ----------
[01/30/2023-22:49:06] [I] [TRT] ---------- Layers Running on GPU ----------
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/conv1_1 + node_of_gpu_0/res_conv1_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/pool1_1
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res2_0_branch2a_1 + node_of_gpu_0/res2_0_branch2a_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res2_0_branch2b_1 + node_of_gpu_0/res2_0_branch2b_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res2_0_branch1_1
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res2_0_branch2c_1 + node_of_gpu_0/res2_0_branch2c_bn_2 + node_of_gpu_0/res2_0_branch2c_bn_3
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res2_1_branch2a_1 + node_of_gpu_0/res2_1_branch2a_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res2_1_branch2b_1 + node_of_gpu_0/res2_1_branch2b_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res2_1_branch2c_1 + node_of_gpu_0/res2_1_branch2c_bn_2 + node_of_gpu_0/res2_1_branch2c_bn_3
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res2_2_branch2a_1 + node_of_gpu_0/res2_2_branch2a_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res2_2_branch2b_1 + node_of_gpu_0/res2_2_branch2b_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res2_2_branch2c_1 + node_of_gpu_0/res2_2_branch2c_bn_2 + node_of_gpu_0/res2_2_branch2c_bn_3
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res3_0_branch2a_1 + node_of_gpu_0/res3_0_branch2a_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res3_0_branch2b_1 + node_of_gpu_0/res3_0_branch2b_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res3_0_branch1_1
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res3_0_branch2c_1 + node_of_gpu_0/res3_0_branch2c_bn_2 + node_of_gpu_0/res3_0_branch2c_bn_3
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res3_1_branch2a_1 + node_of_gpu_0/res3_1_branch2a_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res3_1_branch2b_1 + node_of_gpu_0/res3_1_branch2b_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res3_1_branch2c_1 + node_of_gpu_0/res3_1_branch2c_bn_2 + node_of_gpu_0/res3_1_branch2c_bn_3
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res3_2_branch2a_1 + node_of_gpu_0/res3_2_branch2a_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res3_2_branch2b_1 + node_of_gpu_0/res3_2_branch2b_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res3_2_branch2c_1 + node_of_gpu_0/res3_2_branch2c_bn_2 + node_of_gpu_0/res3_2_branch2c_bn_3
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res3_3_branch2a_1 + node_of_gpu_0/res3_3_branch2a_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res3_3_branch2b_1 + node_of_gpu_0/res3_3_branch2b_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res3_3_branch2c_1 + node_of_gpu_0/res3_3_branch2c_bn_2 + node_of_gpu_0/res3_3_branch2c_bn_3
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res4_0_branch2a_1 + node_of_gpu_0/res4_0_branch2a_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res4_0_branch2b_1 + node_of_gpu_0/res4_0_branch2b_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res4_0_branch1_1
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res4_0_branch2c_1 + node_of_gpu_0/res4_0_branch2c_bn_2 + node_of_gpu_0/res4_0_branch2c_bn_3
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res4_1_branch2a_1 + node_of_gpu_0/res4_1_branch2a_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res4_1_branch2b_1 + node_of_gpu_0/res4_1_branch2b_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res4_1_branch2c_1 + node_of_gpu_0/res4_1_branch2c_bn_2 + node_of_gpu_0/res4_1_branch2c_bn_3
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res4_2_branch2a_1 + node_of_gpu_0/res4_2_branch2a_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res4_2_branch2b_1 + node_of_gpu_0/res4_2_branch2b_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res4_2_branch2c_1 + node_of_gpu_0/res4_2_branch2c_bn_2 + node_of_gpu_0/res4_2_branch2c_bn_3
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res4_3_branch2a_1 + node_of_gpu_0/res4_3_branch2a_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res4_3_branch2b_1 + node_of_gpu_0/res4_3_branch2b_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res4_3_branch2c_1 + node_of_gpu_0/res4_3_branch2c_bn_2 + node_of_gpu_0/res4_3_branch2c_bn_3
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res4_4_branch2a_1 + node_of_gpu_0/res4_4_branch2a_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res4_4_branch2b_1 + node_of_gpu_0/res4_4_branch2b_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res4_4_branch2c_1 + node_of_gpu_0/res4_4_branch2c_bn_2 + node_of_gpu_0/res4_4_branch2c_bn_3
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res4_5_branch2a_1 + node_of_gpu_0/res4_5_branch2a_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res4_5_branch2b_1 + node_of_gpu_0/res4_5_branch2b_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res4_5_branch2c_1 + node_of_gpu_0/res4_5_branch2c_bn_2 + node_of_gpu_0/res4_5_branch2c_bn_3
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res5_0_branch2a_1 + node_of_gpu_0/res5_0_branch2a_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res5_0_branch2b_1 + node_of_gpu_0/res5_0_branch2b_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res5_0_branch1_1
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res5_0_branch2c_1 + node_of_gpu_0/res5_0_branch2c_bn_2 + node_of_gpu_0/res5_0_branch2c_bn_3
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res5_1_branch2a_1 + node_of_gpu_0/res5_1_branch2a_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res5_1_branch2b_1 + node_of_gpu_0/res5_1_branch2b_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res5_1_branch2c_1 + node_of_gpu_0/res5_1_branch2c_bn_2 + node_of_gpu_0/res5_1_branch2c_bn_3
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res5_2_branch2a_1 + node_of_gpu_0/res5_2_branch2a_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res5_2_branch2b_1 + node_of_gpu_0/res5_2_branch2b_bn_2
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/res5_2_branch2c_1 + node_of_gpu_0/res5_2_branch2c_bn_2 + node_of_gpu_0/res5_2_branch2c_bn_3
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/pool5_1
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] node_of_gpu_0/pred_1
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] (Unnamed Layer* 176) [Shuffle] + (Unnamed Layer* 177) [Shuffle]
[01/30/2023-22:49:06] [I] [TRT] [GpuLayer] (Unnamed Layer* 178) [Softmax]
[01/30/2023-22:49:07] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +158, GPU +72, now: CPU 552, GPU 3622 (MiB)
[01/30/2023-22:49:09] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +241, GPU -9, now: CPU 793, GPU 3613 (MiB)
[01/30/2023-22:49:09] [W] [TRT] Detected invalid timing cache, setup a local cache instead
[01/30/2023-22:49:17] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
[01/30/2023-22:50:40] [I] [TRT] Detected 1 inputs and 1 output network tensors.
[01/30/2023-22:50:42] [I] [TRT] Total Host Persistent Memory: 131296
[01/30/2023-22:50:42] [I] [TRT] Total Device Persistent Memory: 82422784
[01/30/2023-22:50:42] [I] [TRT] Total Scratch Memory: 0
[01/30/2023-22:50:42] [I] [TRT] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 98 MiB, GPU 192 MiB
[01/30/2023-22:50:42] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +3, now: CPU 1045, GPU 3669 (MiB)
[01/30/2023-22:50:42] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +1, GPU +1, now: CPU 1046, GPU 3670 (MiB)
[01/30/2023-22:50:42] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1045, GPU 3671 (MiB)
[01/30/2023-22:50:42] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1045, GPU 3671 (MiB)
[01/30/2023-22:50:42] [I] [TRT] [MemUsageSnapshot] Builder end: CPU 1045 MiB, GPU 3671 MiB
[01/30/2023-22:50:43] [I] [TRT] Loaded engine size: 121 MB
[01/30/2023-22:50:43] [I] [TRT] [MemUsageSnapshot] deserializeCudaEngine begin: CPU 1077 MiB, GPU 3771 MiB
[01/30/2023-22:50:44] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +1, now: CPU 1077, GPU 3778 (MiB)
[01/30/2023-22:50:44] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +1, now: CPU 1077, GPU 3779 (MiB)
[01/30/2023-22:50:44] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1077, GPU 3779 (MiB)
[01/30/2023-22:50:44] [I] [TRT] [MemUsageSnapshot] deserializeCudaEngine end: CPU 1077 MiB, GPU 3779 MiB
[01/30/2023-22:50:44] [I] Engine built in 100.338 sec.
[01/30/2023-22:50:44] [I] [TRT] [MemUsageSnapshot] ExecutionContext creation begin: CPU 856 MiB, GPU 3622 MiB
[01/30/2023-22:50:44] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +1, now: CPU 856, GPU 3622 (MiB)
[01/30/2023-22:50:44] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +1, now: CPU 856, GPU 3623 (MiB)
[01/30/2023-22:50:44] [I] [TRT] [MemUsageSnapshot] ExecutionContext creation end: CPU 857 MiB, GPU 3690 MiB
[01/30/2023-22:50:44] [I] Created input binding for gpu_0/data_0 with dimensions 1x3x224x224
[01/30/2023-22:50:44] [I] Created output binding for gpu_0/softmax_1 with dimensions 1x1000
[01/30/2023-22:50:44] [I] Starting inference
[01/30/2023-22:50:47] [I] Warmup completed 4 queries over 200 ms
[01/30/2023-22:50:47] [I] Timing trace has 56 queries over 3.08943 s
[01/30/2023-22:50:47] [I] 
[01/30/2023-22:50:47] [I] === Trace details ===
[01/30/2023-22:50:47] [I] Trace averages of 10 runs:
[01/30/2023-22:50:47] [I] Average on 10 runs - GPU latency: 55.6723 ms - Host latency: 55.7345 ms (end to end 55.7758 ms, enqueue 3.94339 ms)
[01/30/2023-22:50:47] [I] Average on 10 runs - GPU latency: 55.139 ms - Host latency: 55.2022 ms (end to end 55.2432 ms, enqueue 4.25462 ms)
[01/30/2023-22:50:47] [I] Average on 10 runs - GPU latency: 55.1018 ms - Host latency: 55.1644 ms (end to end 55.2058 ms, enqueue 3.00271 ms)
[01/30/2023-22:50:47] [I] Average on 10 runs - GPU latency: 54.8281 ms - Host latency: 54.8893 ms (end to end 54.9259 ms, enqueue 2.22249 ms)
[01/30/2023-22:50:47] [I] Average on 10 runs - GPU latency: 54.7418 ms - Host latency: 54.8026 ms (end to end 54.8458 ms, enqueue 2.19004 ms)
[01/30/2023-22:50:47] [I] 
[01/30/2023-22:50:47] [I] === Performance summary ===
[01/30/2023-22:50:47] [I] Throughput: 18.1263 qps
[01/30/2023-22:50:47] [I] Latency: min = 54.4517 ms, max = 58.1728 ms, mean = 55.1214 ms, median = 55.0273 ms, percentile(99%) = 58.1728 ms
[01/30/2023-22:50:47] [I] End-to-End Host Latency: min = 54.4902 ms, max = 58.2183 ms, mean = 55.1679 ms, median = 55.0585 ms, percentile(99%) = 58.2183 ms
[01/30/2023-22:50:47] [I] Enqueue Time: min = 2.00269 ms, max = 16.5146 ms, mean = 3.04357 ms, median = 2.31653 ms, percentile(99%) = 16.5146 ms
[01/30/2023-22:50:47] [I] H2D Latency: min = 0.0561523 ms, max = 0.0667725 ms, mean = 0.0589101 ms, median = 0.0580444 ms, percentile(99%) = 0.0667725 ms
[01/30/2023-22:50:47] [I] GPU Compute Time: min = 54.3936 ms, max = 58.1107 ms, mean = 55.0595 ms, median = 54.9636 ms, percentile(99%) = 58.1107 ms
[01/30/2023-22:50:47] [I] D2H Latency: min = 0.00195312 ms, max = 0.00354004 ms, mean = 0.00301089 ms, median = 0.0030365 ms, percentile(99%) = 0.00354004 ms
[01/30/2023-22:50:47] [I] Total Host Walltime: 3.08943 s
[01/30/2023-22:50:47] [I] Total GPU Compute Time: 3.08333 s
[01/30/2023-22:50:47] [I] Explanations of the performance metrics are printed in the verbose logs.
[01/30/2023-22:50:47] [I] 
&&&& PASSED TensorRT.trtexec [TensorRT v8001] # /usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/resnet50/ResNet50.onnx
[01/30/2023-22:50:47] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 856, GPU 3699 (MiB)

AastaLLL · February 1, 2023, 6:33am

Hi,

You can download the packages at the below link and install them via dpkg.

https://repo.download.nvidia.com/jetson/

$ sudo dpkg -i libnvinfer8_8.0.1-1+cuda10.2_arm64.deb
$ sudo dpkg -i libnvinfer-dev_8.0.1-1+cuda10.2_arm64.deb
$ sudo dpkg -i libnvinfer-plugin8_8.0.1-1+cuda10.2_arm64.deb
$ sudo dpkg -i libnvinfer-plugin-dev_8.0.1-1+cuda10.2_arm64.deb
$ sudo dpkg -i libnvonnxparsers8_8.0.1-1+cuda10.2_arm64.deb
$ sudo dpkg -i libnvonnxparsers-dev_8.0.1-1+cuda10.2_arm64.deb
$ sudo dpkg -i libnvparsers8_8.0.1-1+cuda10.2_arm64.deb
$ sudo dpkg -i libnvparsers-dev_8.0.1-1+cuda10.2_arm64.deb
$ sudo dpkg -i libnvinfer-bin_8.0.1-1+cuda10.2_arm64.deb
$ sudo dpkg -i libnvinfer-doc_8.0.1-1+cuda10.2_all.deb
$ sudo dpkg -i libnvinfer-samples_8.0.1-1+cuda10.2_all.deb
$ sudo dpkg -i tensorrt_8.0.1.6-1+cuda10.2_arm64.deb
$ sudo dpkg -i python3-libnvinfer_8.0.1-1+cuda10.2_arm64.deb
$ sudo dpkg -i python3-libnvinfer-dev_8.0.1-1+cuda10.2_arm64.deb
$ sudo dpkg -i graphsurgeon-tf_8.0.1-1+cuda10.2_arm64.deb
$ sudo dpkg -i uff-converter-tf_8.0.1-1+cuda10.2_arm64.deb

But please noted that we don’t run a full test on such a combination.
It might contain some unknown issues.

Thanks.

system · February 21, 2023, 1:12pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.