tensorRT plan file not working

jhpark7 · April 21, 2022, 8:08am

Description

I’m trying to do this post.
https://towardsdatascience.com/detecting-pedestrians-and-bikers-on-a-drone-with-jetson-xavier-93ce92e2c597

The export.cpp file of retinanet-example is also use tensorRT and convert onnx to plan file.
also it is going well.

But the model.plan file result of trtexec utility is not working.

I got model.plan file with this command.
sudo ./trtexec --onnx=stanford_resnext50.onnx --best --minShapes=1,3,1280,1280 --optShapes=16,3,1280,1280 --maxShapes=32,3,1280,1280 --saveEngine=model.plan

And this is the result.

/usr/src/tensorrt/bin$ sudo ./trtexec --onnx=stanford_resnext50.onnx --best --minShapes=1,3,1280,1280 --optShapes=16,3,1280,1280 --maxShapes=32,3,1280,1280 --saveEngine=model.plan
[sudo] password for openmakerlab-agx:
&&&& RUNNING TensorRT.trtexec # ./trtexec --onnx=stanford_resnext50.onnx --best --minShapes=1,3,1280,1280 --optShapes=16,3,1280,1280 --maxShapes=32,3,1280,1280 --saveEngine=model.plan
[04/21/2022-15:37:33] [I] === Model Options ===
[04/21/2022-15:37:33] [I] Format: ONNX
[04/21/2022-15:37:33] [I] Model: stanford_resnext50.onnx
[04/21/2022-15:37:33] [I] Output:
[04/21/2022-15:37:33] [I] === Build Options ===
[04/21/2022-15:37:33] [I] Max batch: explicit
[04/21/2022-15:37:33] [I] Workspace: 16 MB
[04/21/2022-15:37:33] [I] minTiming: 1
[04/21/2022-15:37:33] [I] avgTiming: 8
[04/21/2022-15:37:33] [I] Precision: FP32+FP16+INT8
[04/21/2022-15:37:33] [I] Calibration: Dynamic
[04/21/2022-15:37:33] [I] Safe mode: Disabled
[04/21/2022-15:37:33] [I] Save engine: model.plan
[04/21/2022-15:37:33] [I] Load engine:
[04/21/2022-15:37:33] [I] Builder Cache: Enabled
[04/21/2022-15:37:33] [I] NVTX verbosity: 0
[04/21/2022-15:37:33] [I] Inputs format: fp32:CHW
[04/21/2022-15:37:33] [I] Outputs format: fp32:CHW
[04/21/2022-15:37:33] [I] Input build shape: 16=
[04/21/2022-15:37:33] [I] Input build shape: 1280=1280+1280+1280
[04/21/2022-15:37:33] [I] Input build shape: 1=1
[04/21/2022-15:37:33] [I] Input build shape: 32=
[04/21/2022-15:37:33] [I] Input build shape: 3=3+3+3
[04/21/2022-15:37:33] [I] Input calibration shapes: model
[04/21/2022-15:37:33] [I] === System Options ===
[04/21/2022-15:37:33] [I] Device: 0
[04/21/2022-15:37:33] [I] DLACore:
[04/21/2022-15:37:33] [I] Plugins:
[04/21/2022-15:37:33] [I] === Inference Options ===
[04/21/2022-15:37:33] [I] Batch: Explicit
[04/21/2022-15:37:33] [I] Input inference shape: 3=3
[04/21/2022-15:37:33] [I] Input inference shape: 32=
[04/21/2022-15:37:33] [I] Input inference shape: 1=
[04/21/2022-15:37:33] [I] Input inference shape: 16=16
[04/21/2022-15:37:33] [I] Input inference shape: 1280=1280
[04/21/2022-15:37:33] [I] Iterations: 10
[04/21/2022-15:37:33] [I] Duration: 3s (+ 200ms warm up)
[04/21/2022-15:37:33] [I] Sleep time: 0ms
[04/21/2022-15:37:33] [I] Streams: 1
[04/21/2022-15:37:33] [I] ExposeDMA: Disabled
[04/21/2022-15:37:33] [I] Spin-wait: Disabled
[04/21/2022-15:37:33] [I] Multithreading: Disabled
[04/21/2022-15:37:33] [I] CUDA Graph: Disabled
[04/21/2022-15:37:33] [I] Skip inference: Disabled
[04/21/2022-15:37:33] [I] Inputs:
[04/21/2022-15:37:33] [I] === Reporting Options ===
[04/21/2022-15:37:33] [I] Verbose: Disabled
[04/21/2022-15:37:33] [I] Averages: 10 inferences
[04/21/2022-15:37:33] [I] Percentile: 99
[04/21/2022-15:37:33] [I] Dump output: Disabled
[04/21/2022-15:37:33] [I] Profile: Disabled
[04/21/2022-15:37:33] [I] Export timing to JSON file:
[04/21/2022-15:37:33] [I] Export output to JSON file:
[04/21/2022-15:37:33] [I] Export profile to JSON file:
[04/21/2022-15:37:33] [I]
Input filename: stanford_resnext50.onnx
ONNX IR version: 0.0.4
Opset version: 9
Producer name: pytorch
Producer version: 1.1
Domain:
Model version: 0
Doc string:
[04/21/2022-15:37:35] [W] [TRT] onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[04/21/2022-15:37:35] [W] [TRT] Calibrator is not being used. Users must provide dynamic range for all tensors that are not Int32.
[04/21/2022-15:37:35] [I] [TRT]
[04/21/2022-15:37:35] [I] [TRT] --------------- Layers running on DLA:
[04/21/2022-15:37:35] [I] [TRT]
[04/21/2022-15:37:35] [I] [TRT] --------------- Layers running on GPU:
[04/21/2022-15:37:35] [I] [TRT] (Unnamed Layer* 0) [Convolution] + (Unnamed Layer* 2) [Activation] + (Unnamed Layer* 3) [Pooling], (Unnamed Layer* 4) [Convolution] + (Unnamed Layer* 6) [Activation], (Unnamed Layer* 7) [Convolution] + (Unnamed Layer* 9) [Activation], (Unnamed Layer* 10) [Convolution], (Unnamed Layer* 12) [Convolution] + (Unnamed Layer* 14) [ElementWise] + (Unnamed Layer* 15) [Activation], (Unnamed Layer* 16) [Convolution] + (Unnamed Layer* 18) [Activation], (Unnamed Layer* 19) [Convolution] + (Unnamed Layer* 21) [Activation], (Unnamed Layer* 22) [Convolution] + (Unnamed Layer* 24) [ElementWise] + (Unnamed Layer* 25) [Activation], (Unnamed Layer* 26) [Convolution] + (Unnamed Layer* 28) [Activation], (Unnamed Layer* 29) [Convolution] + (Unnamed Layer* 31) [Activation], (Unnamed Layer* 32) [Convolution] + (Unnamed Layer* 34) [ElementWise] + (Unnamed Layer* 35) [Activation], (Unnamed Layer* 36) [Convolution] + (Unnamed Layer* 38) [Activation], (Unnamed Layer* 39) [Convolution] + (Unnamed Layer* 41) [Activation], (Unnamed Layer* 42) [Convolution], (Unnamed Layer* 44) [Convolution] + (Unnamed Layer* 46) [ElementWise] + (Unnamed Layer* 47) [Activation], (Unnamed Layer* 48) [Convolution] + (Unnamed Layer* 50) [Activation], (Unnamed Layer* 51) [Convolution] + (Unnamed Layer* 53) [Activation], (Unnamed Layer* 54) [Convolution] + (Unnamed Layer* 56) [ElementWise] + (Unnamed Layer* 57) [Activation], (Unnamed Layer* 58) [Convolution] + (Unnamed Layer* 60) [Activation], (Unnamed Layer* 61) [Convolution] + (Unnamed Layer* 63) [Activation], (Unnamed Layer* 64) [Convolution] + (Unnamed Layer* 66) [ElementWise] + (Unnamed Layer* 67) [Activation], (Unnamed Layer* 68) [Convolution] + (Unnamed Layer* 70) [Activation], (Unnamed Layer* 71) [Convolution] + (Unnamed Layer* 73) [Activation], (Unnamed Layer* 74) [Convolution] + (Unnamed Layer* 76) [ElementWise] + (Unnamed Layer* 77) [Activation], (Unnamed Layer* 78) [Convolution] + (Unnamed Layer* 80) [Activation], (Unnamed Layer* 81) [Convolution] + (Unnamed Layer* 83) [Activation], (Unnamed Layer* 84) [Convolution], (Unnamed Layer* 86) [Convolution] + (Unnamed Layer* 88) [ElementWise] + (Unnamed Layer* 89) [Activation], (Unnamed Layer* 90) [Convolution] + (Unnamed Layer* 92) [Activation], (Unnamed Layer* 93) [Convolution] + (Unnamed Layer* 95) [Activation], (Unnamed Layer* 96) [Convolution] + (Unnamed Layer* 98) [ElementWise] + (Unnamed Layer* 99) [Activation], (Unnamed Layer* 100) [Convolution] + (Unnamed Layer* 102) [Activation], (Unnamed Layer* 103) [Convolution] + (Unnamed Layer* 105) [Activation], (Unnamed Layer* 106) [Convolution] + (Unnamed Layer* 108) [ElementWise] + (Unnamed Layer* 109) [Activation], (Unnamed Layer* 110) [Convolution] + (Unnamed Layer* 112) [Activation], (Unnamed Layer* 113) [Convolution] + (Unnamed Layer* 115) [Activation], (Unnamed Layer* 116) [Convolution] + (Unnamed Layer* 118) [ElementWise] + (Unnamed Layer* 119) [Activation], (Unnamed Layer* 120) [Convolution] + (Unnamed Layer* 122) [Activation], (Unnamed Layer* 123) [Convolution] + (Unnamed Layer* 125) [Activation], (Unnamed Layer* 126) [Convolution] + (Unnamed Layer* 128) [ElementWise] + (Unnamed Layer* 129) [Activation], (Unnamed Layer* 130) [Convolution] + (Unnamed Layer* 132) [Activation], (Unnamed Layer* 133) [Convolution] + (Unnamed Layer* 135) [Activation], (Unnamed Layer* 136) [Convolution] + (Unnamed Layer* 138) [ElementWise] + (Unnamed Layer* 139) [Activation], (Unnamed Layer* 140) [Convolution] + (Unnamed Layer* 142) [Activation], (Unnamed Layer* 143) [Convolution] + (Unnamed Layer* 145) [Activation], (Unnamed Layer* 146) [Convolution], (Unnamed Layer* 148) [Convolution] + (Unnamed Layer* 150) [ElementWise] + (Unnamed Layer* 151) [Activation], (Unnamed Layer* 152) [Convolution] + (Unnamed Layer* 154) [Activation], (Unnamed Layer* 155) [Convolution] + (Unnamed Layer* 157) [Activation], (Unnamed Layer* 158) [Convolution] + (Unnamed Layer* 160) [ElementWise] + (Unnamed Layer* 161) [Activation], (Unnamed Layer* 162) [Convolution] + (Unnamed Layer* 164) [Activation], (Unnamed Layer* 165) [Convolution] + (Unnamed Layer* 167) [Activation], (Unnamed Layer* 168) [Convolution] + (Unnamed Layer* 170) [ElementWise] + (Unnamed Layer* 171) [Activation], (Unnamed Layer* 179) [Convolution], (Unnamed Layer* 257) [Convolution] + (Unnamed Layer* 258) [Activation] || (Unnamed Layer* 212) [Convolution] + (Unnamed Layer* 213) [Activation], (Unnamed Layer* 172) [Convolution], (Unnamed Layer* 184) [Convolution], (Unnamed Layer* 248) [Convolution] + (Unnamed Layer* 249) [Activation] || (Unnamed Layer* 203) [Convolution] + (Unnamed Layer* 204) [Activation], (Unnamed Layer* 174) [Resize], (Unnamed Layer* 180) [Activation], (Unnamed Layer* 173) [Convolution] + (Unnamed Layer* 175) [ElementWise], (Unnamed Layer* 183) [Convolution], (Unnamed Layer* 239) [Convolution] + (Unnamed Layer* 240) [Activation] || (Unnamed Layer* 194) [Convolution] + (Unnamed Layer* 195) [Activation], (Unnamed Layer* 177) [Resize], (Unnamed Layer* 205) [Convolution] + (Unnamed Layer* 206) [Activation], (Unnamed Layer* 250) [Convolution] + (Unnamed Layer* 251) [Activation], (Unnamed Layer* 214) [Convolution] + (Unnamed Layer* 215) [Activation], (Unnamed Layer* 259) [Convolution] + (Unnamed Layer* 260) [Activation], (Unnamed Layer* 181) [Convolution], (Unnamed Layer* 266) [Convolution] + (Unnamed Layer* 267) [Activation] || (Unnamed Layer* 221) [Convolution] + (Unnamed Layer* 222) [Activation], (Unnamed Layer* 176) [Convolution] + (Unnamed Layer* 178) [ElementWise], (Unnamed Layer* 223) [Convolution] + (Unnamed Layer* 224) [Activation], (Unnamed Layer* 268) [Convolution] + (Unnamed Layer* 269) [Activation], (Unnamed Layer* 261) [Convolution] + (Unnamed Layer* 262) [Activation], (Unnamed Layer* 216) [Convolution] + (Unnamed Layer* 217) [Activation], (Unnamed Layer* 252) [Convolution] + (Unnamed Layer* 253) [Activation], (Unnamed Layer* 207) [Convolution] + (Unnamed Layer* 208) [Activation], (Unnamed Layer* 182) [Convolution], (Unnamed Layer* 230) [Convolution] + (Unnamed Layer* 231) [Activation] || (Unnamed Layer* 185) [Convolution] + (Unnamed Layer* 186) [Activation], (Unnamed Layer* 196) [Convolution] + (Unnamed Layer* 197) [Activation], (Unnamed Layer* 241) [Convolution] + (Unnamed Layer* 242) [Activation], (Unnamed Layer* 243) [Convolution] + (Unnamed Layer* 244) [Activation], (Unnamed Layer* 198) [Convolution] + (Unnamed Layer* 199) [Activation], (Unnamed Layer* 187) [Convolution] + (Unnamed Layer* 188) [Activation], (Unnamed Layer* 232) [Convolution] + (Unnamed Layer* 233) [Activation], (Unnamed Layer* 209) [Convolution] + (Unnamed Layer* 210) [Activation], (Unnamed Layer* 254) [Convolution] + (Unnamed Layer* 255) [Activation], (Unnamed Layer* 218) [Convolution] + (Unnamed Layer* 219) [Activation], (Unnamed Layer* 263) [Convolution] + (Unnamed Layer* 264) [Activation], (Unnamed Layer* 270) [Convolution] + (Unnamed Layer* 271) [Activation], (Unnamed Layer* 225) [Convolution] + (Unnamed Layer* 226) [Activation], (Unnamed Layer* 227) [Convolution] + (Unnamed Layer* 228) [Activation], (Unnamed Layer* 272) [Convolution] + (Unnamed Layer* 273) [Activation], (Unnamed Layer* 265) [Convolution], (Unnamed Layer* 220) [Convolution], (Unnamed Layer* 278) [Activation], (Unnamed Layer* 256) [Convolution], (Unnamed Layer* 211) [Convolution], (Unnamed Layer* 277) [Activation], (Unnamed Layer* 234) [Convolution] + (Unnamed Layer* 235) [Activation], (Unnamed Layer* 189) [Convolution] + (Unnamed Layer* 190) [Activation], (Unnamed Layer* 200) [Convolution] + (Unnamed Layer* 201) [Activation], (Unnamed Layer* 245) [Convolution] + (Unnamed Layer* 246) [Activation], (Unnamed Layer* 247) [Convolution], (Unnamed Layer* 202) [Convolution], (Unnamed Layer* 276) [Activation], (Unnamed Layer* 191) [Convolution] + (Unnamed Layer* 192) [Activation], (Unnamed Layer* 236) [Convolution] + (Unnamed Layer* 237) [Activation], (Unnamed Layer* 274) [Convolution], (Unnamed Layer* 229) [Convolution], (Unnamed Layer* 279) [Activation], (Unnamed Layer* 238) [Convolution], (Unnamed Layer* 193) [Convolution], (Unnamed Layer* 275) [Activation],
[04/21/2022-15:37:44] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
[04/21/2022-15:52:53] [I] [TRT] Detected 1 inputs and 10 output network tensors.
[04/21/2022-15:52:55] [I] Starting inference threads
[04/21/2022-15:52:58] [I] Warmup completed 0 queries over 200 ms
[04/21/2022-15:52:58] [I] Timing trace has 0 queries over 3.29845 s
[04/21/2022-15:52:58] [I] Trace averages of 10 runs:
[04/21/2022-15:52:58] [I] Average on 10 runs - GPU latency: 111.904 ms - Host latency: 113.743 ms (end to end 113.753 ms, enqueue 4.17942 ms)
[04/21/2022-15:52:58] [I] Average on 10 runs - GPU latency: 111.9 ms - Host latency: 113.735 ms (end to end 113.747 ms, enqueue 4.09928 ms)
[04/21/2022-15:52:58] [I] Host Latency
[04/21/2022-15:52:58] [I] min: 113.5 ms (end to end 113.519 ms)
[04/21/2022-15:52:58] [I] max: 113.942 ms (end to end 113.955 ms)
[04/21/2022-15:52:58] [I] mean: 113.729 ms (end to end 113.739 ms)
[04/21/2022-15:52:58] [I] median: 113.722 ms (end to end 113.729 ms)
[04/21/2022-15:52:58] [I] percentile: 113.942 ms at 99% (end to end 113.955 ms at 99%)
[04/21/2022-15:52:58] [I] throughput: 0 qps
[04/21/2022-15:52:58] [I] walltime: 3.29845 s
[04/21/2022-15:52:58] [I] Enqueue Time
[04/21/2022-15:52:58] [I] min: 3.80664 ms
[04/21/2022-15:52:58] [I] max: 4.59406 ms
[04/21/2022-15:52:58] [I] median: 4.07446 ms
[04/21/2022-15:52:58] [I] GPU Compute
[04/21/2022-15:52:58] [I] min: 111.667 ms
[04/21/2022-15:52:58] [I] max: 112.103 ms
[04/21/2022-15:52:58] [I] mean: 111.889 ms
[04/21/2022-15:52:58] [I] median: 111.871 ms
[04/21/2022-15:52:58] [I] percentile: 112.103 ms at 99%
[04/21/2022-15:52:58] [I] total compute time: 3.24479 s
&&&& PASSED TensorRT.trtexec # ./trtexec --onnx=stanford_resnext50.onnx --best --minShapes=1,3,1280,1280 --optShapes=16,3,1280,1280 --maxShapes=32,3,1280,1280 --saveEngine=model.plan

And this is the issue.

Environment

TensorRT Version: 7.1
GPU Type: Xavier AGX Jetpack 4.5.1
Nvidia Driver Version:
CUDA Version: 10.2
CUDNN Version:
Operating System + Version:
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)
The stanford_resnext50.onnx is the model file.

Steps To Reproduce

Please include:

Exact steps/commands to build your repro
Exact steps/commands to run your repro
Full traceback of errors encountered

NVES · April 21, 2022, 8:37am

Hi,
Can you try running your model with trtexec command, and share the “”–verbose"" log in case if the issue persist

You can refer below link for all the supported operators list, in case any operator is not supported you need to create a custom plugin to support that operation

github.com

onnx/onnx-tensorrt/blob/main/docs/operators.md

<!--- SPDX-License-Identifier: Apache-2.0 -->

# Supported ONNX Operators

TensorRT 8.4 supports operators up to Opset 17. Latest information of ONNX operators can be found [here](https://github.com/onnx/onnx/blob/master/docs/Operators.md)

TensorRT supports the following ONNX data types: DOUBLE, FLOAT32, FLOAT16, INT8, and BOOL

> Note: There is limited support for INT32, INT64, and DOUBLE types. TensorRT will attempt to cast down INT64 to INT32 and DOUBLE down to FLOAT, clamping values to `+-INT_MAX` or `+-FLT_MAX` if necessary.

See below for the support matrix of ONNX operators in ONNX-TensorRT.

## Operator Support Matrix

| Operator                  | Supported  | Supported Types | Restrictions                                                                                                           |
|---------------------------|------------|-----------------|------------------------------------------------------------------------------------------------------------------------|
| Abs                       | Y          | FP32, FP16, INT32 |
| Acos                      | Y          | FP32, FP16 |
| Acosh                     | Y          | FP32, FP16 |
| Add                       | Y          | FP32, FP16, INT32 |

This file has been truncated. show original

Also, request you to share your model and script if not shared already so that we can help you better.

Meanwhile, for some common errors and queries please refer to below link:

Thanks!

jhpark7 · April 21, 2022, 8:41am

Hi NVES,
here is the verbose log.
Thanks.

/usr/src/tensorrt/bin$ ./trtexec --loadEngine=model.plan --verbose
&&&& RUNNING TensorRT.trtexec # ./trtexec --loadEngine=model.plan --verbose
[04/21/2022-17:40:01] [I] === Model Options ===
[04/21/2022-17:40:01] [I] Format: *
[04/21/2022-17:40:01] [I] Model: 
[04/21/2022-17:40:01] [I] Output:
[04/21/2022-17:40:01] [I] === Build Options ===
[04/21/2022-17:40:01] [I] Max batch: 1
[04/21/2022-17:40:01] [I] Workspace: 16 MB
[04/21/2022-17:40:01] [I] minTiming: 1
[04/21/2022-17:40:01] [I] avgTiming: 8
[04/21/2022-17:40:01] [I] Precision: FP32
[04/21/2022-17:40:01] [I] Calibration: 
[04/21/2022-17:40:01] [I] Safe mode: Disabled
[04/21/2022-17:40:01] [I] Save engine: 
[04/21/2022-17:40:01] [I] Load engine: model.plan
[04/21/2022-17:40:01] [I] Builder Cache: Enabled
[04/21/2022-17:40:01] [I] NVTX verbosity: 0
[04/21/2022-17:40:01] [I] Inputs format: fp32:CHW
[04/21/2022-17:40:01] [I] Outputs format: fp32:CHW
[04/21/2022-17:40:01] [I] Input build shapes: model
[04/21/2022-17:40:01] [I] Input calibration shapes: model
[04/21/2022-17:40:01] [I] === System Options ===
[04/21/2022-17:40:01] [I] Device: 0
[04/21/2022-17:40:01] [I] DLACore: 
[04/21/2022-17:40:01] [I] Plugins:
[04/21/2022-17:40:01] [I] === Inference Options ===
[04/21/2022-17:40:01] [I] Batch: 1
[04/21/2022-17:40:01] [I] Input inference shapes: model
[04/21/2022-17:40:01] [I] Iterations: 10
[04/21/2022-17:40:01] [I] Duration: 3s (+ 200ms warm up)
[04/21/2022-17:40:01] [I] Sleep time: 0ms
[04/21/2022-17:40:01] [I] Streams: 1
[04/21/2022-17:40:01] [I] ExposeDMA: Disabled
[04/21/2022-17:40:01] [I] Spin-wait: Disabled
[04/21/2022-17:40:01] [I] Multithreading: Disabled
[04/21/2022-17:40:01] [I] CUDA Graph: Disabled
[04/21/2022-17:40:01] [I] Skip inference: Disabled
[04/21/2022-17:40:01] [I] Inputs:
[04/21/2022-17:40:01] [I] === Reporting Options ===
[04/21/2022-17:40:01] [I] Verbose: Enabled
[04/21/2022-17:40:01] [I] Averages: 10 inferences
[04/21/2022-17:40:01] [I] Percentile: 99
[04/21/2022-17:40:01] [I] Dump output: Disabled
[04/21/2022-17:40:01] [I] Profile: Disabled
[04/21/2022-17:40:01] [I] Export timing to JSON file: 
[04/21/2022-17:40:01] [I] Export output to JSON file: 
[04/21/2022-17:40:01] [I] Export profile to JSON file: 
[04/21/2022-17:40:01] [I] 
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::GridAnchor_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::NMS_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::Reorg_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::Region_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::Clip_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::LReLU_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::PriorBox_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::Normalize_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::RPROI_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::BatchedNMS_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::FlattenConcat_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::CropAndResize version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::DetectionLayer_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::Proposal version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::ProposalLayer_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::PyramidROIAlign_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::ResizeNearest_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::Split version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::SpecialSlice_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 1
[04/21/2022-17:40:05] [V] [TRT] Deserialize required 2381926 microseconds.
[04/21/2022-17:40:05] [I] Starting inference threads
[04/21/2022-17:40:09] [I] Warmup completed 1 queries over 200 ms
[04/21/2022-17:40:09] [I] Timing trace has 27 queries over 3.16933 s
[04/21/2022-17:40:09] [I] Trace averages of 10 runs:
[04/21/2022-17:40:09] [I] Average on 10 runs - GPU latency: 116.781 ms - Host latency: 118.73 ms (end to end 118.741 ms, enqueue 4.09866 ms)
[04/21/2022-17:40:09] [I] Average on 10 runs - GPU latency: 111.825 ms - Host latency: 113.663 ms (end to end 113.673 ms, enqueue 3.53608 ms)
[04/21/2022-17:40:09] [I] Host Latency
[04/21/2022-17:40:09] [I] min: 113.526 ms (end to end 113.531 ms)
[04/21/2022-17:40:09] [I] max: 148.301 ms (end to end 148.308 ms)
[04/21/2022-17:40:09] [I] mean: 117.36 ms (end to end 117.383 ms)
[04/21/2022-17:40:09] [I] median: 113.701 ms (end to end 113.713 ms)
[04/21/2022-17:40:09] [I] percentile: 148.301 ms at 99% (end to end 148.308 ms at 99%)
[04/21/2022-17:40:09] [I] throughput: 8.51915 qps
[04/21/2022-17:40:09] [I] walltime: 3.16933 s
[04/21/2022-17:40:09] [I] Enqueue Time
[04/21/2022-17:40:09] [I] min: 3.2078 ms
[04/21/2022-17:40:09] [I] max: 5.80618 ms
[04/21/2022-17:40:09] [I] median: 3.60144 ms
[04/21/2022-17:40:09] [I] GPU Compute
[04/21/2022-17:40:09] [I] min: 111.685 ms
[04/21/2022-17:40:09] [I] max: 145.308 ms
[04/21/2022-17:40:09] [I] mean: 115.465 ms
[04/21/2022-17:40:09] [I] median: 111.865 ms
[04/21/2022-17:40:09] [I] percentile: 145.308 ms at 99%
[04/21/2022-17:40:09] [I] total compute time: 3.11755 s
&&&& PASSED TensorRT.trtexec # ./trtexec --loadEngine=model.plan --verbose

spolisetty · April 21, 2022, 1:33pm

Hi,

Are you always facing this issue on frame 6 or it randomly happens?
Based on the screenshot looks like you’re facing a segfault for frame 6, successfully inferred before 5 frames?
Also looks like you’re using an old version of the TensorRT. We recommend you to please use the latest TensorRT version. Release Notes :: NVIDIA Deep Learning TensorRT Documentation
If you still face this issue, please share with us the minimal issue repro script and error logs for better debugging.

Thank you.

shivachauhan843 · September 4, 2023, 8:56am

i am using this command for my model conversion from onnx to trt via trtexec but got this error /usr/src/tensorrt/samples/trtexec$ trtexec --help -bash: trtexec: command not found

Topic		Replies	Views
ONNX Model Inference on Jetson Nano - Segmentation fault Jetson Nano tensorrt , jetson-inference	8	1445	October 15, 2021
Floor - Cast - Resize(or Slice) cause internal error TensorRT tensorrt	7	2006	January 12, 2022
Duplicated reshapes triggers "[graphOptimizer.cpp::findOne::510] Error Code 2: Internal Error (Assertion it != v.end() failed. )" TensorRT tensorrt , onnx	4	918	February 22, 2022
BUG: Output TRT engine from trtexec has completely different inference than input model TensorRT tensorrt , debugging-and-troubleshooting	3	2207	January 4, 2022
Cuda OutOfMemory when creating tensor with 2^29 (~0.5 G) elements TensorRT tensorrt , cuda , onnx	6	1754	March 9, 2022
TensorRT does not see all GPU memory TensorRT	1	996	November 18, 2022
Process killed during tensorrt conversion on Jetson orin NX (8 GB) Jetson Orin NX tensorrt	15	711	April 30, 2024
I do not get any performance improvement after using TensorRT provider for object detection model Jetson Nano tensorrt , onnx	7	1401	July 12, 2022
Slice & resize operators error: N shape values provided by optimization profile, which is not product of its dimensions [N] TensorRT	3	1052	April 12, 2022
I am trying to convert the ONNX SSD mobilnet v3 model into TensorRT Engine. I am getting the below error Jetson TX2 tensorrt , tensorflow	24	3690	February 17, 2022

tensorRT plan file not working

Description

Environment

Relevant Files

Steps To Reproduce

Related topics