tensorRT plan file not working

Description

I’m trying to do this post.
https://towardsdatascience.com/detecting-pedestrians-and-bikers-on-a-drone-with-jetson-xavier-93ce92e2c597

The export.cpp file of retinanet-example is also use tensorRT and convert onnx to plan file.
also it is going well.

But the model.plan file result of trtexec utility is not working.

I got model.plan file with this command.
sudo ./trtexec --onnx=stanford_resnext50.onnx --best --minShapes=1,3,1280,1280 --optShapes=16,3,1280,1280 --maxShapes=32,3,1280,1280 --saveEngine=model.plan

And this is the result.

/usr/src/tensorrt/bin$ sudo ./trtexec --onnx=stanford_resnext50.onnx --best --minShapes=1,3,1280,1280 --optShapes=16,3,1280,1280 --maxShapes=32,3,1280,1280 --saveEngine=model.plan
[sudo] password for openmakerlab-agx:
&&&& RUNNING TensorRT.trtexec # ./trtexec --onnx=stanford_resnext50.onnx --best --minShapes=1,3,1280,1280 --optShapes=16,3,1280,1280 --maxShapes=32,3,1280,1280 --saveEngine=model.plan
[04/21/2022-15:37:33] [I] === Model Options ===
[04/21/2022-15:37:33] [I] Format: ONNX
[04/21/2022-15:37:33] [I] Model: stanford_resnext50.onnx
[04/21/2022-15:37:33] [I] Output:
[04/21/2022-15:37:33] [I] === Build Options ===
[04/21/2022-15:37:33] [I] Max batch: explicit
[04/21/2022-15:37:33] [I] Workspace: 16 MB
[04/21/2022-15:37:33] [I] minTiming: 1
[04/21/2022-15:37:33] [I] avgTiming: 8
[04/21/2022-15:37:33] [I] Precision: FP32+FP16+INT8
[04/21/2022-15:37:33] [I] Calibration: Dynamic
[04/21/2022-15:37:33] [I] Safe mode: Disabled
[04/21/2022-15:37:33] [I] Save engine: model.plan
[04/21/2022-15:37:33] [I] Load engine:
[04/21/2022-15:37:33] [I] Builder Cache: Enabled
[04/21/2022-15:37:33] [I] NVTX verbosity: 0
[04/21/2022-15:37:33] [I] Inputs format: fp32:CHW
[04/21/2022-15:37:33] [I] Outputs format: fp32:CHW
[04/21/2022-15:37:33] [I] Input build shape: 16=
[04/21/2022-15:37:33] [I] Input build shape: 1280=1280+1280+1280
[04/21/2022-15:37:33] [I] Input build shape: 1=1
[04/21/2022-15:37:33] [I] Input build shape: 32=
[04/21/2022-15:37:33] [I] Input build shape: 3=3+3+3
[04/21/2022-15:37:33] [I] Input calibration shapes: model
[04/21/2022-15:37:33] [I] === System Options ===
[04/21/2022-15:37:33] [I] Device: 0
[04/21/2022-15:37:33] [I] DLACore:
[04/21/2022-15:37:33] [I] Plugins:
[04/21/2022-15:37:33] [I] === Inference Options ===
[04/21/2022-15:37:33] [I] Batch: Explicit
[04/21/2022-15:37:33] [I] Input inference shape: 3=3
[04/21/2022-15:37:33] [I] Input inference shape: 32=
[04/21/2022-15:37:33] [I] Input inference shape: 1=
[04/21/2022-15:37:33] [I] Input inference shape: 16=16
[04/21/2022-15:37:33] [I] Input inference shape: 1280=1280
[04/21/2022-15:37:33] [I] Iterations: 10
[04/21/2022-15:37:33] [I] Duration: 3s (+ 200ms warm up)
[04/21/2022-15:37:33] [I] Sleep time: 0ms
[04/21/2022-15:37:33] [I] Streams: 1
[04/21/2022-15:37:33] [I] ExposeDMA: Disabled
[04/21/2022-15:37:33] [I] Spin-wait: Disabled
[04/21/2022-15:37:33] [I] Multithreading: Disabled
[04/21/2022-15:37:33] [I] CUDA Graph: Disabled
[04/21/2022-15:37:33] [I] Skip inference: Disabled
[04/21/2022-15:37:33] [I] Inputs:
[04/21/2022-15:37:33] [I] === Reporting Options ===
[04/21/2022-15:37:33] [I] Verbose: Disabled
[04/21/2022-15:37:33] [I] Averages: 10 inferences
[04/21/2022-15:37:33] [I] Percentile: 99
[04/21/2022-15:37:33] [I] Dump output: Disabled
[04/21/2022-15:37:33] [I] Profile: Disabled
[04/21/2022-15:37:33] [I] Export timing to JSON file:
[04/21/2022-15:37:33] [I] Export output to JSON file:
[04/21/2022-15:37:33] [I] Export profile to JSON file:
[04/21/2022-15:37:33] [I]
Input filename: stanford_resnext50.onnx
ONNX IR version: 0.0.4
Opset version: 9
Producer name: pytorch
Producer version: 1.1
Domain:
Model version: 0
Doc string:
[04/21/2022-15:37:35] [W] [TRT] onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[04/21/2022-15:37:35] [W] [TRT] Calibrator is not being used. Users must provide dynamic range for all tensors that are not Int32.
[04/21/2022-15:37:35] [I] [TRT]
[04/21/2022-15:37:35] [I] [TRT] --------------- Layers running on DLA:
[04/21/2022-15:37:35] [I] [TRT]
[04/21/2022-15:37:35] [I] [TRT] --------------- Layers running on GPU:
[04/21/2022-15:37:35] [I] [TRT] (Unnamed Layer* 0) [Convolution] + (Unnamed Layer* 2) [Activation] + (Unnamed Layer* 3) [Pooling], (Unnamed Layer* 4) [Convolution] + (Unnamed Layer* 6) [Activation], (Unnamed Layer* 7) [Convolution] + (Unnamed Layer* 9) [Activation], (Unnamed Layer* 10) [Convolution], (Unnamed Layer* 12) [Convolution] + (Unnamed Layer* 14) [ElementWise] + (Unnamed Layer* 15) [Activation], (Unnamed Layer* 16) [Convolution] + (Unnamed Layer* 18) [Activation], (Unnamed Layer* 19) [Convolution] + (Unnamed Layer* 21) [Activation], (Unnamed Layer* 22) [Convolution] + (Unnamed Layer* 24) [ElementWise] + (Unnamed Layer* 25) [Activation], (Unnamed Layer* 26) [Convolution] + (Unnamed Layer* 28) [Activation], (Unnamed Layer* 29) [Convolution] + (Unnamed Layer* 31) [Activation], (Unnamed Layer* 32) [Convolution] + (Unnamed Layer* 34) [ElementWise] + (Unnamed Layer* 35) [Activation], (Unnamed Layer* 36) [Convolution] + (Unnamed Layer* 38) [Activation], (Unnamed Layer* 39) [Convolution] + (Unnamed Layer* 41) [Activation], (Unnamed Layer* 42) [Convolution], (Unnamed Layer* 44) [Convolution] + (Unnamed Layer* 46) [ElementWise] + (Unnamed Layer* 47) [Activation], (Unnamed Layer* 48) [Convolution] + (Unnamed Layer* 50) [Activation], (Unnamed Layer* 51) [Convolution] + (Unnamed Layer* 53) [Activation], (Unnamed Layer* 54) [Convolution] + (Unnamed Layer* 56) [ElementWise] + (Unnamed Layer* 57) [Activation], (Unnamed Layer* 58) [Convolution] + (Unnamed Layer* 60) [Activation], (Unnamed Layer* 61) [Convolution] + (Unnamed Layer* 63) [Activation], (Unnamed Layer* 64) [Convolution] + (Unnamed Layer* 66) [ElementWise] + (Unnamed Layer* 67) [Activation], (Unnamed Layer* 68) [Convolution] + (Unnamed Layer* 70) [Activation], (Unnamed Layer* 71) [Convolution] + (Unnamed Layer* 73) [Activation], (Unnamed Layer* 74) [Convolution] + (Unnamed Layer* 76) [ElementWise] + (Unnamed Layer* 77) [Activation], (Unnamed Layer* 78) [Convolution] + (Unnamed Layer* 80) [Activation], (Unnamed Layer* 81) [Convolution] + (Unnamed Layer* 83) [Activation], (Unnamed Layer* 84) [Convolution], (Unnamed Layer* 86) [Convolution] + (Unnamed Layer* 88) [ElementWise] + (Unnamed Layer* 89) [Activation], (Unnamed Layer* 90) [Convolution] + (Unnamed Layer* 92) [Activation], (Unnamed Layer* 93) [Convolution] + (Unnamed Layer* 95) [Activation], (Unnamed Layer* 96) [Convolution] + (Unnamed Layer* 98) [ElementWise] + (Unnamed Layer* 99) [Activation], (Unnamed Layer* 100) [Convolution] + (Unnamed Layer* 102) [Activation], (Unnamed Layer* 103) [Convolution] + (Unnamed Layer* 105) [Activation], (Unnamed Layer* 106) [Convolution] + (Unnamed Layer* 108) [ElementWise] + (Unnamed Layer* 109) [Activation], (Unnamed Layer* 110) [Convolution] + (Unnamed Layer* 112) [Activation], (Unnamed Layer* 113) [Convolution] + (Unnamed Layer* 115) [Activation], (Unnamed Layer* 116) [Convolution] + (Unnamed Layer* 118) [ElementWise] + (Unnamed Layer* 119) [Activation], (Unnamed Layer* 120) [Convolution] + (Unnamed Layer* 122) [Activation], (Unnamed Layer* 123) [Convolution] + (Unnamed Layer* 125) [Activation], (Unnamed Layer* 126) [Convolution] + (Unnamed Layer* 128) [ElementWise] + (Unnamed Layer* 129) [Activation], (Unnamed Layer* 130) [Convolution] + (Unnamed Layer* 132) [Activation], (Unnamed Layer* 133) [Convolution] + (Unnamed Layer* 135) [Activation], (Unnamed Layer* 136) [Convolution] + (Unnamed Layer* 138) [ElementWise] + (Unnamed Layer* 139) [Activation], (Unnamed Layer* 140) [Convolution] + (Unnamed Layer* 142) [Activation], (Unnamed Layer* 143) [Convolution] + (Unnamed Layer* 145) [Activation], (Unnamed Layer* 146) [Convolution], (Unnamed Layer* 148) [Convolution] + (Unnamed Layer* 150) [ElementWise] + (Unnamed Layer* 151) [Activation], (Unnamed Layer* 152) [Convolution] + (Unnamed Layer* 154) [Activation], (Unnamed Layer* 155) [Convolution] + (Unnamed Layer* 157) [Activation], (Unnamed Layer* 158) [Convolution] + (Unnamed Layer* 160) [ElementWise] + (Unnamed Layer* 161) [Activation], (Unnamed Layer* 162) [Convolution] + (Unnamed Layer* 164) [Activation], (Unnamed Layer* 165) [Convolution] + (Unnamed Layer* 167) [Activation], (Unnamed Layer* 168) [Convolution] + (Unnamed Layer* 170) [ElementWise] + (Unnamed Layer* 171) [Activation], (Unnamed Layer* 179) [Convolution], (Unnamed Layer* 257) [Convolution] + (Unnamed Layer* 258) [Activation] || (Unnamed Layer* 212) [Convolution] + (Unnamed Layer* 213) [Activation], (Unnamed Layer* 172) [Convolution], (Unnamed Layer* 184) [Convolution], (Unnamed Layer* 248) [Convolution] + (Unnamed Layer* 249) [Activation] || (Unnamed Layer* 203) [Convolution] + (Unnamed Layer* 204) [Activation], (Unnamed Layer* 174) [Resize], (Unnamed Layer* 180) [Activation], (Unnamed Layer* 173) [Convolution] + (Unnamed Layer* 175) [ElementWise], (Unnamed Layer* 183) [Convolution], (Unnamed Layer* 239) [Convolution] + (Unnamed Layer* 240) [Activation] || (Unnamed Layer* 194) [Convolution] + (Unnamed Layer* 195) [Activation], (Unnamed Layer* 177) [Resize], (Unnamed Layer* 205) [Convolution] + (Unnamed Layer* 206) [Activation], (Unnamed Layer* 250) [Convolution] + (Unnamed Layer* 251) [Activation], (Unnamed Layer* 214) [Convolution] + (Unnamed Layer* 215) [Activation], (Unnamed Layer* 259) [Convolution] + (Unnamed Layer* 260) [Activation], (Unnamed Layer* 181) [Convolution], (Unnamed Layer* 266) [Convolution] + (Unnamed Layer* 267) [Activation] || (Unnamed Layer* 221) [Convolution] + (Unnamed Layer* 222) [Activation], (Unnamed Layer* 176) [Convolution] + (Unnamed Layer* 178) [ElementWise], (Unnamed Layer* 223) [Convolution] + (Unnamed Layer* 224) [Activation], (Unnamed Layer* 268) [Convolution] + (Unnamed Layer* 269) [Activation], (Unnamed Layer* 261) [Convolution] + (Unnamed Layer* 262) [Activation], (Unnamed Layer* 216) [Convolution] + (Unnamed Layer* 217) [Activation], (Unnamed Layer* 252) [Convolution] + (Unnamed Layer* 253) [Activation], (Unnamed Layer* 207) [Convolution] + (Unnamed Layer* 208) [Activation], (Unnamed Layer* 182) [Convolution], (Unnamed Layer* 230) [Convolution] + (Unnamed Layer* 231) [Activation] || (Unnamed Layer* 185) [Convolution] + (Unnamed Layer* 186) [Activation], (Unnamed Layer* 196) [Convolution] + (Unnamed Layer* 197) [Activation], (Unnamed Layer* 241) [Convolution] + (Unnamed Layer* 242) [Activation], (Unnamed Layer* 243) [Convolution] + (Unnamed Layer* 244) [Activation], (Unnamed Layer* 198) [Convolution] + (Unnamed Layer* 199) [Activation], (Unnamed Layer* 187) [Convolution] + (Unnamed Layer* 188) [Activation], (Unnamed Layer* 232) [Convolution] + (Unnamed Layer* 233) [Activation], (Unnamed Layer* 209) [Convolution] + (Unnamed Layer* 210) [Activation], (Unnamed Layer* 254) [Convolution] + (Unnamed Layer* 255) [Activation], (Unnamed Layer* 218) [Convolution] + (Unnamed Layer* 219) [Activation], (Unnamed Layer* 263) [Convolution] + (Unnamed Layer* 264) [Activation], (Unnamed Layer* 270) [Convolution] + (Unnamed Layer* 271) [Activation], (Unnamed Layer* 225) [Convolution] + (Unnamed Layer* 226) [Activation], (Unnamed Layer* 227) [Convolution] + (Unnamed Layer* 228) [Activation], (Unnamed Layer* 272) [Convolution] + (Unnamed Layer* 273) [Activation], (Unnamed Layer* 265) [Convolution], (Unnamed Layer* 220) [Convolution], (Unnamed Layer* 278) [Activation], (Unnamed Layer* 256) [Convolution], (Unnamed Layer* 211) [Convolution], (Unnamed Layer* 277) [Activation], (Unnamed Layer* 234) [Convolution] + (Unnamed Layer* 235) [Activation], (Unnamed Layer* 189) [Convolution] + (Unnamed Layer* 190) [Activation], (Unnamed Layer* 200) [Convolution] + (Unnamed Layer* 201) [Activation], (Unnamed Layer* 245) [Convolution] + (Unnamed Layer* 246) [Activation], (Unnamed Layer* 247) [Convolution], (Unnamed Layer* 202) [Convolution], (Unnamed Layer* 276) [Activation], (Unnamed Layer* 191) [Convolution] + (Unnamed Layer* 192) [Activation], (Unnamed Layer* 236) [Convolution] + (Unnamed Layer* 237) [Activation], (Unnamed Layer* 274) [Convolution], (Unnamed Layer* 229) [Convolution], (Unnamed Layer* 279) [Activation], (Unnamed Layer* 238) [Convolution], (Unnamed Layer* 193) [Convolution], (Unnamed Layer* 275) [Activation],
[04/21/2022-15:37:44] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
[04/21/2022-15:52:53] [I] [TRT] Detected 1 inputs and 10 output network tensors.
[04/21/2022-15:52:55] [I] Starting inference threads
[04/21/2022-15:52:58] [I] Warmup completed 0 queries over 200 ms
[04/21/2022-15:52:58] [I] Timing trace has 0 queries over 3.29845 s
[04/21/2022-15:52:58] [I] Trace averages of 10 runs:
[04/21/2022-15:52:58] [I] Average on 10 runs - GPU latency: 111.904 ms - Host latency: 113.743 ms (end to end 113.753 ms, enqueue 4.17942 ms)
[04/21/2022-15:52:58] [I] Average on 10 runs - GPU latency: 111.9 ms - Host latency: 113.735 ms (end to end 113.747 ms, enqueue 4.09928 ms)
[04/21/2022-15:52:58] [I] Host Latency
[04/21/2022-15:52:58] [I] min: 113.5 ms (end to end 113.519 ms)
[04/21/2022-15:52:58] [I] max: 113.942 ms (end to end 113.955 ms)
[04/21/2022-15:52:58] [I] mean: 113.729 ms (end to end 113.739 ms)
[04/21/2022-15:52:58] [I] median: 113.722 ms (end to end 113.729 ms)
[04/21/2022-15:52:58] [I] percentile: 113.942 ms at 99% (end to end 113.955 ms at 99%)
[04/21/2022-15:52:58] [I] throughput: 0 qps
[04/21/2022-15:52:58] [I] walltime: 3.29845 s
[04/21/2022-15:52:58] [I] Enqueue Time
[04/21/2022-15:52:58] [I] min: 3.80664 ms
[04/21/2022-15:52:58] [I] max: 4.59406 ms
[04/21/2022-15:52:58] [I] median: 4.07446 ms
[04/21/2022-15:52:58] [I] GPU Compute
[04/21/2022-15:52:58] [I] min: 111.667 ms
[04/21/2022-15:52:58] [I] max: 112.103 ms
[04/21/2022-15:52:58] [I] mean: 111.889 ms
[04/21/2022-15:52:58] [I] median: 111.871 ms
[04/21/2022-15:52:58] [I] percentile: 112.103 ms at 99%
[04/21/2022-15:52:58] [I] total compute time: 3.24479 s
&&&& PASSED TensorRT.trtexec # ./trtexec --onnx=stanford_resnext50.onnx --best --minShapes=1,3,1280,1280 --optShapes=16,3,1280,1280 --maxShapes=32,3,1280,1280 --saveEngine=model.plan

And this is the issue.

Environment

TensorRT Version: 7.1
GPU Type: Xavier AGX Jetpack 4.5.1
Nvidia Driver Version:
CUDA Version: 10.2
CUDNN Version:
Operating System + Version:
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)
The stanford_resnext50.onnx is the model file.

Steps To Reproduce

Please include:

  • Exact steps/commands to build your repro
  • Exact steps/commands to run your repro
  • Full traceback of errors encountered

Hi,
Can you try running your model with trtexec command, and share the “”–verbose"" log in case if the issue persist

You can refer below link for all the supported operators list, in case any operator is not supported you need to create a custom plugin to support that operation

Also, request you to share your model and script if not shared already so that we can help you better.

Meanwhile, for some common errors and queries please refer to below link:

Thanks!

Hi NVES,
here is the verbose log.
Thanks.

/usr/src/tensorrt/bin$ ./trtexec --loadEngine=model.plan --verbose
&&&& RUNNING TensorRT.trtexec # ./trtexec --loadEngine=model.plan --verbose
[04/21/2022-17:40:01] [I] === Model Options ===
[04/21/2022-17:40:01] [I] Format: *
[04/21/2022-17:40:01] [I] Model: 
[04/21/2022-17:40:01] [I] Output:
[04/21/2022-17:40:01] [I] === Build Options ===
[04/21/2022-17:40:01] [I] Max batch: 1
[04/21/2022-17:40:01] [I] Workspace: 16 MB
[04/21/2022-17:40:01] [I] minTiming: 1
[04/21/2022-17:40:01] [I] avgTiming: 8
[04/21/2022-17:40:01] [I] Precision: FP32
[04/21/2022-17:40:01] [I] Calibration: 
[04/21/2022-17:40:01] [I] Safe mode: Disabled
[04/21/2022-17:40:01] [I] Save engine: 
[04/21/2022-17:40:01] [I] Load engine: model.plan
[04/21/2022-17:40:01] [I] Builder Cache: Enabled
[04/21/2022-17:40:01] [I] NVTX verbosity: 0
[04/21/2022-17:40:01] [I] Inputs format: fp32:CHW
[04/21/2022-17:40:01] [I] Outputs format: fp32:CHW
[04/21/2022-17:40:01] [I] Input build shapes: model
[04/21/2022-17:40:01] [I] Input calibration shapes: model
[04/21/2022-17:40:01] [I] === System Options ===
[04/21/2022-17:40:01] [I] Device: 0
[04/21/2022-17:40:01] [I] DLACore: 
[04/21/2022-17:40:01] [I] Plugins:
[04/21/2022-17:40:01] [I] === Inference Options ===
[04/21/2022-17:40:01] [I] Batch: 1
[04/21/2022-17:40:01] [I] Input inference shapes: model
[04/21/2022-17:40:01] [I] Iterations: 10
[04/21/2022-17:40:01] [I] Duration: 3s (+ 200ms warm up)
[04/21/2022-17:40:01] [I] Sleep time: 0ms
[04/21/2022-17:40:01] [I] Streams: 1
[04/21/2022-17:40:01] [I] ExposeDMA: Disabled
[04/21/2022-17:40:01] [I] Spin-wait: Disabled
[04/21/2022-17:40:01] [I] Multithreading: Disabled
[04/21/2022-17:40:01] [I] CUDA Graph: Disabled
[04/21/2022-17:40:01] [I] Skip inference: Disabled
[04/21/2022-17:40:01] [I] Inputs:
[04/21/2022-17:40:01] [I] === Reporting Options ===
[04/21/2022-17:40:01] [I] Verbose: Enabled
[04/21/2022-17:40:01] [I] Averages: 10 inferences
[04/21/2022-17:40:01] [I] Percentile: 99
[04/21/2022-17:40:01] [I] Dump output: Disabled
[04/21/2022-17:40:01] [I] Profile: Disabled
[04/21/2022-17:40:01] [I] Export timing to JSON file: 
[04/21/2022-17:40:01] [I] Export output to JSON file: 
[04/21/2022-17:40:01] [I] Export profile to JSON file: 
[04/21/2022-17:40:01] [I] 
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::GridAnchor_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::NMS_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::Reorg_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::Region_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::Clip_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::LReLU_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::PriorBox_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::Normalize_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::RPROI_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::BatchedNMS_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::FlattenConcat_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::CropAndResize version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::DetectionLayer_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::Proposal version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::ProposalLayer_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::PyramidROIAlign_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::ResizeNearest_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::Split version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::SpecialSlice_TRT version 1
[04/21/2022-17:40:01] [V] [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 1
[04/21/2022-17:40:05] [V] [TRT] Deserialize required 2381926 microseconds.
[04/21/2022-17:40:05] [I] Starting inference threads
[04/21/2022-17:40:09] [I] Warmup completed 1 queries over 200 ms
[04/21/2022-17:40:09] [I] Timing trace has 27 queries over 3.16933 s
[04/21/2022-17:40:09] [I] Trace averages of 10 runs:
[04/21/2022-17:40:09] [I] Average on 10 runs - GPU latency: 116.781 ms - Host latency: 118.73 ms (end to end 118.741 ms, enqueue 4.09866 ms)
[04/21/2022-17:40:09] [I] Average on 10 runs - GPU latency: 111.825 ms - Host latency: 113.663 ms (end to end 113.673 ms, enqueue 3.53608 ms)
[04/21/2022-17:40:09] [I] Host Latency
[04/21/2022-17:40:09] [I] min: 113.526 ms (end to end 113.531 ms)
[04/21/2022-17:40:09] [I] max: 148.301 ms (end to end 148.308 ms)
[04/21/2022-17:40:09] [I] mean: 117.36 ms (end to end 117.383 ms)
[04/21/2022-17:40:09] [I] median: 113.701 ms (end to end 113.713 ms)
[04/21/2022-17:40:09] [I] percentile: 148.301 ms at 99% (end to end 148.308 ms at 99%)
[04/21/2022-17:40:09] [I] throughput: 8.51915 qps
[04/21/2022-17:40:09] [I] walltime: 3.16933 s
[04/21/2022-17:40:09] [I] Enqueue Time
[04/21/2022-17:40:09] [I] min: 3.2078 ms
[04/21/2022-17:40:09] [I] max: 5.80618 ms
[04/21/2022-17:40:09] [I] median: 3.60144 ms
[04/21/2022-17:40:09] [I] GPU Compute
[04/21/2022-17:40:09] [I] min: 111.685 ms
[04/21/2022-17:40:09] [I] max: 145.308 ms
[04/21/2022-17:40:09] [I] mean: 115.465 ms
[04/21/2022-17:40:09] [I] median: 111.865 ms
[04/21/2022-17:40:09] [I] percentile: 145.308 ms at 99%
[04/21/2022-17:40:09] [I] total compute time: 3.11755 s
&&&& PASSED TensorRT.trtexec # ./trtexec --loadEngine=model.plan --verbose

Hi,

Are you always facing this issue on frame 6 or it randomly happens?
Based on the screenshot looks like you’re facing a segfault for frame 6, successfully inferred before 5 frames?
Also looks like you’re using an old version of the TensorRT. We recommend you to please use the latest TensorRT version. Release Notes :: NVIDIA Deep Learning TensorRT Documentation
If you still face this issue, please share with us the minimal issue repro script and error logs for better debugging.

Thank you.