Failure：yolov5_pytorch->onnx->tensorrt

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

I recently tried tensorRT acceleration on the Yolov5s model.
I successfully converted Pytorch’s Yolov5s model into onNX model.
But there were problems converting the ONNx model to engine and accelerating it.
The specific questions are as follows：
&&&& RUNNING TensorRT.trtexec # trtexec --onnx=yolov5_1_3_640_640_static.onnx --explicitBatch --saveEngine=yolov5_1_3_640_640_static.engine --workspace=2048 --fp16
[08/19/2020-12:20:09] [I] === Model Options ===
[08/19/2020-12:20:09] [I] Format: ONNX
[08/19/2020-12:20:09] [I] Model: yolov5_1_3_640_640_static.onnx
[08/19/2020-12:20:09] [I] Output:
[08/19/2020-12:20:09] [I] === Build Options ===
[08/19/2020-12:20:09] [I] Max batch: explicit
[08/19/2020-12:20:09] [I] Workspace: 2048 MB
[08/19/2020-12:20:09] [I] minTiming: 1
[08/19/2020-12:20:09] [I] avgTiming: 8
[08/19/2020-12:20:09] [I] Precision: FP16
[08/19/2020-12:20:09] [I] Calibration:
[08/19/2020-12:20:09] [I] Safe mode: Disabled
[08/19/2020-12:20:09] [I] Save engine: yolov5_1_3_640_640_static.engine
[08/19/2020-12:20:09] [I] Load engine:
[08/19/2020-12:20:09] [I] Inputs format: fp32:CHW
[08/19/2020-12:20:09] [I] Outputs format: fp32:CHW
[08/19/2020-12:20:09] [I] Input build shapes: model
[08/19/2020-12:20:09] [I] === System Options ===
[08/19/2020-12:20:09] [I] Device: 0
[08/19/2020-12:20:09] [I] DLACore:
[08/19/2020-12:20:09] [I] Plugins:
[08/19/2020-12:20:09] [I] === Inference Options ===
[08/19/2020-12:20:09] [I] Batch: Explicit
[08/19/2020-12:20:09] [I] Iterations: 10
[08/19/2020-12:20:09] [I] Duration: 3s (+ 200ms warm up)
[08/19/2020-12:20:09] [I] Sleep time: 0ms
[08/19/2020-12:20:09] [I] Streams: 1
[08/19/2020-12:20:09] [I] ExposeDMA: Disabled
[08/19/2020-12:20:09] [I] Spin-wait: Disabled
[08/19/2020-12:20:09] [I] Multithreading: Disabled
[08/19/2020-12:20:09] [I] CUDA Graph: Disabled
[08/19/2020-12:20:09] [I] Skip inference: Disabled
[08/19/2020-12:20:09] [I] Inputs:
[08/19/2020-12:20:09] [I] === Reporting Options ===
[08/19/2020-12:20:09] [I] Verbose: Disabled
[08/19/2020-12:20:09] [I] Averages: 10 inferences
[08/19/2020-12:20:09] [I] Percentile: 99
[08/19/2020-12:20:09] [I] Dump output: Disabled
[08/19/2020-12:20:09] [I] Profile: Disabled
[08/19/2020-12:20:09] [I] Export timing to JSON file:
[08/19/2020-12:20:09] [I] Export output to JSON file:
[08/19/2020-12:20:09] [I] Export profile to JSON file:
[08/19/2020-12:20:09] [I]

Input filename: yolov5_1_3_640_640_static.onnx
ONNX IR version: 0.0.6
Opset version: 11
Producer name: pytorch
Producer version: 1.6
Domain:
Model version: 0
Doc string:

[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:222: One or more weights outside the range of INT32 was clamped
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:222: One or more weights outside the range of INT32 was clamped
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:222: One or more weights outside the range of INT32 was clamped
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:222: One or more weights outside the range of INT32 was clamped
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:222: One or more weights outside the range of INT32 was clamped
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:222: One or more weights outside the range of INT32 was clamped
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:222: One or more weights outside the range of INT32 was clamped
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:222: One or more weights outside the range of INT32 was clamped
[08/19/2020-12:20:10] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
While parsing node number 169 [Resize]:
ERROR: ModelImporter.cpp:124 In function parseGraph:
[5] Assertion failed: ctx->tensors().count(inputName)
[08/19/2020-12:20:10] [E] Failed to parse onnx file
[08/19/2020-12:20:10] [E] Parsing model failed
[08/19/2020-12:20:10] [E] Engine creation failed
[08/19/2020-12:20:10] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec # trtexec --onnx=yolov5_1_3_640_640_static.onnx --explicitBatch --saveEngine=yolov5_1_3_640_640_static.engine --workspace=2048 --fp16

Environment

TensorRT Version: 7.0.0.11
GPU Type: Tesla V100-SXM2-32GB
Nvidia Driver Version: 418.67
CUDA Version: 10.1
CUDNN Version: 7.6.5
Operating System + Version: Ubuntu18.04
Python Version (if applicable): 3.6
TensorFlow Version (if applicable):
PyTorch Version (if applicable): 1.6
Baremetal or Container (if container which image + tag):

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)

Steps To Reproduce

Please include:

  • Exact steps/commands to build your repro
  • Exact steps/commands to run your repro
  • Full traceback of errors encountered
2

Hi @1965281904,
Request you to share your onnx model and the script, so that i can assist you better.
Thanks!

3

My file is larger than 2M and cannot be transmitted. Could you please provide an email address？

4

You can upload it on drive and share the link.
Also, i just noticed the TRT version you are using.

Would you mind running your model on latest TRT release(7.3.1)

Thanks!

5

The details of the bug are as follows：

Thanks.

6

Kindly allow access to the files.
Thanks!

7

changed ！you can try it again.
Thanks.

8

Hi @1965281904,
I could successfully run your model.

image
image1984×826 58.6 KB

This was a known issue in TRT7 and has been fixed in the next release.
Hence request you to try running it on latest TRT release(7.3.1)

Thanks!

9

hi！
It is very nice! Can you send me your environment configuration?
cuda：
cudnn：
tensorrt：
In addition, the latest version of Tensorrt that I have seen is as follows：

image
image829×545 34.7 KB

ifI choice it,it’s right?

10

Hi @1965281904,
Below is the support matrix for TRT7.1.3

Thanks!

11

@1965281904 try this to accelerate the yolov5

12

@AakankshaS

For TRT7, it checks all inputs whether tensor is filled with data.

If find any input is empty, TRT will report assertion error: Assertion failed: ctx->tensors().count(inputName)

But I think this assertion is not suitable for all operations.
For example, onnx Resize operation with opset 11, we just need to set either scales or sizes parameter.

微信截图_20200901111140
微信截图_202009011111401080×887 22.5 KB

TRT error:

image
image1469×563 27.7 KB

When I set data to rois and sizes by ONNX Python API, this error gone.

image
image915×737 22 KB

This issus occurs in all ONNX operations which has optional inputs.

13

HI @AakankshaS,Since TensorRT7.1.3 requires a higher version of the gpu driver, however, the device I am currently using cannot be upgraded.
In this case, how can I achieve the normal transformation of the Yolov5s model with the lower version（7.0.0） of TensorRT ？