Description
I’ve trained a yolor model (GitHub - WongKinYiu/yolor: implementation of paper - You Only Learn One Representation: Unified Network for Multiple Tasks (https://arxiv.org/abs/2105.04206)) and I’ve exported it to onnx this snippet of code.
git clone https://github.com/WongKinYiu/yolor.git
then
import sys
sys.path.append('yolor')
from yolor.models.models import Darknet
import numpy as np
import torch
import onnx
device = 'cuda'
half = False
weight_file = 'yolor_csp.pt'
img_size = 640
img = torch.rand((1, 3,img_size, img_size)).to(device)
model = Darknet('yolor/cfg/yolor_csp.cfg', img_size).cuda()
print('\nStarting ONNX export with onnx %s...' % onnx.__version__)
f = weight_file.replace('.pt', '.onnx') # filename
model.fuse() # only for ONNX
torch.onnx.export(model, img, f, verbose=False, opset_version=12, input_names=['images'],
output_names=['classes', 'boxes'] if y is None else ['output'])
onnx_model = onnx.load(f) # load onnx model
onnx.checker.check_model(onnx_model) # check onnx model
print(onnx.helper.printable_graph(onnx_model.graph)) # print a human readable model
print('ONNX export success, saved as %s' % f)
the model I’m using is Darknet model (yolor/models.py at main · WongKinYiu/yolor · GitHub)
I tried to run polygraph to check if the results are accurate and this is the output of the command
[I] Will generate inference input data according to provided TensorMetadata: {images [shape=(1, 3, 640, 640)]}
[I] trt-runner-N0-09/14/22-11:08:17 | Activating and starting inference
[09/14/2022-11:08:21] [TRT] [W] onnx2trt_utils.cpp:367: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[09/14/2022-11:08:21] [TRT] [W] onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
[09/14/2022-11:08:21] [TRT] [W] onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
[09/14/2022-11:08:22] [TRT] [W] onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
[09/14/2022-11:08:22] [TRT] [W] onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
[09/14/2022-11:08:22] [TRT] [W] onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
[09/14/2022-11:08:22] [TRT] [W] onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
[09/14/2022-11:08:22] [TRT] [W] onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
[09/14/2022-11:08:22] [TRT] [W] onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
[09/14/2022-11:08:23] [TRT] [W] onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
[09/14/2022-11:08:23] [TRT] [W] onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
[09/14/2022-11:08:23] [TRT] [W] onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
[09/14/2022-11:08:23] [TRT] [W] onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
[I] Configuring with profiles: [Profile().add('images', min=[1, 3, 640, 640], opt=[1, 3, 640, 640], max=[1, 3, 640, 640])]
[I] Building engine with configuration:
Workspace | 16777216 bytes (16.00 MiB)
Precision | TF32: False, FP16: False, INT8: False, Obey Precision Constraints: False, Strict Types: False
Tactic Sources | ['CUBLAS', 'CUBLAS_LT', 'CUDNN', 'EDGE_MASK_CONVOLUTIONS']
Safety Restricted | False
Profiles | 1 profile(s)
[09/14/2022-11:11:13] [TRT] [W] The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
[09/14/2022-11:11:13] [TRT] [W] The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
[I] Finished engine building in 169.685 seconds
[I] trt-runner-N0-09/14/22-11:08:17
---- Inference Input(s) ----
{images [dtype=float32, shape=(1, 3, 640, 640)]}
[I] trt-runner-N0-09/14/22-11:08:17
---- Inference Output(s) ----
{onnx::Sigmoid_638 [dtype=float32, shape=(1, 3, 80, 80, 7)],
onnx::Sigmoid_977 [dtype=float32, shape=(1, 3, 40, 40, 7)],
onnx::Sigmoid_1316 [dtype=float32, shape=(1, 3, 20, 20, 7)],
output [dtype=float32, shape=(1, 25200, 7)]}
[I] trt-runner-N0-09/14/22-11:08:17 | Completed 1 iteration(s) in 67.43 ms | Average inference time: 67.43 ms.
[I] onnxrt-runner-N0-09/14/22-11:08:17 | Activating and starting inference
[I] Creating ONNX-Runtime Inference Session with providers: ['CPUExecutionProvider']
[I] onnxrt-runner-N0-09/14/22-11:08:17
---- Inference Input(s) ----
{images [dtype=float32, shape=(1, 3, 640, 640)]}
[I] onnxrt-runner-N0-09/14/22-11:08:17
---- Inference Output(s) ----
{output [dtype=float32, shape=(1, 25200, 7)],
onnx::Sigmoid_638 [dtype=float32, shape=(1, 3, 80, 80, 7)],
onnx::Sigmoid_977 [dtype=float32, shape=(1, 3, 40, 40, 7)],
onnx::Sigmoid_1316 [dtype=float32, shape=(1, 3, 20, 20, 7)]}
[I] onnxrt-runner-N0-09/14/22-11:08:17 | Completed 1 iteration(s) in 801.1 ms | Average inference time: 801.1 ms.
[I] Accuracy Comparison | trt-runner-N0-09/14/22-11:08:17 vs. onnxrt-runner-N0-09/14/22-11:08:17
[I] Comparing Output: 'onnx::Sigmoid_638' (dtype=float32, shape=(1, 3, 80, 80, 7)) with 'onnx::Sigmoid_638' (dtype=float32, shape=(1, 3, 80, 80, 7))
[I] Tolerance: [abs=1e-05, rel=1e-05] | Checking elemwise error
[I] trt-runner-N0-09/14/22-11:08:17: onnx::Sigmoid_638 | Stats: mean=-1.5821, std-dev=4.023, var=16.185, median=-0.035941, min=-13.831 at (0, 0, 69, 1, 4), max=2.5998 at (0, 0, 65, 56, 5), avg-magnitude=1.9761
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(-13.8 , -12.2 ) | 3189 | #
(-12.2 , -10.5 ) | 12571 | #####
(-10.5 , -8.9 ) | 3174 | #
(-8.9 , -7.26 ) | 265 |
(-7.26 , -5.62 ) | 1 |
(-5.62 , -3.97 ) | 0 |
(-3.97 , -2.33 ) | 14 |
(-2.33 , -0.686) | 9883 | ###
(-0.686, 0.957 ) | 99604 | ########################################
(0.957 , 2.6 ) | 5699 | ##
[I] onnxrt-runner-N0-09/14/22-11:08:17: onnx::Sigmoid_638 | Stats: mean=-1.5821, std-dev=4.023, var=16.185, median=-0.035944, min=-13.831 at (0, 0, 69, 1, 4), max=2.5998 at (0, 0, 65, 56, 5), avg-magnitude=1.9761
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(-13.8 , -12.2 ) | 3189 | #
(-12.2 , -10.5 ) | 12571 | #####
(-10.5 , -8.9 ) | 3174 | #
(-8.9 , -7.26 ) | 265 |
(-7.26 , -5.62 ) | 1 |
(-5.62 , -3.97 ) | 0 |
(-3.97 , -2.33 ) | 14 |
(-2.33 , -0.686) | 9883 | ###
(-0.686, 0.957 ) | 99604 | ########################################
(0.957 , 2.6 ) | 5699 | ##
[I] Error Metrics: onnx::Sigmoid_638
[I] Minimum Required Tolerance: elemwise error | [abs=7.1526e-05] OR [rel=1.4975] (requirements may be lower if both abs/rel tolerances are set)
[I] Absolute Difference | Stats: mean=5.3967e-06, std-dev=6.0205e-06, var=3.6247e-11, median=3.5465e-06, min=0 at (0, 0, 0, 4, 5), max=7.1526e-05 at (0, 1, 44, 42, 4), avg-magnitude=5.3967e-06
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(0 , 7.15e-06) | 101980 | ########################################
(7.15e-06, 1.43e-05) | 21997 | ########
(1.43e-05, 2.15e-05) | 6389 | ##
(2.15e-05, 2.86e-05) | 2356 |
(2.86e-05, 3.58e-05) | 1149 |
(3.58e-05, 4.29e-05) | 307 |
(4.29e-05, 5.01e-05) | 154 |
(5.01e-05, 5.72e-05) | 40 |
(5.72e-05, 6.44e-05) | 20 |
(6.44e-05, 7.15e-05) | 8 |
[I] Relative Difference | Stats: mean=7.0739e-05, std-dev=0.0043159, var=1.8627e-05, median=7.4942e-06, min=0 at (0, 0, 0, 4, 5), max=1.4975 at (0, 1, 12, 7, 0), avg-magnitude=7.0739e-05
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(0 , 0.15 ) | 134396 | ########################################
(0.15 , 0.3 ) | 3 |
(0.3 , 0.449) | 0 |
(0.449, 0.599) | 0 |
(0.599, 0.749) | 0 |
(0.749, 0.899) | 0 |
(0.899, 1.05 ) | 0 |
(1.05 , 1.2 ) | 0 |
(1.2 , 1.35 ) | 0 |
(1.35 , 1.5 ) | 1 |
[E] FAILED | Difference exceeds tolerance (rel=1e-05, abs=1e-05)
[I] Comparing Output: 'onnx::Sigmoid_977' (dtype=float32, shape=(1, 3, 40, 40, 7)) with 'onnx::Sigmoid_977' (dtype=float32, shape=(1, 3, 40, 40, 7))
[I] Tolerance: [abs=1e-05, rel=1e-05] | Checking elemwise error
[I] trt-runner-N0-09/14/22-11:08:17: onnx::Sigmoid_977 | Stats: mean=-1.3664, std-dev=3.6837, var=13.57, median=-0.054346, min=-13.749 at (0, 2, 39, 12, 4), max=2.6531 at (0, 2, 26, 5, 5), avg-magnitude=1.9664
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(-13.7 , -12.1 ) | 45 |
(-12.1 , -10.5 ) | 1769 | ###
(-10.5 , -8.83 ) | 2720 | #####
(-8.83 , -7.19 ) | 266 |
(-7.19 , -5.55 ) | 0 |
(-5.55 , -3.91 ) | 0 |
(-3.91 , -2.27 ) | 103 |
(-2.27 , -0.627) | 4179 | ########
(-0.627, 1.01 ) | 20353 | ########################################
(1.01 , 2.65 ) | 4165 | ########
[I] onnxrt-runner-N0-09/14/22-11:08:17: onnx::Sigmoid_977 | Stats: mean=-1.3664, std-dev=3.6837, var=13.57, median=-0.054349, min=-13.749 at (0, 2, 39, 12, 4), max=2.6531 at (0, 2, 26, 5, 5), avg-magnitude=1.9664
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(-13.7 , -12.1 ) | 45 |
(-12.1 , -10.5 ) | 1769 | ###
(-10.5 , -8.83 ) | 2720 | #####
(-8.83 , -7.19 ) | 266 |
(-7.19 , -5.55 ) | 0 |
(-5.55 , -3.91 ) | 0 |
(-3.91 , -2.27 ) | 103 |
(-2.27 , -0.627) | 4179 | ########
(-0.627, 1.01 ) | 20353 | ########################################
(1.01 , 2.65 ) | 4165 | ########
[I] Error Metrics: onnx::Sigmoid_977
[I] Minimum Required Tolerance: elemwise error | [abs=4.4823e-05] OR [rel=0.52614] (requirements may be lower if both abs/rel tolerances are set)
[I] Absolute Difference | Stats: mean=3.733e-06, std-dev=4.1101e-06, var=1.6893e-11, median=2.4438e-06, min=0 at (0, 0, 0, 13, 4), max=4.4823e-05 at (0, 0, 21, 11, 4), avg-magnitude=3.733e-06
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(0 , 4.48e-06) | 24109 | ########################################
(4.48e-06, 8.96e-06) | 6489 | ##########
(8.96e-06, 1.34e-05) | 1825 | ###
(1.34e-05, 1.79e-05) | 609 | #
(1.79e-05, 2.24e-05) | 354 |
(2.24e-05, 2.69e-05) | 134 |
(2.69e-05, 3.14e-05) | 50 |
(3.14e-05, 3.59e-05) | 23 |
(3.59e-05, 4.03e-05) | 5 |
(4.03e-05, 4.48e-05) | 2 |
[I] Relative Difference | Stats: mean=5.4347e-05, std-dev=0.0029416, var=8.6528e-06, median=3.9592e-06, min=0 at (0, 0, 0, 13, 4), max=0.52614 at (0, 2, 29, 18, 0), avg-magnitude=5.4347e-05
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(0 , 0.0526) | 33598 | ########################################
(0.0526, 0.105 ) | 1 |
(0.105 , 0.158 ) | 0 |
(0.158 , 0.21 ) | 0 |
(0.21 , 0.263 ) | 0 |
(0.263 , 0.316 ) | 0 |
(0.316 , 0.368 ) | 0 |
(0.368 , 0.421 ) | 0 |
(0.421 , 0.474 ) | 0 |
(0.474 , 0.526 ) | 1 |
[E] FAILED | Difference exceeds tolerance (rel=1e-05, abs=1e-05)
[I] Comparing Output: 'onnx::Sigmoid_1316' (dtype=float32, shape=(1, 3, 20, 20, 7)) with 'onnx::Sigmoid_1316' (dtype=float32, shape=(1, 3, 20, 20, 7))
[I] Tolerance: [abs=1e-05, rel=1e-05] | Checking elemwise error
[I] trt-runner-N0-09/14/22-11:08:17: onnx::Sigmoid_1316 | Stats: mean=-1.261, std-dev=3.3875, var=11.475, median=-0.12893, min=-14.461 at (0, 2, 19, 0, 4), max=3.371 at (0, 2, 12, 3, 5), avg-magnitude=1.9811
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(-14.5 , -12.7 ) | 32 |
(-12.7 , -10.9 ) | 72 |
(-10.9 , -9.11 ) | 480 | #####
(-9.11 , -7.33 ) | 496 | #####
(-7.33 , -5.55 ) | 120 | #
(-5.55 , -3.76 ) | 0 |
(-3.76 , -1.98 ) | 253 | ##
(-1.98 , -0.195) | 2468 | ##########################
(-0.195, 1.59 ) | 3738 | ########################################
(1.59 , 3.37 ) | 741 | #######
[I] onnxrt-runner-N0-09/14/22-11:08:17: onnx::Sigmoid_1316 | Stats: mean=-1.261, std-dev=3.3875, var=11.475, median=-0.12893, min=-14.461 at (0, 2, 19, 0, 4), max=3.371 at (0, 2, 12, 3, 5), avg-magnitude=1.9811
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(-14.5 , -12.7 ) | 32 |
(-12.7 , -10.9 ) | 72 |
(-10.9 , -9.11 ) | 480 | #####
(-9.11 , -7.33 ) | 496 | #####
(-7.33 , -5.55 ) | 120 | #
(-5.55 , -3.76 ) | 0 |
(-3.76 , -1.98 ) | 253 | ##
(-1.98 , -0.195) | 2468 | ##########################
(-0.195, 1.59 ) | 3738 | ########################################
(1.59 , 3.37 ) | 741 | #######
[I] Error Metrics: onnx::Sigmoid_1316
[I] Minimum Required Tolerance: elemwise error | [abs=3.767e-05] OR [rel=0.11636] (requirements may be lower if both abs/rel tolerances are set)
[I] Absolute Difference | Stats: mean=3.1095e-06, std-dev=3.6627e-06, var=1.3415e-11, median=1.8603e-06, min=0 at (0, 0, 1, 7, 4), max=3.767e-05 at (0, 1, 9, 2, 4), avg-magnitude=3.1095e-06
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(0 , 3.77e-06) | 6009 | ########################################
(3.77e-06, 7.53e-06) | 1501 | #########
(7.53e-06, 1.13e-05) | 525 | ###
(1.13e-05, 1.51e-05) | 211 | #
(1.51e-05, 1.88e-05) | 87 |
(1.88e-05, 2.26e-05) | 48 |
(2.26e-05, 2.64e-05) | 13 |
(2.64e-05, 3.01e-05) | 2 |
(3.01e-05, 3.39e-05) | 1 |
(3.39e-05, 3.77e-05) | 3 |
[I] Relative Difference | Stats: mean=4.7262e-05, std-dev=0.0015394, var=2.3698e-06, median=2.2524e-06, min=0 at (0, 0, 1, 7, 4), max=0.11636 at (0, 0, 7, 16, 1), avg-magnitude=4.7262e-05
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(0 , 0.0116) | 8397 | ########################################
(0.0116, 0.0233) | 1 |
(0.0233, 0.0349) | 0 |
(0.0349, 0.0465) | 0 |
(0.0465, 0.0582) | 0 |
(0.0582, 0.0698) | 0 |
(0.0698, 0.0814) | 1 |
(0.0814, 0.0931) | 0 |
(0.0931, 0.105 ) | 0 |
(0.105 , 0.116 ) | 1 |
[E] FAILED | Difference exceeds tolerance (rel=1e-05, abs=1e-05)
[I] Comparing Output: 'output' (dtype=float32, shape=(1, 25200, 7)) with 'output' (dtype=float32, shape=(1, 25200, 7))
[I] Tolerance: [abs=1e-05, rel=1e-05] | Checking elemwise error
[I] trt-runner-N0-09/14/22-11:08:17: output | Stats: mean=106.78, std-dev=173.18, var=29992, median=18.356, min=5.2426e-07 at (0, 25180, 4), max=637.66 at (0, 2639, 0), avg-magnitude=106.78
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(5.24e-07, 63.8) | 122026 | ########################################
(63.8 , 128 ) | 9459 | ###
(128 , 191 ) | 7071 | ##
(191 , 255 ) | 5742 | #
(255 , 319 ) | 5580 | #
(319 , 383 ) | 5685 | #
(383 , 446 ) | 5435 | #
(446 , 510 ) | 5163 | #
(510 , 574 ) | 5120 | #
(574 , 638 ) | 5119 | #
[I] onnxrt-runner-N0-09/14/22-11:08:17: output | Stats: mean=106.78, std-dev=173.18, var=29992, median=18.356, min=5.3644e-07 at (0, 25180, 4), max=637.66 at (0, 2639, 0), avg-magnitude=106.78
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(5.24e-07, 63.8) | 122026 | ########################################
(63.8 , 128 ) | 9459 | ###
(128 , 191 ) | 7071 | ##
(191 , 255 ) | 5742 | #
(255 , 319 ) | 5580 | #
(319 , 383 ) | 5685 | #
(383 , 446 ) | 5435 | #
(446 , 510 ) | 5163 | #
(510 , 574 ) | 5120 | #
(574 , 638 ) | 5119 | #
[I] Error Metrics: output
[I] Minimum Required Tolerance: elemwise error | [abs=0.002594] OR [rel=0.077808] (requirements may be lower if both abs/rel tolerances are set)
[I] Absolute Difference | Stats: mean=3.5768e-05, std-dev=9.4047e-05, var=8.8448e-09, median=1.5497e-06, min=0 at (0, 4, 5), max=0.002594 at (0, 23419, 3), avg-magnitude=3.5768e-05
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(0 , 0.000259) | 171100 | ########################################
(0.000259, 0.000519) | 3908 |
(0.000519, 0.000778) | 960 |
(0.000778, 0.00104 ) | 266 |
(0.00104 , 0.0013 ) | 94 |
(0.0013 , 0.00156 ) | 38 |
(0.00156 , 0.00182 ) | 23 |
(0.00182 , 0.00208 ) | 5 |
(0.00208 , 0.00233 ) | 4 |
(0.00233 , 0.00259 ) | 2 |
[I] Relative Difference | Stats: mean=0.00037051, std-dev=0.0016869, var=2.8458e-06, median=1.2529e-06, min=0 at (0, 4, 5), max=0.077808 at (0, 24800, 4), avg-magnitude=0.00037051
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(0 , 0.00778) | 174425 | ########################################
(0.00778, 0.0156 ) | 1618 |
(0.0156 , 0.0233 ) | 267 |
(0.0233 , 0.0311 ) | 61 |
(0.0311 , 0.0389 ) | 16 |
(0.0389 , 0.0467 ) | 6 |
(0.0467 , 0.0545 ) | 2 |
(0.0545 , 0.0622 ) | 3 |
(0.0622 , 0.07 ) | 1 |
(0.07 , 0.0778 ) | 1 |
[E] FAILED | Difference exceeds tolerance (rel=1e-05, abs=1e-05)
[E] FAILED | Mismatched outputs: ['onnx::Sigmoid_638', 'onnx::Sigmoid_977', 'onnx::Sigmoid_1316', 'output']
[!] FAILED | Command: /usr/local/bin/polygraphy run best_units.onnx --trt --onnxrt --input-shapes images:[1,3,640,640]
As you can see the network returns “FAILED”, there are some warnings
Environment
Device: Jetson AGX Orin Dev Kit
Jetpack: 5.0.2
Baremetal or Container (if container which image + tag): Container nvidia/l4t-ml:r35.1.0-py3
Relevant Files
yolor_csp.pt - Google Drive the weights of the net in .pt format
Issue Reproduction
Convert the .pt to onnx using the above script
run poligraphy to check output
polygraphy run yolo_csp.onnx --trt --onnxrt
Further testing
I tried to sanitize my model using this:
polygraphy surgeon sanitize yolor_csp.onnx --fold-constants -o folded.onnx
The result is of the polygraphy run is very similar:
[I] trt-runner-N0-09/14/22-13:32:43 | Activating and starting inference
[09/14/2022-13:32:47] [TRT] [W] onnx2trt_utils.cpp:367: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[I] Configuring with profiles: [Profile().add('images', min=[1, 3, 640, 640], opt=[1, 3, 640, 640], max=[1, 3, 640, 640])]
[I] Building engine with configuration:
Workspace | 16777216 bytes (16.00 MiB)
Precision | TF32: False, FP16: False, INT8: False, Obey Precision Constraints: False, Strict Types: False
Tactic Sources | ['CUBLAS', 'CUBLAS_LT', 'CUDNN', 'EDGE_MASK_CONVOLUTIONS']
Safety Restricted | False
Profiles | 1 profile(s)
[09/14/2022-13:34:47] [TRT] [W] The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
[09/14/2022-13:34:47] [TRT] [W] The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
[I] Finished engine building in 120.563 seconds
[I] trt-runner-N0-09/14/22-13:32:43
---- Inference Input(s) ----
{images [dtype=float32, shape=(1, 3, 640, 640)]}
[I] trt-runner-N0-09/14/22-13:32:43
---- Inference Output(s) ----
{onnx::Sigmoid_638 [dtype=float32, shape=(1, 3, 80, 80, 7)],
onnx::Sigmoid_977 [dtype=float32, shape=(1, 3, 40, 40, 7)],
onnx::Sigmoid_1316 [dtype=float32, shape=(1, 3, 20, 20, 7)],
output [dtype=float32, shape=(1, 25200, 7)]}
[I] trt-runner-N0-09/14/22-13:32:43 | Completed 1 iteration(s) in 50.5 ms | Average inference time: 50.5 ms.
[I] onnxrt-runner-N0-09/14/22-13:32:43 | Activating and starting inference
[I] Creating ONNX-Runtime Inference Session with providers: ['CPUExecutionProvider']
[I] onnxrt-runner-N0-09/14/22-13:32:43
---- Inference Input(s) ----
{images [dtype=float32, shape=(1, 3, 640, 640)]}
[I] onnxrt-runner-N0-09/14/22-13:32:43
---- Inference Output(s) ----
{output [dtype=float32, shape=(1, 25200, 7)],
onnx::Sigmoid_638 [dtype=float32, shape=(1, 3, 80, 80, 7)],
onnx::Sigmoid_977 [dtype=float32, shape=(1, 3, 40, 40, 7)],
onnx::Sigmoid_1316 [dtype=float32, shape=(1, 3, 20, 20, 7)]}
[I] onnxrt-runner-N0-09/14/22-13:32:43 | Completed 1 iteration(s) in 784.9 ms | Average inference time: 784.9 ms.
[I] Accuracy Comparison | trt-runner-N0-09/14/22-13:32:43 vs. onnxrt-runner-N0-09/14/22-13:32:43
[I] Comparing Output: 'onnx::Sigmoid_638' (dtype=float32, shape=(1, 3, 80, 80, 7)) with 'onnx::Sigmoid_638' (dtype=float32, shape=(1, 3, 80, 80, 7))
[I] Tolerance: [abs=1e-05, rel=1e-05] | Checking elemwise error
[I] trt-runner-N0-09/14/22-13:32:43: onnx::Sigmoid_638 | Stats: mean=-1.5821, std-dev=4.023, var=16.185, median=-0.035941, min=-13.831 at (0, 0, 69, 1, 4), max=2.5998 at (0, 0, 65, 56, 5), avg-magnitude=1.9761
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(-13.8 , -12.2 ) | 3189 | #
(-12.2 , -10.5 ) | 12571 | #####
(-10.5 , -8.9 ) | 3174 | #
(-8.9 , -7.26 ) | 265 |
(-7.26 , -5.62 ) | 1 |
(-5.62 , -3.97 ) | 0 |
(-3.97 , -2.33 ) | 14 |
(-2.33 , -0.686) | 9883 | ###
(-0.686, 0.957 ) | 99604 | ########################################
(0.957 , 2.6 ) | 5699 | ##
[I] onnxrt-runner-N0-09/14/22-13:32:43: onnx::Sigmoid_638 | Stats: mean=-1.5821, std-dev=4.023, var=16.185, median=-0.035944, min=-13.831 at (0, 0, 69, 1, 4), max=2.5998 at (0, 0, 65, 56, 5), avg-magnitude=1.9761
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(-13.8 , -12.2 ) | 3189 | #
(-12.2 , -10.5 ) | 12571 | #####
(-10.5 , -8.9 ) | 3174 | #
(-8.9 , -7.26 ) | 265 |
(-7.26 , -5.62 ) | 1 |
(-5.62 , -3.97 ) | 0 |
(-3.97 , -2.33 ) | 14 |
(-2.33 , -0.686) | 9883 | ###
(-0.686, 0.957 ) | 99604 | ########################################
(0.957 , 2.6 ) | 5699 | ##
[I] Error Metrics: onnx::Sigmoid_638
[I] Minimum Required Tolerance: elemwise error | [abs=6.9618e-05] OR [rel=1.4264] (requirements may be lower if both abs/rel tolerances are set)
[I] Absolute Difference | Stats: mean=5.3915e-06, std-dev=6.0069e-06, var=3.6082e-11, median=3.5465e-06, min=0 at (0, 0, 0, 4, 5), max=6.9618e-05 at (0, 1, 44, 42, 4), avg-magnitude=5.3915e-06
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(0 , 6.96e-06) | 101071 | ########################################
(6.96e-06, 1.39e-05) | 22680 | ########
(1.39e-05, 2.09e-05) | 6195 | ##
(2.09e-05, 2.78e-05) | 2783 | #
(2.78e-05, 3.48e-05) | 1055 |
(3.48e-05, 4.18e-05) | 382 |
(4.18e-05, 4.87e-05) | 160 |
(4.87e-05, 5.57e-05) | 42 |
(5.57e-05, 6.27e-05) | 21 |
(6.27e-05, 6.96e-05) | 11 |
[I] Relative Difference | Stats: mean=7.0141e-05, std-dev=0.0041236, var=1.7004e-05, median=7.4882e-06, min=0 at (0, 0, 0, 4, 5), max=1.4264 at (0, 1, 12, 7, 0), avg-magnitude=7.0141e-05
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(0 , 0.143) | 134397 | ########################################
(0.143, 0.285) | 2 |
(0.285, 0.428) | 0 |
(0.428, 0.571) | 0 |
(0.571, 0.713) | 0 |
(0.713, 0.856) | 0 |
(0.856, 0.998) | 0 |
(0.998, 1.14 ) | 0 |
(1.14 , 1.28 ) | 0 |
(1.28 , 1.43 ) | 1 |
[E] FAILED | Difference exceeds tolerance (rel=1e-05, abs=1e-05)
[I] Comparing Output: 'onnx::Sigmoid_977' (dtype=float32, shape=(1, 3, 40, 40, 7)) with 'onnx::Sigmoid_977' (dtype=float32, shape=(1, 3, 40, 40, 7))
[I] Tolerance: [abs=1e-05, rel=1e-05] | Checking elemwise error
[I] trt-runner-N0-09/14/22-13:32:43: onnx::Sigmoid_977 | Stats: mean=-1.3664, std-dev=3.6837, var=13.57, median=-0.054345, min=-13.749 at (0, 2, 39, 12, 4), max=2.6531 at (0, 2, 26, 5, 5), avg-magnitude=1.9664
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(-13.7 , -12.1 ) | 45 |
(-12.1 , -10.5 ) | 1769 | ###
(-10.5 , -8.83 ) | 2720 | #####
(-8.83 , -7.19 ) | 266 |
(-7.19 , -5.55 ) | 0 |
(-5.55 , -3.91 ) | 0 |
(-3.91 , -2.27 ) | 103 |
(-2.27 , -0.627) | 4179 | ########
(-0.627, 1.01 ) | 20353 | ########################################
(1.01 , 2.65 ) | 4165 | ########
[I] onnxrt-runner-N0-09/14/22-13:32:43: onnx::Sigmoid_977 | Stats: mean=-1.3664, std-dev=3.6837, var=13.57, median=-0.054349, min=-13.749 at (0, 2, 39, 12, 4), max=2.6531 at (0, 2, 26, 5, 5), avg-magnitude=1.9664
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(-13.7 , -12.1 ) | 45 |
(-12.1 , -10.5 ) | 1769 | ###
(-10.5 , -8.83 ) | 2720 | #####
(-8.83 , -7.19 ) | 266 |
(-7.19 , -5.55 ) | 0 |
(-5.55 , -3.91 ) | 0 |
(-3.91 , -2.27 ) | 103 |
(-2.27 , -0.627) | 4179 | ########
(-0.627, 1.01 ) | 20353 | ########################################
(1.01 , 2.65 ) | 4165 | ########
[I] Error Metrics: onnx::Sigmoid_977
[I] Minimum Required Tolerance: elemwise error | [abs=4.2915e-05] OR [rel=0.47712] (requirements may be lower if both abs/rel tolerances are set)
[I] Absolute Difference | Stats: mean=3.7101e-06, std-dev=4.0824e-06, var=1.6666e-11, median=2.414e-06, min=0 at (0, 0, 0, 30, 3), max=4.2915e-05 at (0, 0, 21, 11, 4), avg-magnitude=3.7101e-06
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(0 , 4.29e-06) | 23717 | ########################################
(4.29e-06, 8.58e-06) | 6467 | ##########
(8.58e-06, 1.29e-05) | 2033 | ###
(1.29e-05, 1.72e-05) | 723 | #
(1.72e-05, 2.15e-05) | 412 |
(2.15e-05, 2.57e-05) | 140 |
(2.57e-05, 3e-05 ) | 72 |
(3e-05 , 3.43e-05) | 26 |
(3.43e-05, 3.86e-05) | 6 |
(3.86e-05, 4.29e-05) | 4 |
[I] Relative Difference | Stats: mean=5.341e-05, std-dev=0.0026877, var=7.2235e-06, median=3.9517e-06, min=0 at (0, 0, 0, 30, 3), max=0.47712 at (0, 2, 29, 18, 0), avg-magnitude=5.341e-05
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(0 , 0.0477) | 33597 | ########################################
(0.0477, 0.0954) | 2 |
(0.0954, 0.143 ) | 0 |
(0.143 , 0.191 ) | 0 |
(0.191 , 0.239 ) | 0 |
(0.239 , 0.286 ) | 0 |
(0.286 , 0.334 ) | 0 |
(0.334 , 0.382 ) | 0 |
(0.382 , 0.429 ) | 0 |
(0.429 , 0.477 ) | 1 |
[E] FAILED | Difference exceeds tolerance (rel=1e-05, abs=1e-05)
[I] Comparing Output: 'onnx::Sigmoid_1316' (dtype=float32, shape=(1, 3, 20, 20, 7)) with 'onnx::Sigmoid_1316' (dtype=float32, shape=(1, 3, 20, 20, 7))
[I] Tolerance: [abs=1e-05, rel=1e-05] | Checking elemwise error
[I] trt-runner-N0-09/14/22-13:32:43: onnx::Sigmoid_1316 | Stats: mean=-1.261, std-dev=3.3875, var=11.475, median=-0.12893, min=-14.461 at (0, 2, 19, 0, 4), max=3.371 at (0, 2, 12, 3, 5), avg-magnitude=1.9811
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(-14.5 , -12.7 ) | 32 |
(-12.7 , -10.9 ) | 72 |
(-10.9 , -9.11 ) | 480 | #####
(-9.11 , -7.33 ) | 496 | #####
(-7.33 , -5.55 ) | 120 | #
(-5.55 , -3.76 ) | 0 |
(-3.76 , -1.98 ) | 253 | ##
(-1.98 , -0.195) | 2468 | ##########################
(-0.195, 1.59 ) | 3738 | ########################################
(1.59 , 3.37 ) | 741 | #######
[I] onnxrt-runner-N0-09/14/22-13:32:43: onnx::Sigmoid_1316 | Stats: mean=-1.261, std-dev=3.3875, var=11.475, median=-0.12893, min=-14.461 at (0, 2, 19, 0, 4), max=3.371 at (0, 2, 12, 3, 5), avg-magnitude=1.9811
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(-14.5 , -12.7 ) | 32 |
(-12.7 , -10.9 ) | 72 |
(-10.9 , -9.11 ) | 480 | #####
(-9.11 , -7.33 ) | 496 | #####
(-7.33 , -5.55 ) | 120 | #
(-5.55 , -3.76 ) | 0 |
(-3.76 , -1.98 ) | 253 | ##
(-1.98 , -0.195) | 2468 | ##########################
(-0.195, 1.59 ) | 3738 | ########################################
(1.59 , 3.37 ) | 741 | #######
[I] Error Metrics: onnx::Sigmoid_1316
[I] Minimum Required Tolerance: elemwise error | [abs=3.767e-05] OR [rel=0.11714] (requirements may be lower if both abs/rel tolerances are set)
[I] Absolute Difference | Stats: mean=3.1082e-06, std-dev=3.6633e-06, var=1.342e-11, median=1.8477e-06, min=0 at (0, 0, 0, 7, 6), max=3.767e-05 at (0, 1, 9, 2, 4), avg-magnitude=3.1082e-06
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(0 , 3.77e-06) | 6017 | ########################################
(3.77e-06, 7.53e-06) | 1492 | #########
(7.53e-06, 1.13e-05) | 531 | ###
(1.13e-05, 1.51e-05) | 208 | #
(1.51e-05, 1.88e-05) | 85 |
(1.88e-05, 2.26e-05) | 47 |
(2.26e-05, 2.64e-05) | 14 |
(2.64e-05, 3.01e-05) | 1 |
(3.01e-05, 3.39e-05) | 2 |
(3.39e-05, 3.77e-05) | 3 |
[I] Relative Difference | Stats: mean=4.685e-05, std-dev=0.0015497, var=2.4015e-06, median=2.2267e-06, min=0 at (0, 0, 0, 7, 6), max=0.11714 at (0, 0, 7, 16, 1), avg-magnitude=4.685e-05
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(0 , 0.0117) | 8397 | ########################################
(0.0117, 0.0234) | 1 |
(0.0234, 0.0351) | 0 |
(0.0351, 0.0469) | 0 |
(0.0469, 0.0586) | 0 |
(0.0586, 0.0703) | 0 |
(0.0703, 0.082 ) | 1 |
(0.082 , 0.0937) | 0 |
(0.0937, 0.105 ) | 0 |
(0.105 , 0.117 ) | 1 |
[E] FAILED | Difference exceeds tolerance (rel=1e-05, abs=1e-05)
[I] Comparing Output: 'output' (dtype=float32, shape=(1, 25200, 7)) with 'output' (dtype=float32, shape=(1, 25200, 7))
[I] Tolerance: [abs=1e-05, rel=1e-05] | Checking elemwise error
[I] trt-runner-N0-09/14/22-13:32:43: output | Stats: mean=106.78, std-dev=173.18, var=29992, median=18.356, min=5.2426e-07 at (0, 25180, 4), max=637.66 at (0, 2639, 0), avg-magnitude=106.78
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(5.24e-07, 63.8) | 122026 | ########################################
(63.8 , 128 ) | 9459 | ###
(128 , 191 ) | 7071 | ##
(191 , 255 ) | 5742 | #
(255 , 319 ) | 5580 | #
(319 , 383 ) | 5685 | #
(383 , 446 ) | 5435 | #
(446 , 510 ) | 5163 | #
(510 , 574 ) | 5120 | #
(574 , 638 ) | 5119 | #
[I] onnxrt-runner-N0-09/14/22-13:32:43: output | Stats: mean=106.78, std-dev=173.18, var=29992, median=18.356, min=5.3644e-07 at (0, 25180, 4), max=637.66 at (0, 2639, 0), avg-magnitude=106.78
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(5.24e-07, 63.8) | 122026 | ########################################
(63.8 , 128 ) | 9459 | ###
(128 , 191 ) | 7071 | ##
(191 , 255 ) | 5742 | #
(255 , 319 ) | 5580 | #
(319 , 383 ) | 5685 | #
(383 , 446 ) | 5435 | #
(446 , 510 ) | 5163 | #
(510 , 574 ) | 5120 | #
(574 , 638 ) | 5119 | #
[I] Error Metrics: output
[I] Minimum Required Tolerance: elemwise error | [abs=0.002594] OR [rel=0.077809] (requirements may be lower if both abs/rel tolerances are set)
[I] Absolute Difference | Stats: mean=3.569e-05, std-dev=9.3724e-05, var=8.7842e-09, median=1.5497e-06, min=0 at (0, 4, 0), max=0.002594 at (0, 23419, 3), avg-magnitude=3.569e-05
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(0 , 0.000259) | 171144 | ########################################
(0.000259, 0.000519) | 3850 |
(0.000519, 0.000778) | 991 |
(0.000778, 0.00104 ) | 251 |
(0.00104 , 0.0013 ) | 93 |
(0.0013 , 0.00156 ) | 40 |
(0.00156 , 0.00182 ) | 19 |
(0.00182 , 0.00208 ) | 8 |
(0.00208 , 0.00233 ) | 1 |
(0.00233 , 0.00259 ) | 3 |
[I] Relative Difference | Stats: mean=0.00037051, std-dev=0.0016869, var=2.8458e-06, median=1.2479e-06, min=0 at (0, 4, 0), max=0.077809 at (0, 24800, 4), avg-magnitude=0.00037051
[I] ---- Histogram ----
Bin Range | Num Elems | Visualization
(0 , 0.00778) | 174425 | ########################################
(0.00778, 0.0156 ) | 1618 |
(0.0156 , 0.0233 ) | 267 |
(0.0233 , 0.0311 ) | 61 |
(0.0311 , 0.0389 ) | 16 |
(0.0389 , 0.0467 ) | 6 |
(0.0467 , 0.0545 ) | 2 |
(0.0545 , 0.0622 ) | 3 |
(0.0622 , 0.07 ) | 1 |
(0.07 , 0.0778 ) | 1 |
[E] FAILED | Difference exceeds tolerance (rel=1e-05, abs=1e-05)
[E] FAILED | Mismatched outputs: ['onnx::Sigmoid_638', 'onnx::Sigmoid_977', 'onnx::Sigmoid_1316', 'output']
[!] FAILED | Command: /usr/local/bin/polygraphy run folded.onnx --trt --onnxrt
Am I doing something wrong or are some of the layers not accurately converted because of some incorrect casting which causes different outputs?