TensorRT 7.1.3.4 Segmentation fault on Simple ONNX model

1034092330 · July 27, 2020, 6:57am

Description:

When I test my ONNX model on TensorRT 7.1.3.4, it cased segmentation fault error.
This ONNX model was passed on TensorRT 6.0.1.5 / 7.0.0.11 and only Failed on the newest 7.1.3.4.

Log:

 $ ./trtexec.exe --onnx=/d/onnx_test/simple_unsqueeze_opset11.onnx --explicitBatch --verbose
&&&& RUNNING TensorRT.trtexec # D:\nvidia\TensorRT-7.1.3.4.Windows10.x86_64.cuda-11.0.cudnn8.0\TensorRT-7.1.3.4\                                             bin\trtexec.exe --onnx=D:/onnx_test/simple_unsqueeze_opset11.onnx --explicitBatch --verbose
[07/27/2020-15:19:24] [I] === Model Options ===
[07/27/2020-15:19:24] [I] Format: ONNX
[07/27/2020-15:19:24] [I] Model: D:/onnx_test/simple_unsqueeze_opset11.onnx
[07/27/2020-15:19:24] [I] Output:
[07/27/2020-15:19:24] [I] === Build Options ===
[07/27/2020-15:19:24] [I] Max batch: explicit
[07/27/2020-15:19:24] [I] Workspace: 16 MB
[07/27/2020-15:19:24] [I] minTiming: 1
[07/27/2020-15:19:24] [I] avgTiming: 8
[07/27/2020-15:19:24] [I] Precision: FP32
[07/27/2020-15:19:24] [I] Calibration:
[07/27/2020-15:19:24] [I] Safe mode: Disabled
[07/27/2020-15:19:24] [I] Save engine:
[07/27/2020-15:19:24] [I] Load engine:
[07/27/2020-15:19:24] [I] Builder Cache: Enabled
[07/27/2020-15:19:24] [I] NVTX verbosity: 0
[07/27/2020-15:19:24] [I] Inputs format: fp32:CHW
[07/27/2020-15:19:24] [I] Outputs format: fp32:CHW
[07/27/2020-15:19:24] [I] Input build shapes: model
[07/27/2020-15:19:24] [I] Input calibration shapes: model
[07/27/2020-15:19:24] [I] === System Options ===
[07/27/2020-15:19:24] [I] Device: 0
[07/27/2020-15:19:24] [I] DLACore:
[07/27/2020-15:19:24] [I] Plugins:
[07/27/2020-15:19:24] [I] === Inference Options ===
[07/27/2020-15:19:24] [I] Batch: Explicit
[07/27/2020-15:19:24] [I] Input inference shapes: model
[07/27/2020-15:19:24] [I] Iterations: 10
[07/27/2020-15:19:24] [I] Duration: 3s (+ 200ms warm up)
[07/27/2020-15:19:24] [I] Sleep time: 0ms
[07/27/2020-15:19:24] [I] Streams: 1
[07/27/2020-15:19:24] [I] ExposeDMA: Disabled
[07/27/2020-15:19:24] [I] Spin-wait: Disabled
[07/27/2020-15:19:24] [I] Multithreading: Disabled
[07/27/2020-15:19:24] [I] CUDA Graph: Disabled
[07/27/2020-15:19:24] [I] Skip inference: Disabled
[07/27/2020-15:19:24] [I] Inputs:
[07/27/2020-15:19:24] [I] === Reporting Options ===
[07/27/2020-15:19:24] [I] Verbose: Enabled
[07/27/2020-15:19:24] [I] Averages: 10 inferences
[07/27/2020-15:19:24] [I] Percentile: 99
[07/27/2020-15:19:24] [I] Dump output: Disabled
[07/27/2020-15:19:24] [I] Profile: Disabled
[07/27/2020-15:19:24] [I] Export timing to JSON file:
[07/27/2020-15:19:24] [I] Export output to JSON file:
[07/27/2020-15:19:24] [I] Export profile to JSON file:
[07/27/2020-15:19:24] [I]
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ::GridAnchor_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ::NMS_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ::Reorg_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ::Region_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ::Clip_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ::LReLU_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ::PriorBox_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ::Normalize_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ::RPROI_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ::BatchedNMS_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ::FlattenConcat_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ::CropAndResize
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ::DetectionLayer_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ::Proposal
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ::ProposalLayer_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ::PyramidROIAlign_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ::ResizeNearest_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ::Split
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ::SpecialSlice_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ::InstanceNormalization_TRT
----------------------------------------------------------------
Input filename:   D:/onnx_test/simple_unsqueeze_opset11.onnx
ONNX IR version:  0.0.6
Opset version:    11
Producer name:    pytorch
Producer version: 1.5
Domain:
Model version:    0
Doc string:
----------------------------------------------------------------
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ONNXTRT_NAMESPACE::GridAnchor_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ONNXTRT_NAMESPACE::NMS_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ONNXTRT_NAMESPACE::Reorg_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ONNXTRT_NAMESPACE::Region_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ONNXTRT_NAMESPACE::Clip_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ONNXTRT_NAMESPACE::LReLU_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ONNXTRT_NAMESPACE::PriorBox_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ONNXTRT_NAMESPACE::Normalize_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ONNXTRT_NAMESPACE::RPROI_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ONNXTRT_NAMESPACE::BatchedNMS_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ONNXTRT_NAMESPACE::FlattenConcat_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ONNXTRT_NAMESPACE::CropAndResize
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ONNXTRT_NAMESPACE::DetectionLayer_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ONNXTRT_NAMESPACE::Proposal
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ONNXTRT_NAMESPACE::ProposalLayer_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ONNXTRT_NAMESPACE::PyramidROIAlign_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ONNXTRT_NAMESPACE::ResizeNearest_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ONNXTRT_NAMESPACE::Split
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ONNXTRT_NAMESPACE::SpecialSlice_TRT
[07/27/2020-15:19:25] [V] [TRT] Plugin creator registration succeeded - ONNXTRT_NAMESPACE::InstanceNormalization                                             _TRT
[07/27/2020-15:19:25] [V] [TRT] ModelImporter.cpp:203: Adding network input: 0 with dtype: float32, dimensions:                                              (3, 300, 400)
[07/27/2020-15:19:25] [V] [TRT] ImporterContext.hpp:97: Registering tensor: 0 for ONNX tensor: 0
[07/27/2020-15:19:25] [V] [TRT] ModelImporter.cpp:107: Parsing node: Unsqueeze_0 [Unsqueeze]
[07/27/2020-15:19:25] [V] [TRT] ModelImporter.cpp:123: Searching for input: 0
[07/27/2020-15:19:25] [V] [TRT] ModelImporter.cpp:129: Unsqueeze_0 [Unsqueeze] inputs: [0 -> (3, 300, 400)],
[07/27/2020-15:19:25] [V] [TRT] onnx2trt_utils.cpp:1411: Original shape: (3, 300, 400), unsqueezing to: (1, 3, 3                                             00, 400)
[07/27/2020-15:19:25] [V] [TRT] ImporterContext.hpp:122: Registering layer: Unsqueeze_0 for ONNX node: Unsqueeze                                             _0
[07/27/2020-15:19:25] [V] [TRT] ImporterContext.hpp:97: Registering tensor: 1_1 for ONNX tensor: 1
[07/27/2020-15:19:25] [V] [TRT] ModelImporter.cpp:180: Unsqueeze_0 [Unsqueeze] outputs: [1 -> (1, 3, 300, 400)],                                             
[07/27/2020-15:19:25] [V] [TRT] ModelImporter.cpp:494: Marking 1_1 as output: 1
 ----- Parsing of ONNX model D:/onnx_test/simple_unsqueeze_opset11.onnx is Done ----
Segmentation fault

ONNX model details

This is an very simple ONNX model. The Netron show model graph as follow:

It is created by code:

import torch

class TestModel(torch.nn.Module):
  def __init__(self):
    super(TestModel, self).__init__()

  def forward(self, input):
    output = input.unsqueeze(0)
    return output

model = TestModel()
x = torch.ones((3, 300, 400))
torch.onnx.export(model, x, "simple_unsqueeze_opset11.onnx", opset_version=11)

simple_unsqueeze_opset11.zip (326 Bytes)

My dev enviroments:

RTX2060
Windows10
Python 3.7 64bit
Pytorch 1.5.1
TensorRT-7.1.3.4.Windows10.x86_64.cuda-11.0.cudnn8.0
TensorRT-7.0.0.11.Windows10.x86_64.cuda-10.2.cudnn7.6
TensorRT-6.0.1.5.Windows10.x86_64.cuda-10.1.cudnn7.6

Thank you all for your patience.

AakankshaS · July 27, 2020, 10:26am

Hi @1034092330,

I tried running your model using trtexec command, and it worked fine for me.
trtexec --onnx=simple_unsqueeze_opset11.onnx --verbose --explicitBatch --shapes=0:3x300x400 .
Can you please try this once.
Also please refer to this example

Thanks!

1034092330 · July 27, 2020, 2:47pm

Thanks for your kindly reply!

I tried running model use your command, but it failed with segmentation fault error again:

trtexec --onnx=simple_unsqueeze_opset11.onnx --verbose --explicitBatch --shapes=0:3x300x400

This error only occurs under TensorRT 7.1.3.4 in windows.

Maybe I will use 7.0.0.11 to continue my work…

AakankshaS · July 28, 2020, 4:37am

Hi @1034092330,

Please provide details on the platforms you are using:
o Linux distro and version
o GPU type
o Nvidia driver version
o CUDA version
o CUDNN version
o Python version [if using python]
o Tensorflow and PyTorch version
o TensorRT version

Thanks!