Use trtexec to run LSTM int8 calibrator failed with Error Code 2: Internal Error (Assertion mIndex >= 0 failed. symbol is not concrete)

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

use trtexec to run int8 calibrator of a simple LSTM network failed with: “[E] Error[2]: [graph.cpp::getDefinition::356] Error Code 2: Internal Error (Assertion mIndex >= 0 failed. symbol is not concrete)”

Environment

TensorRT Version: 8.4.12
GPU Type: GTX3080
Nvidia Driver Version: 470.199.02
CUDA Version: 11.4
CUDNN Version:
Operating System + Version: ubuntu
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

onnx file:
single_LSTM.zip (3.7 MB)

Steps To Reproduce

  1. download single_LSTM.zip and unzip it, get single_LSTM.onnx
  2. command: trtexec --onnx=single_LSTM.onnx --int8 --fp16 --saveEngine=single_LSTM.trt
  3. LOG:
    &&&& RUNNING TensorRT.trtexec [TensorRT v8412] # trtexec --onnx=single_LSTM.onnx --int8 --fp16 --saveEngine=single_LSTM.trt
    [10/11/2023-09:21:10] [I] === Model Options ===
    [10/11/2023-09:21:10] [I] Format: ONNX
    [10/11/2023-09:21:10] [I] Model: single_LSTM.onnx
    [10/11/2023-09:21:10] [I] Output:
    [10/11/2023-09:21:10] [I] === Build Options ===
    [10/11/2023-09:21:10] [I] Max batch: explicit batch
    [10/11/2023-09:21:10] [I] Memory Pools: workspace: default, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default
    [10/11/2023-09:21:10] [I] minTiming: 1
    [10/11/2023-09:21:10] [I] avgTiming: 8
    [10/11/2023-09:21:10] [I] Precision: FP32+FP16+INT8
    [10/11/2023-09:21:10] [I] LayerPrecisions:
    [10/11/2023-09:21:10] [I] Calibration: Dynamic
    [10/11/2023-09:21:10] [I] Refit: Disabled
    [10/11/2023-09:21:10] [I] Sparsity: Disabled
    [10/11/2023-09:21:10] [I] Safe mode: Disabled
    [10/11/2023-09:21:10] [I] DirectIO mode: Disabled
    [10/11/2023-09:21:10] [I] Restricted mode: Disabled
    [10/11/2023-09:21:10] [I] Build only: Disabled
    [10/11/2023-09:21:10] [I] Save engine: single_LSTM.trt
    [10/11/2023-09:21:10] [I] Load engine:
    [10/11/2023-09:21:10] [I] Profiling verbosity: 0
    [10/11/2023-09:21:10] [I] Tactic sources: Using default tactic sources
    [10/11/2023-09:21:10] [I] timingCacheMode: local
    [10/11/2023-09:21:10] [I] timingCacheFile:
    [10/11/2023-09:21:10] [I] Input(s)s format: fp32:CHW
    [10/11/2023-09:21:10] [I] Output(s)s format: fp32:CHW
    [10/11/2023-09:21:10] [I] Input build shapes: model
    [10/11/2023-09:21:10] [I] Input calibration shapes: model
    [10/11/2023-09:21:10] [I] === System Options ===
    [10/11/2023-09:21:10] [I] Device: 0
    [10/11/2023-09:21:10] [I] DLACore:
    [10/11/2023-09:21:10] [I] Plugins:
    [10/11/2023-09:21:10] [I] === Inference Options ===
    [10/11/2023-09:21:10] [I] Batch: Explicit
    [10/11/2023-09:21:10] [I] Input inference shapes: model
    [10/11/2023-09:21:10] [I] Iterations: 10
    [10/11/2023-09:21:10] [I] Duration: 3s (+ 200ms warm up)
    [10/11/2023-09:21:10] [I] Sleep time: 0ms
    [10/11/2023-09:21:10] [I] Idle time: 0ms
    [10/11/2023-09:21:10] [I] Streams: 1
    [10/11/2023-09:21:10] [I] ExposeDMA: Disabled
    [10/11/2023-09:21:10] [I] Data transfers: Enabled
    [10/11/2023-09:21:10] [I] Spin-wait: Disabled
    [10/11/2023-09:21:10] [I] Multithreading: Disabled
    [10/11/2023-09:21:10] [I] CUDA Graph: Disabled
    [10/11/2023-09:21:10] [I] Separate profiling: Disabled
    [10/11/2023-09:21:10] [I] Time Deserialize: Disabled
    [10/11/2023-09:21:10] [I] Time Refit: Disabled
    [10/11/2023-09:21:10] [I] Inputs:
    [10/11/2023-09:21:10] [I] === Reporting Options ===
    [10/11/2023-09:21:10] [I] Verbose: Disabled
    [10/11/2023-09:21:10] [I] Averages: 10 inferences
    [10/11/2023-09:21:10] [I] Percentile: 99
    [10/11/2023-09:21:10] [I] Dump refittable layers:Disabled
    [10/11/2023-09:21:10] [I] Dump output: Disabled
    [10/11/2023-09:21:10] [I] Profile: Disabled
    [10/11/2023-09:21:10] [I] Export timing to JSON file:
    [10/11/2023-09:21:10] [I] Export output to JSON file:
    [10/11/2023-09:21:10] [I] Export profile to JSON file:
    [10/11/2023-09:21:10] [I]
    [10/11/2023-09:21:10] [I] === Device Information ===
    [10/11/2023-09:21:10] [I] Selected Device: NVIDIA GeForce RTX 3080 Laptop GPU
    [10/11/2023-09:21:10] [I] Compute Capability: 8.6
    [10/11/2023-09:21:10] [I] SMs: 48
    [10/11/2023-09:21:10] [I] Compute Clock Rate: 1.545 GHz
    [10/11/2023-09:21:10] [I] Device Global Memory: 16125 MiB
    [10/11/2023-09:21:10] [I] Shared Memory per SM: 100 KiB
    [10/11/2023-09:21:10] [I] Memory Bus Width: 256 bits (ECC disabled)
    [10/11/2023-09:21:10] [I] Memory Clock Rate: 7.001 GHz
    [10/11/2023-09:21:10] [I]
    [10/11/2023-09:21:10] [I] TensorRT version: 8.4.12
    [10/11/2023-09:21:10] [I] [TRT] [MemUsageChange] Init CUDA: CPU +372, GPU +0, now: CPU 381, GPU 415 (MiB)
    [10/11/2023-09:21:11] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +323, GPU +102, now: CPU 723, GPU 517 (MiB)
    [10/11/2023-09:21:11] [I] Start parsing network model
    [10/11/2023-09:21:11] [I] [TRT] ----------------------------------------------------------------
    [10/11/2023-09:21:11] [I] [TRT] Input filename: single_LSTM.onnx
    [10/11/2023-09:21:11] [I] [TRT] ONNX IR version: 0.0.9
    [10/11/2023-09:21:11] [I] [TRT] Opset version: 11
    [10/11/2023-09:21:11] [I] [TRT] Producer name:
    [10/11/2023-09:21:11] [I] [TRT] Producer version:
    [10/11/2023-09:21:11] [I] [TRT] Domain:
    [10/11/2023-09:21:11] [I] [TRT] Model version: 0
    [10/11/2023-09:21:11] [I] [TRT] Doc string:
    [10/11/2023-09:21:11] [I] [TRT] ----------------------------------------------------------------
    [10/11/2023-09:21:11] [W] [TRT] onnx2trt_utils.cpp:367: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
    [10/11/2023-09:21:11] [I] Finish parsing network model
    [10/11/2023-09:21:11] [W] [TRT] Calibrator is not being used. Users must provide dynamic range for all tensors that are not Int32 or Bool.
    [10/11/2023-09:21:11] [E] Error[2]: [graph.cpp::getDefinition::356] Error Code 2: Internal Error (Assertion mIndex >= 0 failed. symbol is not concrete)
    [10/11/2023-09:21:11] [E] Error[2]: [builder.cpp::buildSerializedNetwork::636] Error Code 2: Internal Error (Assertion engine != nullptr failed. )
    [10/11/2023-09:21:11] [E] Engine could not be created from network
    [10/11/2023-09:21:11] [E] Building engine failed
    [10/11/2023-09:21:11] [E] Failed to create engine from model or file.
    [10/11/2023-09:21:11] [E] Engine set up failed
    &&&& FAILED TensorRT.trtexec [TensorRT v8412] # trtexec --onnx=single_LSTM.onnx --int8 --fp16 --saveEngine=single_LSTM.trt
3

We recommend you to please try on the latest TensorRT version 8.6.1