Conversion fails with `[weightsPtr.h::setCount::144] Error Code 2: Internal Error (Assertion count >= 0 failed.)`

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

While converting one of our models to TensorRT, it fails with the following error:

[X] Merging conv layers: Gemm_27 || Gemm_30
[weightsPtr.h::setCount::144] Error Code 2: Internal Error (Assertion count >= 0 failed. )

Environment

TensorRT Version: 8.6.0
GPU Type: 4090
Nvidia Driver Version: 520.61.05
CUDA Version: 11.8
CUDNN Version: 8.6.0
Operating System + Version: Ubuntu 22.04
Python Version (if applicable): 3.8
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

Attached is a script that will generate a minimal ONNX model which passes the model checker, but fails to convert to TensorRT and produces the above error.

main.py (1.3 KB)

Steps To Reproduce

pip install onnx numpy tensorrt polygraphy
python main.py
polygraphy convert -vvvv --convert-to trt --output trt-bug.trt trt-bug.onnx
2

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

4

Is there a better place to report TensorRT bugs?