Error an ILoopOutputLayer cannot be used to compute a shape tensor

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

Our model is not able to run in TensorRT, I have tried both running in ORT with TensorrtExecutionProvider and using trtexec to convert it to a plan.

For compliance reason, I cannot share my full model, but I can build a mini one to repro my issue.

please check the model binary directly:
tiny_model.onnx (15.9 KB)
OR build from the script:
tiny_model_builder.py (1.1 KB)

  • Using ORT + TensorrtExecutionProvider reports error:

    Error Code 4: Internal Error ((Unnamed Layer* 23) [LoopOutput]: an ILoopOutputLayer cannot be used to compute a shape tensor)
    ORT_repro.py (690 Bytes)

  • Using trtexec --onnx=tiny_model.onnx --saveEngine=tiny_model.trt --memPoolSize=workspace:10000 --minShapes=input:1x1 --maxShapes=input:1x512 --optShapes=input:1x64 --device=0 --verbose will show below error:
    [02/21/2023-14:36:04] [E] Error[2]: [topSort.cpp::trivialChoice::314] Error Code 2: Internal Error (Assertion c != kNoColor failed. )
    [02/21/2023-14:36:04] [E] Error[2]: [builder.cpp::buildSerializedNetwork::738] Error Code 2: Internal Error (Assertion engine != nullptr failed. )

Environment

TensorRT Version: 8.5.0
GPU Type: T4
Nvidia Driver Version: 470.161.03
CUDA Version: 11.8
CUDNN Version:
Operating System + Version: Ubuntu20.04
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)

Steps To Reproduce

Please include:

  • Exact steps/commands to build your repro
  • Exact steps/commands to run your repro
  • Full traceback of errors encountered
4

Hi @niuzheng168 ,
Apologies for the delayed response.
Can you please share the verbose logs with us, meanwhile we are trying to reproduce the issue from our end.

Thanks

5

I have the same issue.

trtexec_log.txt (936.3 KB)

This is from trying to run/convert an ONNX model to TRT. Using ONNX simplifier didn’t help. The model was fetched from TensorFlow Model Zoo and converted to ONNX. TRT conversion always failed due to the input being UINT8. I used the following script to change it to FP32.

import onnx_graphsurgeon as gs
import onnx
import numpy as np

graph = gs.import_onnx(onnx.load("C:\\Users\\user\\test\\cam1\\model2.onnx"))
for inp in graph.inputs:
    inp.dtype = np.float32

onnx.save(gs.export_onnx(graph), "updated_cam1_model2.onnx")

The ONNX model is too big to post here, so here is a GDrive link:
ONNX model link
This is the model after FP32 script, without simplifying it.

I used the following container to do the conversion:
docker run --gpus all -it --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -v ~/workspace:/workspace --rm nvcr.io/nvidia/tensorflow:23.08-tf2-py3