Profiling fails with [E] Error[1]: [executionContext.cpp::syncShapeBindingsToDevice::1990] Error Code 1: Cuda Runtime (context is destroyed)

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

While running trtexec with a command
trtexec --loadEngine=whisper-tiny-decoder-sim.engine --exportProfile=whisper-tiny-decoder-sim.engine.profile.json --verbose --noDataTransfers --useCudaGraph --separateProfileRun,
I’ve got the following error.

[08/28/2023-06:09:53] [E] Error[1]: [executionContext.cpp::syncShapeBindingsToDevice::1990] Error Code 1: Cuda Runtime (context is destroyed)
trtexec: samples/common/sampleInference.cpp:844: void sample::{anonymous}::Iteration<ContextType>::createEnqueueFunction(const sample::InferenceOptions&, nvinfer1::IExecutionContext&, sample::Bindings&) [with ContextType = nvinfer1::IExecutionContext]: Assertion `ret' failed.

I found out without the --separateProfileRun option on the command, it runs successfully without the error.
However I believe it’s more accurate to get the latency with this option on.
I wonder if it could be a bug or if there’s any solution recommended.

Environment

TensorRT Version: 8.5.2-1+cuda11.8
GPU Type: RTX A6000
Nvidia Driver Version: 510.108.03
CUDA Version: 12.0
CUDNN Version: 8
Operating System + Version: Ubuntu 20.04
Python Version (if applicable): 3.8.10
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag): nvcr.io/nvidia/tensorrt 23.01-py3

Relevant Files

Attaching the input onnx file [link]

Steps To Reproduce

  1. Build trtexec on the above docker image
  2. trtexec --loadEngine=whisper-tiny-decoder-sim.engine --exportProfile=whisper-tiny-decoder-sim.engine.profile.json --verbose --noDataTransfers --useCudaGraph --separateProfileRun
2

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

4

Hi Thank you for the quick response.

I attached the ONNX model link in the “Relevant Files” above.
Also, the error log I attached is from --verbose as I attached the command too.
By the way, with the ONNX checker you’ve shared, there’s no exception.

Thank you!