Using trtexec to generate an engine file from an ONNX works error with two RTSP input source

yanglicheng8 · December 26, 2023, 3:37am

Using trtexec to generate an engine file from an ONNX model works fine with one RTSP input source, but encounters an error when there are two or more RTSP input sources.

Environment

TensorRT Version: 8.5
GPU Type: Jetson Orin Nano(4GB)
Nvidia Driver Version:
CUDA Version: 11.4
CUDNN Version: 8.6.0
Operating System + Version: ubuntu 20.04
Python Version (if applicable):3.8.10
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag): baremetal

Relevant Files

ONNX model :
face.onnx.zip (10.3 MB)

command(I try all of this)

trtexec --shapes=images:6x3x640x640 --optShapes=images:2x3x640x640 --minShapes=images:1x3x640x640 --maxShapes=images:12x3x640x640 --onnx=face.onnx --saveEngine=face.engine

trtexec --onnx=face.onnx --saveEngine=face.engine

trtexec --shapes=images:6x3x640x640 --onnx=face.onnx --saveEngine=face.engine

Error Info

:00:07.557811059 125932     0x2c271330 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1909> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.2/sources/deepstream_python_apps/apps/deepstream_face_detection2/face3.engine
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: [Implicit Engine Info]: layers num: 2
0   INPUT  kFLOAT images          3x640x640       
1   OUTPUT kFLOAT output0         20x8400         

0:00:07.829792653 125932     0x2c271330 WARN                 nvinfer gstnvinfer.cpp:677:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:1841> [UID = 1]: Backend has maxBatchSize 1 whereas 2 has been requested
0:00:07.829884048 125932     0x2c271330 WARN                 nvinfer gstnvinfer.cpp:677:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2018> [UID = 1]: deserialized backend context :/opt/nvidia/deepstream/deepstream-6.2/sources/deepstream_python_apps/apps/deepstream_face_detection2/face3.engine failed to match config params, trying rebuild
0:00:07.847787874 125932     0x2c271330 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1923> [UID = 1]: Trying to create engine from model files
ERROR: failed to build network since there is no model file matched.
ERROR: failed to build network.
0:00:10.413748400 125932     0x2c271330 ERROR                nvinfer gstnvinfer.cpp:674:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1943> [UID = 1]: build engine file failed
0:00:10.625657789 125932     0x2c271330 ERROR                nvinfer gstnvinfer.cpp:674:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2029> [UID = 1]: build backend context failed
0:00:10.626082315 125932     0x2c271330 ERROR                nvinfer gstnvinfer.cpp:674:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1266> [UID = 1]: generate backend failed, check config file settings
0:00:10.628016683 125932     0x2c271330 WARN                 nvinfer gstnvinfer.cpp:888:gst_nvinfer_start:<primary-inference> error: Failed to create NvDsInferContext instance
0:00:10.628146223 125932     0x2c271330 WARN                 nvinfer gstnvinfer.cpp:888:gst_nvinfer_start:<primary-inference> error: Config file path: config_face_nvinfer2.txt, NvDsInfer Error: NVDSINFER_CONFIG_FAILED
[NvMultiObjectTracker] De-initialized

**PERF:  {'stream0': 0.0, 'stream1': 0.0} 

Error: gst-resource-error-quark: Failed to create NvDsInferContext instance (1): gstnvinfer.cpp(888): gst_nvinfer_start (): /GstPipeline:pipeline0/GstNvInfer:primary-inference:
Config file path: config_face_nvinfer2.txt, NvDsInfer Error: NVDSINFER_CONFIG_FAILED
Exiting app

DaneLLL · December 26, 2023, 3:56am

Hi,
Do you use DeepStream SDK? There is information about how you configure the RTSP sources in the description. Please share more detail

yanglicheng8 · December 27, 2023, 9:44am

Hi,
Thank you for your reply
I use DeepstreamSDK
This is my test code ,I put this code to this path

/opt/nvidia/deepstream/deepstream-6.2/sources/deepstream_python_apps/apps/

code-zip:
deepstream-imagedata-multistream-test.zip (16.2 MB)

command

run one input source is fine

sudo python3 deepstream_imagedata-multistream.py file:///opt/nvidia/deepstream/deepstream-6.2/sources/deepstream_python_apps/apps/deepstream-imagedata-multistream-test/darkface2.mp4 frames/

run above tow input source will show errors

sudo python3 deepstream_imagedata-multistream.py file:///opt/nvidia/deepstream/deepstream-6.2/sources/deepstream_python_apps/apps/deepstream-imagedata-multistream-test/darkface2.mp4 file:///opt/nvidia/deepstream/deepstream-6.2/sources/deepstream_python_apps/apps/deepstream-imagedata-multistream-test/darkface2.mp4  frames/

error

WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: [Implicit Engine Info]: layers num: 2
0   INPUT  kFLOAT images          3x640x640       
1   OUTPUT kFLOAT output0         20x8400         

0:00:07.486230283 136305     0x39905600 WARN                 nvinfer gstnvinfer.cpp:677:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:1841> [UID = 1]: Backend has maxBatchSize 1 whereas 2 has been requested
0:00:07.486305614 136305     0x39905600 WARN                 nvinfer gstnvinfer.cpp:677:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2018> [UID = 1]: deserialized backend context :/opt/nvidia/deepstream/deepstream-6.2/sources/deepstream_python_apps/apps/deepstream-imagedata-multistream-test/face1.engine failed to match config params, trying rebuild
0:00:07.502963990 136305     0x39905600 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1923> [UID = 1]: Trying to create engine from model files
ERROR: failed to build network since there is no model file matched.
ERROR: failed to build network.
0:00:10.463399348 136305     0x39905600 ERROR                nvinfer gstnvinfer.cpp:674:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1943> [UID = 1]: build engine file failed
0:00:10.682420140 136305     0x39905600 ERROR                nvinfer gstnvinfer.cpp:674:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2029> [UID = 1]: build backend context failed
0:00:10.682910206 136305     0x39905600 ERROR                nvinfer gstnvinfer.cpp:674:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1266> [UID = 1]: generate backend failed, check config file settings
0:00:10.684895459 136305     0x39905600 WARN                 nvinfer gstnvinfer.cpp:888:gst_nvinfer_start:<primary-inference> error: Failed to create NvDsInferContext instance
0:00:10.684992902 136305     0x39905600 WARN                 nvinfer gstnvinfer.cpp:888:gst_nvinfer_start:<primary-inference> error: Config file path: config_face_nvinfer.txt, NvDsInfer Error: NVDSINFER_CONFIG_FAILED

wish

I expect that after using the trtexec command to generate the engine file from onnx, it can run normally when running with multiple input source parameters.

DaneLLL · December 27, 2023, 10:26am

Hi,
The model has the setting max batch size = 1. Please set the value to 2 while converting the model.

yanglicheng8 · December 27, 2023, 11:51am

Hi:
Can you give me some tips ,code or command on how to modify it?
I tried to use onnx-modifier tools to change it, and used the trtexec command to change it, but the change could not be successful.

AastaLLL · December 28, 2023, 1:31am

Hi,

Please use the --batch=[#] flag to generate the model with trtexec.
You can find more examples in the below document:

Thanks.

yanglicheng8 · December 28, 2023, 2:11am

Hi:
Thank for your reply!
I refer to the information provided for execution. The executed commands and error reports are as follows:

cmd1:trtexec --onnx=face.onnx --saveEngine=face2.engine --batch=2 
cmd2:trtexec --onnx=face.onnx --saveEngine=face2.engine --batch=2 --maxBatch=10 --fp16

The --batch and --maxBatch flags should not be used when the input model is ONNX or when dynamic shapes are provided. Please use --optShapes and --shapes to set input shapes instead.

cmd1:trtexec --optShapes=images:2x3x640x640 --minShapes=images:1x3x640x640 --maxShapes=images:12x3x640x640 --onnx=face.onnx --saveEngine=face4.engine
cmd2:trtexec --shapes=images:6x3x640x640 --optShapes=images:2x3x640x640 --minShapes=images:1x3x640x640 --maxShapes=images:12x3x640x640 --onnx=face.onnx --saveEngine=face4.engine

Static model does not take explicit shapes since the shape of inference tensors will be determined by the model itself

Is there a problem with the onnx file?

Looking forward to your reply~

yanglicheng8 · December 28, 2023, 11:29am

Hi:

To add some content, when using the following commandtrtexec --onnx=face.onnx --saveEngine=face.engine, the engine file can be generated, but when using python to run the multi-input source Deepstream program, the error is as follows

:00:07.557811059 125932     0x2c271330 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1909> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.2/sources/deepstream_python_apps/apps/deepstream_face_detection2/face3.engine
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: [Implicit Engine Info]: layers num: 2
0   INPUT  kFLOAT images          3x640x640       
1   OUTPUT kFLOAT output0         20x8400         

0:00:07.829792653 125932     0x2c271330 WARN                 nvinfer gstnvinfer.cpp:677:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:1841> [UID = 1]: Backend has maxBatchSize 1 whereas 2 has been requested
0:00:07.829884048 125932     0x2c271330 WARN                 nvinfer gstnvinfer.cpp:677:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2018> [UID = 1]: deserialized backend context :/opt/nvidia/deepstream/deepstream-6.2/sources/deepstream_python_apps/apps/deepstream_face_detection2/face3.engine failed to match config params, trying rebuild
0:00:07.847787874 125932     0x2c271330 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1923> [UID = 1]: Trying to create engine from model files
ERROR: failed to build network since there is no model file matched.
ERROR: failed to build network.
0:00:10.413748400 125932     0x2c271330 ERROR                nvinfer gstnvinfer.cpp:674:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1943> [UID = 1]: build engine file failed
0:00:10.625657789 125932     0x2c271330 ERROR                nvinfer gstnvinfer.cpp:674:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2029> [UID = 1]: build backend context failed
0:00:10.626082315 125932     0x2c271330 ERROR                nvinfer gstnvinfer.cpp:674:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1266> [UID = 1]: generate backend failed, check config file settings
0:00:10.628016683 125932     0x2c271330 WARN                 nvinfer gstnvinfer.cpp:888:gst_nvinfer_start:<primary-inference> error: Failed to create NvDsInferContext instance
0:00:10.628146223 125932     0x2c271330 WARN                 nvinfer gstnvinfer.cpp:888:gst_nvinfer_start:<primary-inference> error: Config file path: config_face_nvinfer2.txt, NvDsInfer Error: NVDSINFER_CONFIG_FAILED
[NvMultiObjectTracker] De-initialized

**PERF:  {'stream0': 0.0, 'stream1': 0.0} 

Error: gst-resource-error-quark: Failed to create NvDsInferContext instance (1): gstnvinfer.cpp(888): gst_nvinfer_start (): /GstPipeline:pipeline0/GstNvInfer:primary-inference:
Config file path: config_face_nvinfer2.txt, NvDsInfer Error: NVDSINFER_CONFIG_FAILED
Exiting app

AastaLLL · January 2, 2024, 7:49am

Hi,

Thanks for the update.
We will try to reproduce it and update more info later.

Thanks.

AastaLLL · January 9, 2024, 7:21am

Hi,

Thanks for your patience and sorry for the late update.

Since your model is static, you will need to update the batch size by modifying the model parameter directly.
Please check below for the detailed steps:

1. Install dependencies

$ git clone https://github.com/NVIDIA/TensorRT.git
$ cd TensorRT/tools/onnx-graphsurgeon/
$ make build
$ python3 -m pip install dist/onnx_graphsurgeon-*-py2.py3-none-any.whl
$ pip3 install onnx

2. Modify the batch size to 2

Edit modify.py

import onnx
import onnx_graphsurgeon as gs

batch = 2

graph = gs.import_onnx(onnx.load("face.onnx"))

for input in graph.inputs:
    input.shape[0] = batch

reshape_nodes = [node for node in graph.nodes if node.op == "Reshape"]
for node in reshape_nodes:
    node.inputs[1].values[0] = batch

onnx.save(gs.export_onnx(graph), "dynamic.onnx")

$ python3 modify.py

3. Create TensorRT engine

$ /usr/src/tensorrt/bin/trtexec --onnx=dynamic.onnx --saveEngine=face1.engine

4. Test

Please also update config_face_nvinfer.txt

...
batch-size=2
...

Put the face1.engine to /opt/nvidia/deepstream/deepstream-6.3/sources/deepstream_python_apps/apps/deepstream-imagedata-multistream-test.

$ python3 deepstream_imagedata-multistream.py file:///opt/nvidia/deepstream/deepstream-6.3/sources/deepstream_python_apps/apps/deepstream-imagedata-multistream-test/darkface2.mp4 file:///opt/nvidia/deepstream/deepstream-6.3/sources/deepstream_python_apps/apps/deepstream-imagedata-multistream-test/darkface2.mp4  frames/
Frames will be saved in  2024-01-09-07-07-41
...
0:00:00.249330389 25336     0x3d997560 WARN                 nvinfer gstnvinfer.cpp:887:gst_nvinfer_start:<primary-inference> warning: NvInfer output-tensor-meta is enabled but init_params auto increase memory (auto-inc-mem) is disabled. The bufferpool will not be automatically resized.
0:00:04.110218019 25336     0x3d997560 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1988> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.3/sources/deepstream_python_apps/apps/deepstream-imagedata-multistream-test/face1.engine
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: [Implicit Engine Info]: layers num: 2
0   INPUT  kFLOAT images          3x640x640
1   OUTPUT kFLOAT output0         20x8400

0:00:04.273834882 25336     0x3d997560 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2091> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.3/sources/deepstream_python_apps/apps/deepstream-imagedata-multistream-test/face1.engine
0:00:04.441997274 25336     0x3d997560 INFO                 nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary-inference> [UID 1]: Load new model:config_face_nvinfer.txt successfully
...

**PERF:  {'stream0': 0.0, 'stream1': 0.0}

arr1=1
arr1=1
arr1=2
arr1=2

Thanks.

yanglicheng8 · January 15, 2024, 12:43pm

Hi:
Thanks a lot ! This is helpful!

system · January 29, 2024, 12:44pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
How to use onnx file with deepstream-test1-usbcam + Custom models DeepStream SDK	30	4633	October 12, 2021
Run BACK-TO-BACK-DETECTORS REFERENCE APP under DeepStream SDK 5.0 DeepStream SDK	16	993	October 12, 2021
Batch-size with 9 rtsp streams DeepStream SDK hw , cuda , gstreamer	16	1677	October 12, 2021
Issues with running inference on multiple rtsp streams in deepstream-imagedata-multistream DeepStream SDK jetson-inference	24	605	August 7, 2024
Reshaping error when set batch-size greater than 1 in onnx modle DeepStream SDK	23	1226	February 10, 2023
Parsing custom tensorflow model DeepStream SDK	31	565	September 4, 2023
NvDsInfer Error: NVDSINFER_CONFIG_FAILED for FasterRCNN DeepStream SDK	13	624	November 22, 2023
How to run deepstream-app without building engine? DeepStream SDK	6	462	March 11, 2024
Deepstream-gaze-app segmentation fault DeepStream SDK	10	255	February 13, 2024
Failed to used TensorRT Engine file in deepstream DeepStream SDK	16	2728	October 12, 2021

Using trtexec to generate an engine file from an ONNX works error with two RTSP input source

Environment

Relevant Files

command(I try all of this)

Error Info

command

error

wish

1. Install dependencies

2. Modify the batch size to 2

3. Create TensorRT engine

4. Test

Related topics