Facing error while trying to convert onnx model using TensorRT Optimisation tool

[alksainath.medam]

Please provide the following info (check/uncheck the boxes after creating this topic):
Software Version
DRIVE OS Linux 5.2.6
DRIVE OS Linux 5.2.6 and DriveWorks 4.0
DRIVE OS Linux 5.2.0
DRIVE OS Linux 5.2.0 and DriveWorks 3.5
NVIDIA DRIVE™ Software 10.0 (Linux)
NVIDIA DRIVE™ Software 9.0 (Linux)
other DRIVE OS version
other

Target Operating System
Linux
QNX
other

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)
NVIDIA DRIVE™ AGX Pegasus DevKit (E3550)
other

SDK Manager Version
1.9.1.10844
other

Host Machine Version
native Ubuntu 18.04
other
I had keras Tensorflow model, converted to onnx model using Tf2onnx package.
Then with this converted onnx model trying to optimize using TensorRT optimization tool.
Getting following error
Network has dynamic or shape inputs, but no optimization profile has been defined
Attaching the required files(Onnx model, error screenshot, error log file, Tensorflow keras model).
Mar3_3.onnx (1.9 MB)

log.txt (26.1 KB)
DMS.h5 (5.7 MB)

[SivaRamaKrishnaNV]

Dear @alksainath.medam,
Before trying out tensort optimization tool, I would recommend to test your model using trtexec tool.

[alksainath.medam]

I tested the model using trtexec tool.
Getting the following error.
Dynamic dimensions required for input conv2d_input
Attaching the error screenshot and log file.

log.txt (26.1 KB)

[SivaRamaKrishnaNV]

Dear @alksainath.medam,
Could you check with --minShapes=conv2d_input:1x145x145x3 --optShapes=conv2d_input:16x145x145x3 --maxShapes=conv2d_input:256x256x256x3 --shapes=conv2d_input:5x145x145x3 instead of --minShapes=conv2d_input:0:1x145x145x3 --optShapes=conv2d_input:0:16x145x145x3 --maxShapes=conv2d_input:0:256x256x256x3 --shapes=conv2d_input:0:5x145x145x3

[alksainath.medam]

I had tried as u suggested, getting “Cuda failure: out of memory”
I had shared my model previously and sharing again.
Can, you try from your side and share the optimized model along with process/steps.
Sharing the log file and screenshot along with model.

log.txt (94.3 KB)

Mar3_3.onnx (1.9 MB)

[alksainath.medam]

I had tried, as u suggested getting “Cuda failure: out of memory”
After trying in high-end system got following error.
“Required optimization profile is invalid”
Attaching the screenshots.

[VickNV]

Could you please provide the log of running mar3_3.onnx instead of DMS.onnx?

[alksainath.medam]

Both are same models I just changed the name.

[SivaRamaKrishnaNV]

Dear @alksainath.medam,
The model looks bigger. May I know on which GPU you have tested? Could you attach complete log which has required optimization profile invalid. Also please use text messages instead of images to be able to search by others in community.

[alksainath.medam]

Thanks for you suggestion @SivaRamaKrishnaNV
I am sharing the log, verbose and GPU details which I am tested.

matlab@matlab:/usr/src/tensorrt/bin$ ./trtexec --onnx=/home/matlab/Sainath/mar3_3.onnx --explicitBatch --minShapes=conv2d_input:1x145x145x3 --optShapes=convd_input:16x145x145x3 --maxShapes=conv2d_input:145x145x145x3 --shapes=conv2d_input:5x145x145x3
&&&& RUNNING TensorRT.trtexec # ./trtexec --onnx=/home/matlab/Sainath/mar3_3.onnx --explicitBatch --minShapes=conv2d_input:1x145x145x3 --optShapes=convd_input:16x145x145x3 --maxShapes=conv2d_input:145x145x145x3 --shapes=conv2d_input:5x145x145x3
[03/24/2023-13:33:02] [I] === Model Options ===
[03/24/2023-13:33:02] [I] Format: ONNX
[03/24/2023-13:33:02] [I] Model: /home/matlab/Sainath/mar3_3.onnx
[03/24/2023-13:33:02] [I] Output:
[03/24/2023-13:33:02] [I] === Build Options ===
[03/24/2023-13:33:02] [I] Max batch: explicit
[03/24/2023-13:33:02] [I] Workspace: 16 MB
[03/24/2023-13:33:02] [I] minTiming: 1
[03/24/2023-13:33:02] [I] avgTiming: 8
[03/24/2023-13:33:02] [I] Precision: FP32
[03/24/2023-13:33:02] [I] Calibration:
[03/24/2023-13:33:02] [I] Safe mode: Disabled
[03/24/2023-13:33:02] [I] Save engine:
[03/24/2023-13:33:02] [I] Load engine:
[03/24/2023-13:33:02] [I] Inputs format: fp32:CHW
[03/24/2023-13:33:02] [I] Outputs format: fp32:CHW
[03/24/2023-13:33:02] [I] Input build shape: convd_input=
[03/24/2023-13:33:02] [I] Input build shape: conv2d_input=1x145x145x3
[03/24/2023-13:33:02] [I] === System Options ===
[03/24/2023-13:33:02] [I] Device: 0
[03/24/2023-13:33:02] [I] DLACore:
[03/24/2023-13:33:02] [I] Plugins:
[03/24/2023-13:33:02] [I] === Inference Options ===
[03/24/2023-13:33:02] [I] Batch: Explicit
[03/24/2023-13:33:02] [I] Input inference shape: conv2d_input=5x145x145x3
[03/24/2023-13:33:02] [I] Iterations: 10 (200 ms warm up)
[03/24/2023-13:33:02] [I] Duration: 3s
[03/24/2023-13:33:02] [I] Sleep time: 0ms
[03/24/2023-13:33:02] [I] Streams: 1
[03/24/2023-13:33:02] [I] Spin-wait: Disabled
[03/24/2023-13:33:02] [I] Multithreading: Enabled
[03/24/2023-13:33:02] [I] CUDA Graph: Disabled
[03/24/2023-13:33:02] [I] Skip inference: Disabled
[03/24/2023-13:33:02] [I] Separate profiling: Disabled
[03/24/2023-13:33:02] [I] Consistency: Disabled
[03/24/2023-13:33:02] [I] === Reporting Options ===
[03/24/2023-13:33:02] [I] Verbose: Disabled
[03/24/2023-13:33:02] [I] Averages: 10 inferences
[03/24/2023-13:33:02] [I] Percentile: 99
[03/24/2023-13:33:02] [I] Dump output: Disabled
[03/24/2023-13:33:02] [I] Profile: Disabled
[03/24/2023-13:33:02] [I] Export timing to JSON file:
[03/24/2023-13:33:02] [I] Export profile to JSON file:
[03/24/2023-13:33:02] [I]
[03/24/2023-13:33:02] [I]
----------------------------------------------------------------
Input filename:   /home/matlab/Sainath/mar3_3.onnx
ONNX IR version:  0.0.7
Opset version:    13
Producer name:    tf2onnx
Producer version: 1.13.0 2c1db5
Domain:
Model version:    0
Doc string:

---

[03/24/2023-13:33:02] [W] [TRT] onnx2trt_utils.cpp:194: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[03/24/2023-13:33:02] [W] [TRT] onnx2trt_utils.cpp:194: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[03/24/2023-13:33:02] [W] [TRT] onnx2trt_utils.cpp:194: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[03/24/2023-13:33:02] [E] [TRT] Parameter check failed at: optimizationProfile.cpp::setDimensions::129, condition: validate(newEntry, true)
[03/24/2023-13:33:02] [E] Required optimization profile is invalid
terminate called after throwing an instance of ‘std::runtime_error’
what(): Failed to create object
Aborted (core dumped)
GPU.txt (966 Bytes)
log.txt (3.8 KB)
Verbose.txt (26.1 KB)

[alksainath.medam]

I am waiting for your reply any updates related to this.
@VickNV @SivaRamaKrishnaNV.