Keras->Onnx->TensorRT

Autonomous Machines Jetson & Embedded Systems Jetson AGX Orin
1

Description

I trained a Keras model on the PC side and wanted to transplant the model to orin across platforms, but found incompatibility in the conversion process. The specific process is as follows:

Environment

jtop 4.2.10 - (c) 2024, Raffaello Bonghi [raffaello@rnext.it]
Website: jetson-stats 4.2.11

Platform Serial Number: [s|XX CLICK TO READ XXX]
Machine: aarch64 Hardware
System: Linux Model: Jetson AGX Orin Developer Kit
Distribution: Ubuntu 20.04 Focal Fossa 699-level Part Number: 699-13701-0005-500 M.0
Release: 5.10.216-tegra P-Number: p3701-0005
Python: 3.8.10 Module: NVIDIA Jetson AGX Orin (64GB ram)
SoC: tegra23x
Libraries CUDA Arch BIN: 8.7
CUDA: 11.4.315 Codename: Concord
cuDNN: 8.6.0.166 L4T: 35.6.0
TensorRT: 8.5.2.2 Jetpack: MISSING
VPI: 2.4.8
Vulkan: 1.3.204 Hostname: ubuntu
OpenCV: 4.5.4 with CUDA: NO Interfaces
eth0: 192.168.0.102
docker0: 172.17.0.1

Relevant Files

通过网盘分享的文件：Low light model demo
链接: https://pan.baidu.com/s/1R4tfrGqfYmBhA35L8D4oXg?pwd=p9i2 提取码: p9i2
–来自百度网盘超级会员v5的分享

Steps To Reproduce

1.h5 ->.pb

#!/usr/bin/env python
import tensorflow as tf


# 定义自定义损失函数
def charbonnier_loss(y_true, y_pred):
    epsilon = 1e-6  # 防止除零错误
    return tf.reduce_mean(tf.sqrt(tf.square(y_true - y_pred) + epsilon))


# 定义自定义指标
def peak_signal_noise_ratio(y_true, y_pred):
    return tf.image.psnr(y_true, y_pred, max_val=1.0)


# 使用 custom_object_scope 加载模型
with tf.keras.utils.custom_object_scope({
    'charbonnier_loss': charbonnier_loss,
    'peak_signal_noise_ratio': peak_signal_noise_ratio
}):
    model = tf.keras.models.load_model('model.h5')


# 将模型保存为 TensorFlow SavedModel 格式
model.save('saved_model_directory', save_format='tf')

2.pb ->onnx

python3.8 -m tf2onnx.convert --saved-model saved_model_directory --output model.onnx

3.onnx → plan

trtexec --onnx=model.onnx --saveEngine=model.plan --fp16

&&&& RUNNING TensorRT.trtexec [TensorRT v8502] # trtexec --onnx=model.onnx --saveEngine=model.plan
[09/23/2024-02:05:40] [I] === Model Options ===
[09/23/2024-02:05:40] [I] Format: ONNX
[09/23/2024-02:05:40] [I] Model: model.onnx
[09/23/2024-02:05:40] [I] Output:
[09/23/2024-02:05:40] [I] === Build Options ===
[09/23/2024-02:05:40] [I] Max batch: explicit batch
[09/23/2024-02:05:40] [I] Memory Pools: workspace: default, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default
[09/23/2024-02:05:40] [I] minTiming: 1
[09/23/2024-02:05:40] [I] avgTiming: 8
[09/23/2024-02:05:40] [I] Precision: FP32
[09/23/2024-02:05:40] [I] LayerPrecisions:
[09/23/2024-02:05:40] [I] Calibration:
[09/23/2024-02:05:40] [I] Refit: Disabled
[09/23/2024-02:05:40] [I] Sparsity: Disabled
[09/23/2024-02:05:40] [I] Safe mode: Disabled
[09/23/2024-02:05:40] [I] DirectIO mode: Disabled
[09/23/2024-02:05:40] [I] Restricted mode: Disabled
[09/23/2024-02:05:40] [I] Build only: Disabled
[09/23/2024-02:05:40] [I] Save engine: model.plan
[09/23/2024-02:05:40] [I] Load engine:
[09/23/2024-02:05:40] [I] Profiling verbosity: 0
[09/23/2024-02:05:40] [I] Tactic sources: Using default tactic sources
[09/23/2024-02:05:40] [I] timingCacheMode: local
[09/23/2024-02:05:40] [I] timingCacheFile:
[09/23/2024-02:05:40] [I] Heuristic: Disabled
[09/23/2024-02:05:40] [I] Preview Features: Use default preview flags.
[09/23/2024-02:05:40] [I] Input(s)s format: fp32:CHW
[09/23/2024-02:05:40] [I] Output(s)s format: fp32:CHW
[09/23/2024-02:05:40] [I] Input build shapes: model
[09/23/2024-02:05:40] [I] Input calibration shapes: model
[09/23/2024-02:05:40] [I] === System Options ===
[09/23/2024-02:05:40] [I] Device: 0
[09/23/2024-02:05:40] [I] DLACore:
[09/23/2024-02:05:40] [I] Plugins:
[09/23/2024-02:05:40] [I] === Inference Options ===
[09/23/2024-02:05:40] [I] Batch: Explicit
[09/23/2024-02:05:40] [I] Input inference shapes: model
[09/23/2024-02:05:40] [I] Iterations: 10
[09/23/2024-02:05:40] [I] Duration: 3s (+ 200ms warm up)
[09/23/2024-02:05:40] [I] Sleep time: 0ms
[09/23/2024-02:05:40] [I] Idle time: 0ms
[09/23/2024-02:05:40] [I] Streams: 1
[09/23/2024-02:05:40] [I] ExposeDMA: Disabled
[09/23/2024-02:05:40] [I] Data transfers: Enabled
[09/23/2024-02:05:40] [I] Spin-wait: Disabled
[09/23/2024-02:05:40] [I] Multithreading: Disabled
[09/23/2024-02:05:40] [I] CUDA Graph: Disabled
[09/23/2024-02:05:40] [I] Separate profiling: Disabled
[09/23/2024-02:05:40] [I] Time Deserialize: Disabled
[09/23/2024-02:05:40] [I] Time Refit: Disabled
[09/23/2024-02:05:40] [I] NVTX verbosity: 0
[09/23/2024-02:05:40] [I] Persistent Cache Ratio: 0
[09/23/2024-02:05:40] [I] Inputs:
[09/23/2024-02:05:40] [I] === Reporting Options ===
[09/23/2024-02:05:40] [I] Verbose: Disabled
[09/23/2024-02:05:40] [I] Averages: 10 inferences
[09/23/2024-02:05:40] [I] Percentiles: 90,95,99
[09/23/2024-02:05:40] [I] Dump refittable layers:Disabled
[09/23/2024-02:05:40] [I] Dump output: Disabled
[09/23/2024-02:05:40] [I] Profile: Disabled
[09/23/2024-02:05:40] [I] Export timing to JSON file:
[09/23/2024-02:05:40] [I] Export output to JSON file:
[09/23/2024-02:05:40] [I] Export profile to JSON file:
[09/23/2024-02:05:40] [I]
[09/23/2024-02:05:40] [I] === Device Information ===
[09/23/2024-02:05:40] [I] Selected Device: Orin
[09/23/2024-02:05:40] [I] Compute Capability: 8.7
[09/23/2024-02:05:40] [I] SMs: 8
[09/23/2024-02:05:40] [I] Compute Clock Rate: 1.3 GHz
[09/23/2024-02:05:40] [I] Device Global Memory: 62780 MiB
[09/23/2024-02:05:40] [I] Shared Memory per SM: 164 KiB
[09/23/2024-02:05:40] [I] Memory Bus Width: 256 bits (ECC disabled)
[09/23/2024-02:05:40] [I] Memory Clock Rate: 0.612 GHz
[09/23/2024-02:05:40] [I]
[09/23/2024-02:05:40] [I] TensorRT version: 8.5.2
[09/23/2024-02:05:41] [I] [TRT] [MemUsageChange] Init CUDA: CPU +220, GPU +0, now: CPU 249, GPU 6372 (MiB)
[09/23/2024-02:05:44] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +302, GPU +287, now: CPU 574, GPU 6679 (MiB)
[09/23/2024-02:05:44] [I] Start parsing network model
[09/23/2024-02:05:44] [I] [TRT] ----------------------------------------------------------------
[09/23/2024-02:05:44] [I] [TRT] Input filename: model.onnx
[09/23/2024-02:05:44] [I] [TRT] ONNX IR version: 0.0.8
[09/23/2024-02:05:44] [I] [TRT] Opset version: 15
[09/23/2024-02:05:44] [I] [TRT] Producer name: tf2onnx
[09/23/2024-02:05:44] [I] [TRT] Producer version: 1.16.1 15c810
[09/23/2024-02:05:44] [I] [TRT] Domain:
[09/23/2024-02:05:44] [I] [TRT] Model version: 0
[09/23/2024-02:05:44] [I] [TRT] Doc string:
[09/23/2024-02:05:44] [I] [TRT] ----------------------------------------------------------------
[09/23/2024-02:05:44] [W] [TRT] onnx2trt_utils.cpp:375: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[09/23/2024-02:05:45] [E] [TRT] ModelImporter.cpp:726: While parsing node number 213 [Resize → “Resize__845:0”]:
[09/23/2024-02:05:45] [E] [TRT] ModelImporter.cpp:727: — Begin node —
[09/23/2024-02:05:45] [E] [TRT] ModelImporter.cpp:728: input: “StatefulPartitionedCall/model/conv2d_29/Relu:0”
input: “const_empty_float__5411”
input: “const_empty_float__5411”
input: “Concat__844:0”
output: “Resize__845:0”
name: “Resize__845”
op_type: “Resize”
attribute {
name: “coordinate_transformation_mode”
s: “tf_half_pixel_for_nn”
type: STRING
}
attribute {
name: “exclude_outside”
i: 0
type: INT
}
attribute {
name: “mode”
s: “nearest”
type: STRING
}
attribute {
name: “nearest_mode”
s: “floor”
type: STRING
}
domain: “”

[09/23/2024-02:05:45] [E] [TRT] ModelImporter.cpp:729: — End node —
[09/23/2024-02:05:45] [E] [TRT] ModelImporter.cpp:731: ERROR: builtin_op_importers.cpp:3730 In function importResize:
[8] Assertion failed: (transformationMode != “tf_half_pixel_for_nn” || nearest_mode == “round_prefer_floor”) && “This version of TensorRT only support round_prefer_floor nearest mode in tf_half_pixel_for_nn!”
[09/23/2024-02:05:45] [E] Failed to parse onnx file
[09/23/2024-02:05:45] [I] Finish parsing network model
[09/23/2024-02:05:45] [E] Parsing model failed
[09/23/2024-02:05:45] [E] Failed to create engine from model or file.
[09/23/2024-02:05:45] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8502] # trtexec --onnx=model.onnx --saveEngine=model.plan

5

Hi,

Have you tried to run the model with another TensorRT version?
If yes, could you share which version you require?

Thanks.

7

I haven’t tried it on any other TensorRT version. This is a set of h5 models trained by my client directly on their side. I hope we can deploy them on orin platform and display them on the display screen through the GPU acceleration of orin platform. I don’t know exactly what version the client is training on, which is very confusing to me, meaning I need to try it one by one, and I’m not sure if upgrading to jetpack6.0 will help solve this problem.

8

Hi,

Based on the error:

[8] Assertion failed: (transformationMode != “tf_half_pixel_for_nn” || nearest_mode == “round_prefer_floor”) && “This version of TensorRT only support round_prefer_floor nearest mode in tf_half_pixel_for_nn!”

Could you try to update the mode of the resize layer from floor to round_prefer_floor to see if it can work?

Thanks.

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.