Can not convert onnx model to TensorRT format on the Nvidia AGX Orin dev kit

shahizat · September 24, 2023, 1:39pm

Hello Nvidia community,

I was trying to optimize the Piper text-to-speech models by running on the Nvidia Jetson boards with TensorRT optimization.

Typecast ONNX model parameters from INT64 to INT32 was done. ONNX simplification as well. Opset Version of ONNX Model is 15. The model for different languages can be downloaded from here.

The below issue appears when I run:

 /usr/src/tensorrt/bin/trtexec --onnx=./en_US-libritts-high-sim_int32.onnx  --saveEngine=./model_fp32.engine --workspace=20000

Logs

[09/24/2023-18:16:33] [W] [TRT] onnx2trt_utils.cpp:375: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[09/24/2023-18:16:33] [E] [TRT] ModelImporter.cpp:726: While parsing node number 48 [Pad -> "onnx::Slice_932"]:
[09/24/2023-18:16:33] [E] [TRT] ModelImporter.cpp:727: --- Begin node ---
[09/24/2023-18:16:33] [E] [TRT] ModelImporter.cpp:728: input: "enc_p.encoder.attn_layers.0.emb_rel_k"
input: "onnx::Cast_929"
input: "onnx::Pad_931"
output: "onnx::Slice_932"
name: "Pad_105"
op_type: "Pad"
attribute {
  name: "mode"
  s: "constant"
  type: STRING
}
doc_string: "/home/hansenm/opt/larynx2/src/python/.venv/lib/python3.9/site-packages/torch/nn/functional.py(4364): _pad\n/home/hansenm/opt/larynx2/src/python/piper_train/vits/attentions.py(299): _get_relative_embeddings\n/home/hansenm/opt/larynx2/src/python/piper_train/vits/attentions.py(237): attention\n/home/hansenm/opt/larynx2/src/python/piper_train/vits/attentions.py(220): forward\n/home/hansenm/opt/larynx2/src/python/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py(1098): _slow_forward\n/home/hansenm/opt/larynx2/src/python/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py(1110): _call_impl\n/home/hansenm/opt/larynx2/src/python/piper_train/vits/attentions.py(66): forward\n/home/hansenm/opt/larynx2/src/python/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py(1098): _slow_forward\n/home/hansenm/opt/larynx2/src/python/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py(1110): _call_impl\n/home/hansenm/opt/larynx2/src/python/piper_train/vits/models.py(205): forward\n/home/hansenm/opt/larynx2/src/python/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py(1098): _slow_forward\n/home/hansenm/opt/larynx2/src/python/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py(1110): _call_impl\n/home/hansenm/opt/larynx2/src/python/piper_train/vits/models.py(691): infer\n/home/hansenm/opt/larynx2/src/python/piper_train/export_onnx.py(60): infer_forward\n/home/hansenm/opt/larynx2/src/python/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py(1098): _slow_forward\n/home/hansenm/opt/larynx2/src/python/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py(1110): _call_impl\n/home/hansenm/opt/larynx2/src/python/.venv/lib/python3.9/site-packages/torch/jit/_trace.py(118): wrapper\n/home/hansenm/opt/larynx2/src/python/.venv/lib/python3.9/site-packages/torch/jit/_trace.py(127): forward\n/home/hansenm/opt/larynx2/src/python/.venv/lib/python3.9/site-packages/torch/nn/modules/module.py(1110): _call_impl\n/home/hansenm/opt/larynx2/src/python/.venv/lib/python3.9/site-packages/torch/jit/_trace.py(1166): _get_trace_graph\n/home/hansenm/opt/larynx2/src/python/.venv/lib/python3.9/site-packages/torch/onnx/utils.py(391): _trace_and_get_graph_from_model\n/home/hansenm/opt/larynx2/src/python/.venv/lib/python3.9/site-packages/torch/onnx/utils.py(440): _create_jit_graph\n/home/hansenm/opt/larynx2/src/python/.venv/lib/python3.9/site-packages/torch/onnx/utils.py(499): _model_to_graph\n/home/hansenm/opt/larynx2/src/python/.venv/lib/python3.9/site-packages/torch/onnx/utils.py(719): _export\n/home/hansenm/opt/larynx2/src/python/.venv/lib/python3.9/site-packages/torch/onnx/utils.py(118): export\n/home/hansenm/opt/larynx2/src/python/.venv/lib/python3.9/site-packages/torch/onnx/__init__.py(305): export\n/home/hansenm/opt/larynx2/src/python/piper_train/export_onnx.py(88): main\n/home/hansenm/opt/larynx2/src/python/piper_train/export_onnx.py(109): <module>\n/usr/lib/python3.9/runpy.py(87): _run_code\n/usr/lib/python3.9/runpy.py(197): _run_module_as_main\n"

[09/24/2023-18:16:33] [E] [TRT] ModelImporter.cpp:729: --- End node ---
[09/24/2023-18:16:33] [E] [TRT] ModelImporter.cpp:731: ERROR: builtin_op_importers.cpp:3279 In function importPad:
[8] Assertion failed: onnxPaddingPtr->getDimensions().d[0] == nbDims * 2 && "pads should be a 1D tensor of shape [2 * input_rank]"
[09/24/2023-18:16:33] [E] Failed to parse onnx file
[09/24/2023-18:16:33] [I] Finish parsing network model
[09/24/2023-18:16:33] [E] Parsing model failed
[09/24/2023-18:16:33] [E] Failed to create engine from model or file.
[09/24/2023-18:16:33] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8502] # /usr/src/tensorrt/bin/trtexec --onnx=./en_US-libritts-high-sim_int32.onnx --saveEngine=./model_fp32.engine --workspace=20000

Environment

TensorRT Version: 8.5.2-1+cuda11.4
GPU Type: Nvidia AGX Orin 64GB dev kit
Nvidia Driver Version: N/A
CUDA Version: 11.4
CUDNN Version: 8.6.0.166
Operating System + Version: Jetpack 5.1

I appreciate any insights or suggestions you can provide. I look forward to hearing from the experts on this forum.

Thank you in advance for your help!

Best regards,
Shakhizat

spolisetty · September 25, 2023, 1:27pm

Please upload the ONNX model you are using for better debugging.

Thank you.

shahizat · September 25, 2023, 1:37pm

Hi @spolisetty,

Please find attached.

en_US-ryan-medium.zip (55.7 MB)

shahizat · September 26, 2023, 5:12am

Hi again @spolisetty,

Just additional information.

If you needed, the pytorch checkpoint files are located here: rhasspy/piper-checkpoints at main

The export script to onnx format. https://github.com/rhasspy/piper/blob/master/src/python/piper_train/export_onnx.py

Best regards,
Shakhizat

spolisetty · September 27, 2023, 4:27pm

Hi,

We are facing a different error on trying to build TensorRT engine.

[09/27/2023-16:16:47] [V] [TRT] Registering layer: /Range for ONNX node: /Range
[09/27/2023-16:16:47] [E] Error[4]: [graphShapeAnalyzer.cpp::demandAndResolveInputTensor::1247] Error Code 4: Internal Error (scales: network input that is shape tensor must have type Int32)
[09/27/2023-16:16:47] [E] [TRT] ModelImporter.cpp:771: While parsing node number 2349 [Range -> "/Range_output_0"]:
[09/27/2023-16:16:47] [E] [TRT] ModelImporter.cpp:772: --- Begin node ---
[09/27/2023-16:16:47] [E] [TRT] ModelImporter.cpp:773: input: "/Constant_output_0"
input: "/ReduceMax_output_0"
input: "/Constant_1_output_0"
output: "/Range_output_0"
name: "/Range"
op_type: "Range"

[09/27/2023-16:16:47] [E] [TRT] ModelImporter.cpp:774: --- End node ---
[09/27/2023-16:16:47] [E] [TRT] ModelImporter.cpp:777: ERROR: ModelImporter.cpp:195 In function parseGraph:
[6] Invalid Node - /Range
[graphShapeAnalyzer.cpp::demandAndResolveInputTensor::1247] Error Code 4: Internal Error (scales: network input that is shape tensor must have type Int32)
[09/27/2023-16:16:47] [E] Failed to parse onnx file
[09/27/2023-16:16:47] [I] Finished parsing network model. Parse time: 2.38166
[09/27/2023-16:16:47] [E] Parsing model failed
[09/27/2023-16:16:47] [E] Failed to create engine from model or file.
[09/27/2023-16:16:47] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8601] # trtexec --onnx=en_US-ryan-medium.onnx --verbose

Is above the right ONNX model ?

Thank you.

shahizat · September 28, 2023, 3:25pm

Hello @spolisetty , thank you for your reply. I couldn’t upload the specific model file because it’s larger than 100MB. You can access the models directly from here: rhasspy/piper-voices at v1.0.0.