Can we get an example of how to use TensorRT for BART model in Pytorch?
TRT does not support some ops in BART so would love to get a path with pytorch to fall back on unsupported ops.
Thanks,
Efrat
Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:
- validating your model with the below snippet
check_model.py
import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec
In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!
hi, i try to convert the facebook/bart_base model into tensortrt.
1)check_model.py is ok
2)trtexec --explicitBatch --onnx=Bart/temp5/Bart/bart-base/Bart-bart-base.onnx --minShapes=input_ids:16x1 --maxShapes=input_ids:16x32 --optShapes=input_ids:16x16 --buildOnly --saveEngine=gpt.trt
core dump when load the onnx file
[01/12/2022-09:32:11] [W] --explicitBatch flag has been deprecated and has no effect!
[01/12/2022-09:32:11] [W] Explicit batch dim is automatically enabled if input model is ONNX or if dynamic shapes are provided when the engine is built.
[01/12/2022-09:32:13] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:364: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[01/12/2022-09:32:15] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[01/12/2022-09:32:15] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[01/12/2022-09:32:15] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[01/12/2022-09:32:15] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[01/12/2022-09:32:18] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[01/12/2022-09:32:18] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[01/12/2022-09:33:05] [W] [TRT] Output type must be INT32 for shape outputs
[01/12/2022-09:33:05] [W] [TRT] Output type must be INT32 for shape outputs
[01/12/2022-09:33:05] [W] [TRT] Output type must be INT32 for shape outputs
[01/12/2022-09:33:05] [W] [TRT] Output type must be INT32 for shape outputs
[01/12/2022-09:33:05] [W] [TRT] Output type must be INT32 for shape outputs
[01/12/2022-09:33:08] [W] [TRT] Myelin graph with multiple dynamic values may have poor performance if they differ. Dynamic values are:
[01/12/2022-09:33:08] [W] [TRT] (# 1 (RESHAPE 16 (# 1 (RESHAPE 16 (* E0 (MIN 1 E0)) | 16 E0 zeroIsPlaceholder)) | 16 E0 zeroIsPlaceholder)) where E0=(+ (# 1 (SHAPE input_ids)) -1)
[01/12/2022-09:33:08] [W] [TRT] (# 1 (SHAPE input_ids))
trt.log (1.1 MB)
onnxruntime 1.8.0
onnx 1.9.0
transformers 4.15.0
torch 1.10.1+cu113
TensorRT version: 8.2.1
Driver Version: 470.82.01 CUDA Version: 11.4
T4