TensorRT conversion from tensorflow with custom op

mssaeed · August 3, 2023, 4:43pm

Hi, I am trying to convert my tensorflow model with a custom op to a tensorRT model. I use the trtexec commandline tool to do so. I first convert the model from tensorflow to ONNX with tf2onnx which has my custom op that I included by passing the .so file (custom op library file) to tf2onnx. Later when I try to convert it to a TRT model I am unable to do so. I use the -plugin command line option with trt exec to which I pass the .so file.
This the error that I am currently running into. It is unable to even read the .so file. It says its not at the location mentioned, even though it is there.
Am I doing something wrong here or is there anyother way where I can convert my TF model with custom op in .so file to TRT?
Here is the command I am running and below is the error log.

trtexec --onnx=model_on.onnx --plugins=…/image_patch.so --saveEngine=engine.trt

[08/03/2023-16:41:57] [I] TensorRT version: 8.6.1
[08/03/2023-16:41:57] [I] Loading standard plugins
[08/03/2023-16:41:57] [I] Loading supplied plugin library: …/image_patch.so
[08/03/2023-16:41:57] [E] Uncaught exception detected: Unable to open library: …/image_patch.so due to libtensorflow_framework.so.2: cannot open shared object file: No such file or directory
&&&& FAILED TensorRT.trtexec [TensorRT v8601] # trtexec --onnx=model_on.onnx --plugins=…/image_patch.so --saveEngine=engine.trt

AakankshaS · August 7, 2023, 3:37am

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

mssaeed · August 7, 2023, 5:04pm

Onnx model checker raises no flags and the model graph is also being produced right. I also verified the onnx model with netron.app.
The above documentation that you provided does not help either as it has no documentation on how to deal with cases of custom ops developed in C++.

I am using the trtexec to convert the model so there is no other script. Attached are the .so file that i used to convert TF to onnx and the converted onnx model.

model_on.onnx

Below is the --verbose log:

root@73d5fa6fa06f:/workspace/tflite_learning# trtexec --onnx=model_on.onnx --staticPlugins=image_patch.so --saveEngine=engine.trt --verbose
&&&& RUNNING TensorRT.trtexec [TensorRT v8601] # trtexec --onnx=model_on.onnx --staticPlugins=image_patch.so --saveEngine=engine.trt --verbose
[08/07/2023-17:00:22] [I] === Model Options ===
[08/07/2023-17:00:22] [I] Format: ONNX
[08/07/2023-17:00:22] [I] Model: model_on.onnx
[08/07/2023-17:00:22] [I] Output:
[08/07/2023-17:00:22] [I] === Build Options ===
[08/07/2023-17:00:22] [I] Max batch: explicit batch
[08/07/2023-17:00:22] [I] Memory Pools: workspace: default, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default
[08/07/2023-17:00:22] [I] minTiming: 1
[08/07/2023-17:00:22] [I] avgTiming: 8
[08/07/2023-17:00:22] [I] Precision: FP32
[08/07/2023-17:00:22] [I] LayerPrecisions:
[08/07/2023-17:00:22] [I] Layer Device Types:
[08/07/2023-17:00:22] [I] Calibration:
[08/07/2023-17:00:22] [I] Refit: Disabled
[08/07/2023-17:00:22] [I] Version Compatible: Disabled
[08/07/2023-17:00:22] [I] TensorRT runtime: full
[08/07/2023-17:00:22] [I] Lean DLL Path:
[08/07/2023-17:00:22] [I] Tempfile Controls: { in_memory: allow, temporary: allow }
[08/07/2023-17:00:22] [I] Exclude Lean Runtime: Disabled
[08/07/2023-17:00:22] [I] Sparsity: Disabled
[08/07/2023-17:00:22] [I] Safe mode: Disabled
[08/07/2023-17:00:22] [I] Build DLA standalone loadable: Disabled
[08/07/2023-17:00:22] [I] Allow GPU fallback for DLA: Disabled
[08/07/2023-17:00:22] [I] DirectIO mode: Disabled
[08/07/2023-17:00:22] [I] Restricted mode: Disabled
[08/07/2023-17:00:22] [I] Skip inference: Disabled
[08/07/2023-17:00:22] [I] Save engine: engine.trt
[08/07/2023-17:00:22] [I] Load engine:
[08/07/2023-17:00:22] [I] Profiling verbosity: 0
[08/07/2023-17:00:22] [I] Tactic sources: Using default tactic sources
[08/07/2023-17:00:22] [I] timingCacheMode: local
[08/07/2023-17:00:22] [I] timingCacheFile:
[08/07/2023-17:00:22] [I] Heuristic: Disabled
[08/07/2023-17:00:22] [I] Preview Features: Use default preview flags.
[08/07/2023-17:00:22] [I] MaxAuxStreams: -1
[08/07/2023-17:00:22] [I] BuilderOptimizationLevel: -1
[08/07/2023-17:00:22] [I] Input(s)s format: fp32:CHW
[08/07/2023-17:00:22] [I] Output(s)s format: fp32:CHW
[08/07/2023-17:00:22] [I] Input build shapes: model
[08/07/2023-17:00:22] [I] Input calibration shapes: model
[08/07/2023-17:00:22] [I] === System Options ===
[08/07/2023-17:00:22] [I] Device: 0
[08/07/2023-17:00:22] [I] DLACore:
[08/07/2023-17:00:22] [I] Plugins: image_patch.so
[08/07/2023-17:00:22] [I] setPluginsToSerialize:
[08/07/2023-17:00:22] [I] dynamicPlugins:
[08/07/2023-17:00:22] [I] ignoreParsedPluginLibs: 0
[08/07/2023-17:00:22] [I]
[08/07/2023-17:00:22] [I] === Inference Options ===
[08/07/2023-17:00:22] [I] Batch: Explicit
[08/07/2023-17:00:22] [I] Input inference shapes: model
[08/07/2023-17:00:22] [I] Iterations: 10
[08/07/2023-17:00:22] [I] Duration: 3s (+ 200ms warm up)
[08/07/2023-17:00:22] [I] Sleep time: 0ms
[08/07/2023-17:00:22] [I] Idle time: 0ms
[08/07/2023-17:00:22] [I] Inference Streams: 1
[08/07/2023-17:00:22] [I] ExposeDMA: Disabled
[08/07/2023-17:00:22] [I] Data transfers: Enabled
[08/07/2023-17:00:22] [I] Spin-wait: Disabled
[08/07/2023-17:00:22] [I] Multithreading: Disabled
[08/07/2023-17:00:22] [I] CUDA Graph: Disabled
[08/07/2023-17:00:22] [I] Separate profiling: Disabled
[08/07/2023-17:00:22] [I] Time Deserialize: Disabled
[08/07/2023-17:00:22] [I] Time Refit: Disabled
[08/07/2023-17:00:22] [I] NVTX verbosity: 0
[08/07/2023-17:00:22] [I] Persistent Cache Ratio: 0
[08/07/2023-17:00:22] [I] Inputs:
[08/07/2023-17:00:22] [I] === Reporting Options ===
[08/07/2023-17:00:22] [I] Verbose: Enabled
[08/07/2023-17:00:22] [I] Averages: 10 inferences
[08/07/2023-17:00:22] [I] Percentiles: 90,95,99
[08/07/2023-17:00:22] [I] Dump refittable layers:Disabled
[08/07/2023-17:00:22] [I] Dump output: Disabled
[08/07/2023-17:00:22] [I] Profile: Disabled
[08/07/2023-17:00:22] [I] Export timing to JSON file:
[08/07/2023-17:00:22] [I] Export output to JSON file:
[08/07/2023-17:00:22] [I] Export profile to JSON file:
[08/07/2023-17:00:22] [I]
[08/07/2023-17:00:22] [I] === Device Information ===
[08/07/2023-17:00:22] [I] Selected Device: NVIDIA GeForce RTX 2070
[08/07/2023-17:00:22] [I] Compute Capability: 7.5
[08/07/2023-17:00:22] [I] SMs: 36
[08/07/2023-17:00:22] [I] Device Global Memory: 8191 MiB
[08/07/2023-17:00:22] [I] Shared Memory per SM: 64 KiB
[08/07/2023-17:00:22] [I] Memory Bus Width: 256 bits (ECC disabled)
[08/07/2023-17:00:22] [I] Application Compute Clock Rate: 1.455 GHz
[08/07/2023-17:00:22] [I] Application Memory Clock Rate: 5.501 GHz
[08/07/2023-17:00:22] [I]
[08/07/2023-17:00:22] [I] Note: The application clock rates do not reflect the actual clock rates that the GPU is currently running at.
[08/07/2023-17:00:22] [I]
[08/07/2023-17:00:22] [I] TensorRT version: 8.6.1
[08/07/2023-17:00:22] [I] Loading standard plugins
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::BatchedNMS_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::BatchTilePlugin_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::Clip_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::CoordConvAC version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::CropAndResizeDynamic version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::CropAndResize version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::DecodeBbox3DPlugin version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::DetectionLayer_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::EfficientNMS_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::FlattenConcat_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::GenerateDetection_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::GridAnchor_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::GridAnchorRect_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 2
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::LReLU_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::ModulatedDeformConv2d version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::MultilevelProposeROI_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::MultiscaleDeformableAttnPlugin_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::NMSDynamic_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::NMS_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::Normalize_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::PillarScatterPlugin version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::PriorBox_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::ProposalDynamic version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::ProposalLayer_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::Proposal version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::PyramidROIAlign_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::Region_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::Reorg_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::ResizeNearest_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::ROIAlign_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::RPROI_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::ScatterND version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::SpecialSlice_TRT version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::Split version 1
[08/07/2023-17:00:22] [V] [TRT] Registered plugin creator - ::VoxelGeneratorPlugin version 1
[08/07/2023-17:00:22] [I] Loading supplied plugin library: image_patch.so
[08/07/2023-17:00:22] [E] Uncaught exception detected: Unable to open library: image_patch.so due to image_patch.so: cannot open shared object file: No such file or directory
&&&& FAILED TensorRT.trtexec [TensorRT v8601] # trtexec --onnx=model_on.onnx --staticPlugins=image_patch.so --saveEngine=engine.trt --verbose

image_patch.so (54.6 KB)

spolisetty · August 11, 2023, 10:25am

Hi,

Please locate the file libtensorflow_framework.so.2 and copy it to your folder, then export the path to the LD_LIBRARY_PATH.

export LD_LIBRARY_PATH=`pwd`:$LD_LIBRARY_PATH

Example path of libtensorflow_framework.so.2

/usr/local/lib/python3.10/dist-packages/tensorflow/libtensorflow_framework.so.2

Thank you.

mssaeed · August 12, 2023, 1:06am

Thank you. It is now able to read my library.
On the other hand it still does not recognize my OP. Is there a way around using a custom op written for tensorflow and using it for tensorRT, like onnx allows you to do it. Or is IPluginV2DynamicExt the only way to do so? Where I have to create new classes for both the plugin and plugin creator?
Attached is the log of my latest execution of trtexec.
Thank you.
logs.txt (43.8 KB)

system · August 26, 2023, 1:07am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
TensorRT does not see all GPU memory TensorRT	1	1004	November 18, 2022
BUG: Output TRT engine from trtexec has completely different inference than input model TensorRT tensorrt , debugging-and-troubleshooting	3	2228	January 4, 2022
Floor - Cast - Resize(or Slice) cause internal error TensorRT tensorrt	7	2020	January 12, 2022
Problem converting TensorFlow 2-> ONNX model to TensorRT Engine (efficientdet_d0) TensorRT	8	1398	November 17, 2022
Trtexec cannot convert QAT onnx model to trt model TensorRT tensorrt	7	1174	January 26, 2023
Duplicated reshapes triggers "[graphOptimizer.cpp::findOne::510] Error Code 2: Internal Error (Assertion it != v.end() failed. )" TensorRT tensorrt , onnx	4	920	February 22, 2022
TensorRT OSS build TensorRT	9	1101	August 31, 2023
[TensorRT] ERROR: Network must have at least one output [TensorRT] ERROR: Network validation failed TensorRT tensorrt , cuda , onnx	10	2775	October 16, 2020
How to export the Pytorch model Keypoint R-CNN to onnx and benchmark with trtexec TensorRT	7	1340	July 14, 2022
Unused input error building TensorRT engine TensorRT tensorrt , tensorflow	6	1378	August 31, 2021

TensorRT conversion from tensorflow with custom op

check_model.py

Related topics