I like you understand my issue.
So let me show again the issue from the beginning in this reply.
I have this tensorflow model. Dropbox - File Deleted
The model is working well upto Mconv7_stage6_L2 and Mconv7_stage6_L1.
The layers after Mconv7_stage6_L2 and Mconv7_stage6_L1 are added for some functions and made a new model.
When I converted the new model to TensorRT engine, I have messages as
Warning: No conversion function registered for layer: ResizeArea yet.
Converting upsample_heatmat as custom op: ResizeArea
Warning: No conversion function registered for layer: Select yet.
Converting Select as custom op: Select
Warning: No conversion function registered for layer: Fill yet.
Converting zeros_like as custom op: Fill
Warning: No conversion function registered for layer: Equal yet.
Converting Equal as custom op: Equal
No. nodes: 475
UFF Output written to /home/coie/Data/coie/Softwares/HumanActivity_MSF/Multithreading/CreateEnginefromNetwork/CMUNet/cmu/frozen_model.uff
UFF Text Output written to /home/coie/Data/coie/Softwares/HumanActivity_MSF/Multithreading/CreateEngi
What I understood from these warnings is
Tensorflow operations ResizeArea, Select, Fill, and Equal have no equivalent TensorRT layers.
So I need to make custom plugin for those layers.
I made custom layers following the tutorials and examples here https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#extending.
After creating all custom plugins, I prepared all necessary steps as shown in examples as follow.
RESIZEAREA_PLUGIN_LIBRARY = os.path.join(
WORKING_DIR,
'plugins/ResizeArea/build/libResizeAreaplugin.so'
)
SELECT_PLUGIN_LIBRARY = os.path.join(
WORKING_DIR,
'plugins/Select/build/libSelectplugin.so'
)
FILL_PLUGIN_LIBRARY = os.path.join(
WORKING_DIR,
'plugins/Fill/build/libFillplugin.so'
)
EQUAL_PLUGIN_LIBRARY = os.path.join(
WORKING_DIR,
'plugins/Equal/build/libEqualplugin.so'
)
ENGINE_PATH = os.path.join(
WORKING_DIR,
'engine/cmu_32.engine'
)
MODEL_PATH = os.path.join(
WORKING_DIR,
'frozen_model.pb'
)
class ModelData(object):
INPUT_NAME = "image"
EQUAL_NAME = "Equal"
SELECT_NAME = "Select"
PMAT_NAME = "upsample_pafmat"
ZERO_LIKE = "zeros_like"
HMAT_NAME = "upsample_heatmat"
OUTPUT_NAME = "Openpose/output"
#engine_path='mobilenet/mobilenet_32.engine'
#model_path='mobilenet/mobilenet_openpose.uff'
width=640
height=480
def get_sec(time_str):
m, s = time_str.split(':')
return int(m) * 60 + int(s)
# Generates mappings from unsupported TensorFlow operations to TensorRT plugins
def prepare_namespace_plugin_map():
# In this sample, the only operation that is not supported by TensorRT
# is tf.nn.relu6, so we create a new node which will tell UffParser which
# plugin to run and with which arguments in place of tf.nn.relu6.
# The "clipMin" and "clipMax" fields of this TensorFlow node will be parsed by createPlugin,
# and used to create a CustomClipPlugin with the appropriate parameters.
trt_resizearea = gs.create_plugin_node(name="trt_resizearea", op="ResizeAreaPlugin", in_width=80.0, in_height=60.0, in_channel=3.0, upscale=4.0)
trt_fill = gs.create_plugin_node(name="trt_fill", op="FillPlugin", in_width=320.0, in_height=240.0, in_channel=3.0, value=0.0)#fill 0
trt_equal = gs.create_plugin_node(name="trt_equal", op="EqualPlugin", in_width=320.0, in_height=240.0, in_channel=3.0)
trt_select = gs.create_plugin_node(name="trt_select", op="SelectPlugin", in_width=320.0, in_height=240.0, value=0.0)
namespace_plugin_map = {
ModelData.SELECT_NAME: trt_select,
ModelData.EQUAL_NAME: trt_equal,
ModelData.PMAT_NAME: trt_resizearea,
ModelData.HMAT_NAME: trt_resizearea,
ModelData.ZERO_LIKE: trt_fill
}
return namespace_plugin_map
def model_to_uff(model_path):
# Transform graph using graphsurgeon to map unsupported TensorFlow
# operations to appropriate TensorRT custom layer plugins
dynamic_graph = gs.DynamicGraph(model_path)
dynamic_graph.collapse_namespaces(prepare_namespace_plugin_map())
# Save resulting graph to UFF file
output_uff_path = model_path_to_uff_path(model_path)
uff.from_tensorflow(
dynamic_graph.as_graph_def(),
[ModelData.OUTPUT_NAME],
output_filename=output_uff_path,
text=True
)
return output_uff_path
def model_path_to_uff_path(model_path):
uff_path = os.path.splitext(model_path)[0] + ".uff"
return uff_path
When I convert again, I still have warnings and have the following error
CHECK failed: (index) < (current_size_):
Warning: No conversion function registered for layer: SelectPlugin yet.
Converting trt_select as custom op: SelectPlugin
Warning: No conversion function registered for layer: FillPlugin yet.
Converting trt_fill as custom op: FillPlugin
Warning: No conversion function registered for layer: EqualPlugin yet.
Converting trt_equal as custom op: EqualPlugin
No. nodes: 470
UFF Output written to /home/coie/Data/coie/Softwares/HumanActivity_MSF/Multithreading/CreateEnginefromNetwork/CMUNet/cmu/frozen_model.uff
UFF Text Output written to /home/coie/Data/coie/Softwares/HumanActivity_MSF/Multithreading/CreateEnginefromNetwork/CMUNet/cmu/frozen_model.pbtxt
[libprotobuf FATAL /home/erisuser/p4sw/sw/gpgpu/MachineLearning/DIT/externals/protobuf/x86_64/10.0/include/google/protobuf/repeated_field.h:1408] CHECK failed: (index) < (current_size_):
Traceback (most recent call last):
File "HumanPoseTRT.py", line 303, in <module>
main()
File "HumanPoseTRT.py", line 256, in main
serializeandsave_engine(MODEL_PATH)
File "HumanPoseTRT.py", line 124, in serializeandsave_engine
parser.parse(uff_path, network)
RuntimeError: CHECK failed: (index) < (current_size_):
What is wrong with my approach?