I have been trying to convert a custom tensorflow model to tensor RT, using TF-TRT workflow.
The conversion prouduces a usable graph but it looks like no optimization was performed. When trying to serialize to the .plan file, no operation is found as “TRTEngineOp”, all the nodes are excluded and no .plan file is generated.
The architecture is mostly composed of basic convolutional blocks (convolution, pooling, non-linearity), plus some bilinear resize.
I attach an extract from the output to give an idea of the type of node that are excluded:
Exclude Node: BiasAdd, architecture_layer1_nin_block_nin_0_BiasAdd Exclude Node: Mul, architecture_layer1_nin_block_nin_0_mul Exclude Node: Maximum, architecture_layer1_nin_block_nin_0_leaky_relu Exclude Node: Conv2D, architecture_layer1_nin_block_nin_1_Conv2D Exclude Node: BiasAdd, architecture_layer1_nin_block_nin_1_BiasAdd Exclude Node: Mul, architecture_layer1_nin_block_nin_1_mul Exclude Node: Maximum, architecture_layer1_nin_block_nin_1_leaky_relu Exclude Node: MaxPool, architecture_layer1_pooling Exclude Node: ResizeBilinear, architecture_layer1_resize_regL
TensorRT Version: 188.8.131.52
GPU Type: RTX2080ti
Nvidia Driver Version: 450.102.04
CUDA Version: 10.2
CUDNN Version: 7.6.5
Operating System + Version: Ubuntu 18.04
TensorFlow Version: 1.14.0
The script I used to convert the graph
import tensorflow as tf from tensorflow.python.compiler.tensorrt import trt_convert as trt with tf.Session() as sess: # First deserialize your frozen graph: with tf.gfile.GFile(pb_model_path, 'rb') as f: frozen_graph = tf.GraphDef() frozen_graph.ParseFromString(f.read()) # Now you can create a TensorRT inference graph from your # frozen graph: converter = trt.TrtGraphConverter( input_graph_def=frozen_graph, is_dynamic_op=False, nodes_blacklist=output_nodes) trt_graph = converter.convert() # Import the TensorRT graph into a new graph and run: output_node = tf.import_graph_def( trt_graph, return_elements=output_nodes) for nn in trt_graph.node: if nn.op == "TRTEngineOp": print("Node: %s, %s" % (nn.op, nn.name.replace("/", "_"))) with tf.gfile.GFile("%s.plan" % (nn.name.replace("/", "_")), 'wb') as f: f.write(nn.attr["serialized_segment"].s) else: print("Exclude Node: %s, %s" % (nn.op, nn.name.replace("/", "_")))