I have some self-define tensorflow operator, such as GPU_NMS, soft-nms, roi_align, etc.
When I read .pb file convert graph using
trt_graph = trt.create_inference_graph(graph_def, OUTPUT_NAMES,
max_batch_size=cfg.test_batch_per_gpu,
max_workspace_size_bytes=5000000000,
precision_mode=“FP16”) # Get optimized graph
The following error report.
I suspect the problem is that .pb graph convert process can’t deal with self-define operator?
report error :
2018-04-24 18:41:13.508884: E tensorflow/contrib/tensorrt/log/trt_logger.cc:38] DefaultLogger Parameter check failed at: …/builder/Network.cpp::addInput::377, condition: isValidDims(dims)
2018-04-24 18:41:13.508913: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:1 due to: “Invalid argument: Failed to create Input layer” SKIPPING…( 23 nodes)
2018-04-24 18:41:13.510451: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:2 due to: “Unimplemented: Require 4 dimensional input. Got 2 resnet_v1_50_5/ExpandDims” SKIPPING…( 9 nodes)
2018-04-24 18:41:13.512027: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:3 due to: “Unimplemented: Require 4 dimensional input. Got 2 resnet_v1_50_5/Exp_1” SKIPPING…( 9 nodes)
2018-04-24 18:41:13.513630: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:4 due to: “Unimplemented: Require 4 dimensional input. Got 1 resnet_v1_50_5/strided_slice_60” SKIPPING…( 6 nodes)
2018-04-24 18:41:13.515232: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:5 due to: “Unimplemented: Require 4 dimensional input. Got 2 resnet_v1_50_5/Exp_6” SKIPPING…( 9 nodes)
2018-04-24 18:41:13.516776: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:6 due to: “Unimplemented: Require 4 dimensional input. Got 1 resnet_v1_50_5/strided_slice_6” SKIPPING…( 6 nodes)
2018-04-24 18:41:13.518406: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:7 due to: “Unimplemented: Require 4 dimensional input. Got 2 resnet_v1_50_5/ones_2” SKIPPING…( 4 nodes)
2018-04-24 18:41:13.519999: E tensorflow/contrib/tensorrt/log/trt_logger.cc:38] DefaultLogger Parameter check failed at: …/builder/Network.cpp::addInput::377, condition: isValidDims(dims)
2018-04-24 18:41:13.520027: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:8 due to: “Invalid argument: Failed to create Input layer” SKIPPING…( 33 nodes)
2018-04-24 18:41:13.521587: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:9 due to: “Unimplemented: Require 4 dimensional input. Got 2 resnet_v1_50_5/ones_1” SKIPPING…( 3 nodes)
2018-04-24 18:41:13.523129: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:10 due to: “Unimplemented: Require 4 dimensional input. Got 2 resnet_v1_50_6/bbox_fc/MatMul” SKIPPING…( 4 nodes)
2018-04-24 18:41:13.524705: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:11 due to: “Unimplemented: Require 4 dimensional input. Got 2 resnet_v1_50_5/Exp_7” SKIPPING…( 9 nodes)
2018-04-24 18:41:13.526373: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:12 due to: “Unimplemented: Require 4 dimensional input. Got 2 resnet_v1_50_5/ones_3” SKIPPING…( 4 nodes)
2018-04-24 18:41:13.528243: E tensorflow/contrib/tensorrt/log/trt_logger.cc:38] DefaultLogger Parameter check failed at: …/builder/Network.cpp::addInput::377, condition: isValidDims(dims)
2018-04-24 18:41:13.528296: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:13 due to: “Invalid argument: Failed to create Input layer” SKIPPING…( 380 nodes)
2018-04-24 18:41:13.530037: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:14 due to: “Unimplemented: Require 4 dimensional input. Got 1 resnet_v1_50_5/strided_slice_8” SKIPPING…( 6 nodes)
2018-04-24 18:41:13.531584: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:15 due to: “Unimplemented: Require 4 dimensional input. Got 1 resnet_v1_50_5/strided_slice_89” SKIPPING…( 6 nodes)
2018-04-24 18:41:13.533209: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:16 due to: “Unimplemented: Require 4 dimensional input. Got 2 resnet_v1_50_5/ones” SKIPPING…( 4 nodes)
2018-04-24 18:41:13.534757: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:17 due to: “Unimplemented: Require 4 dimensional input. Got 2 resnet_v1_50_5/Exp_5” SKIPPING…( 9 nodes)
2018-04-24 18:41:13.536330: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:18 due to: “Unimplemented: Require 4 dimensional input. Got 2 resnet_v1_50_5/Exp_4” SKIPPING…( 9 nodes)
2018-04-24 18:41:13.537874: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:19 due to: “Unimplemented: Require 4 dimensional input. Got 1 resnet_v1_50_5/strided_slice_62” SKIPPING…( 6 nodes)
2018-04-24 18:41:13.539448: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:20 due to: “Unimplemented: Require 4 dimensional input. Got 2 resnet_v1_50_5/Exp_3” SKIPPING…( 9 nodes)
2018-04-24 18:41:13.541017: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:21 due to: “Unimplemented: Require 4 dimensional input. Got 2 resnet_v1_50_6/ps_fc_1/MatMul” SKIPPING…( 3 nodes)
2018-04-24 18:41:13.542590: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:22 due to: “Unimplemented: Require 4 dimensional input. Got 1 resnet_v1_50_5/strided_slice_33” SKIPPING…( 6 nodes)
2018-04-24 18:41:13.544159: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:23 due to: “Unimplemented: Require 4 dimensional input. Got 2 resnet_v1_50_5/Exp_2” SKIPPING…( 9 nodes)
2018-04-24 18:41:13.545757: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:24 due to: “Unimplemented: Require 4 dimensional input. Got 1 resnet_v1_50_5/strided_slice_35” SKIPPING…( 6 nodes)
2018-04-24 18:41:13.547447: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:25 due to: “Unimplemented: Require 4 dimensional input. Got 1 resnet_v1_50_5/strided_slice_87” SKIPPING…( 6 nodes)
2018-04-24 18:41:14.030285: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with pr
+1 to this issue.
I get similar errors while using Resnet101 based object detection graph.
The two main errors are
“Invalid argument: Failed to create Input layer” SKIPPING…
…( 858 nodes)
and
“Unimplemented: Require 4 dimensional input. Got 1 …” SKIPPING…( 64 nodes)
The reason of this your first issue “Invalid argument: Failed to create Input layer” SKIPPING… is because TensorRT only implement 4D input(feed dict can only input 4D tensor in first layer input). You can create your pseudo 4D tensor, (by tf.expand_dims) and after input , you use tf.squeeze to remove the fake dimension.
The second reason based on offical answer of TensorRT is that FC layer and may other layer currenly don’t support and implement, current tensorRT + tensorflow can only deal with conv2D and 4D layer, but you can accelerate part of your code because TensorRT can only accelerate your backbone ResNet, and last few layers cannot. (if it cannot ,TensoRT just use original Tensorflow op).
Offical Answer:
Currently, TF-TRT only supports TensorRT 3.0.4. Our parser also targets 3.0.4 TensorRT API and we are doing this check on the run-time.
What you did is correct - fixing the input shape inference, please wait for our new release which will use TRT 4.0 API that could handle two dimensional input and MatMul.
Our engineering team is actively working on TensorRT 4.0 integrated into TensorFlow, the new release will be available for you soon. Sorry for the inconvenience.
Looks like the code assumes every segment’s input layer to have static compile-time known shape. I was able to solve a good chunk of issues by making my input placeholder have a constant known shape.
It still complains at a few places that 4 Dim tensor is expected, but at least my entire resnet seems to have been converted.
Hi, deveshy I am getting similar errors. How did you make the input placeholder to have a constant known shape?
Hi Sanchezvr7,
It should be in your model’s InputReceiver fn definition. Where ever you are defining the input placeholder (in my case, it was the placeholder for an image), you can set the shape of it to exact value instead of something like [None, None, None, 3] to [None, 640, 640, 3]. I believe batch dimension can still be None.
Hi deveshy, where can I find the InputReceiver fn definition info?, I am using a frozen graph model for an image classification from somebody else and don’t know what they defined in the input placeholder. I introspected the model loading it to TensorBoard and here is what the input node has:
import/IteratorGetNext
Operation: IteratorGetNext
Attributes (2)
output_shapes
{“list”:{“shape”:[{“dim”:[{“size”:8},{“size”:224},{“size”:224},{“size”:3}]},{“dim”:[{“size”:8},{“size”:-1}]}]}}
output_types
{“list”:{“type”:[“DT_FLOAT”,“DT_INT64”]}}
Inputs (1)
import/OneShotIterator scalar
Outputs (1)
import/Pad 8×224×224×3
Thanks
Input placeholder is the input node/tensor.
What is the input tensor that you are specifying when converting your graph to tensorRT graph?
Another thing. I’m not very familiar with Tensorflow nodes and graphs but looks like your graph was exported with training input fn. Because for inference, I don’t think you will have IteratorGetNext nodes.
The normal procedure is:
During training, you specify a training_input_fn.
During export (into a SavedModel), you specify a serving_input_fn. (If you have access to code, see where export is happening. You can’t change the graph if you just have a frozen graph).
After exporting to SavedModel, you freeze the graph.
My input node is something like the following:
node: {
name : “input”
op : “Placeholder”
attr: {
key : “shape”
value:
shape: {
dim: {
size: 640
}
dim: {
size: 640
}
dim: {
size: 3
}
}
}
}
attr: {
key : “dtype”
value: {
type : DT_UINT8
}
}
}
Hope this helps!
Hi deveshy, the input tensor name is IteratorGetNext, I am using somebody else protobuf frozen graph file to accelerate the inference using tensorrt4 and don’t have access to the code. Also I am using as reference the script https://github.com/tensorflow/models/tree/master/research/tensorrt
I see. I’m afraid I don’t know how to fix an already frozen graph protobuf. It was helpful for me to compare in Tensorboard against the graphs at the end of that README file: https://github.com/tensorflow/models/blob/master/research/tensorrt/README.md#model-links
What language/script are you using to convert your resnet?
I’m using python and use a script that’s just a modification of tensorrt.py.