Does TensorRT 4.0 support layers for models trained with Keras framework (TensorFlow backend)

hi, does TensorRT 4.0 support layers for models trained with Keras framework (TensorFlow backend)? I am getting these errors with 2 different models trained and frozen graphs created using Keras:

Error: subgraph conversion error for subgraph_index:15 due to:“Unimplemented: Not supported constant type, at batch_normalization_52/Const_1” SKIPPING…( 447 nodes)

Error: subgraph conversion error for subgraph_index:49 due to: “Unimplemented: Not supported constant type, at bn2a_branch2c_1/Const_5” SKIPPING…( 9 nodes)

Thanks

Hello,

please reference TRT dev guide for operations that are supported in a TensorFlow framework.

https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#support_op

How are you freezing the graph? via convert_variables_to_constants? convert_variables_to_constants may not be sufficicent to freeze graphs.

Hi NVES, I am using the example “8.2.2. Freezing A Keras Model” given by nvidia, here is the link https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html and below my code:

from keras.models import load_model
import keras.backend as K
from tensorflow.python.framework import graph_io
from tensorflow.python.tools import freeze_graph
from tensorflow.core.protobuf import saver_pb2
from tensorflow.python.training import saver as saver_lib

# Convert the trained keras model from .h5 format to .pb format
def convert_kerasmodel_to_frozengraph(keras_model, models_dir, model_filename):
  model = load_model(keras_model)
  model.summary()
  K.set_learning_phase(0)
  sess = K.get_session()
  saver = saver_lib.Saver(write_version=saver_pb2.SaverDef.V2)
  checkpoint_path = saver.save(sess, './saved_ckpt', global_step=0, latest_filename='checkpoint_state')
  graph_io.write_graph(sess.graph, '.', 'tmp.pb')
  freeze_graph.freeze_graph('./tmp.pb', '',
                          	False, checkpoint_path, output_name,
                          	"save/restore_all", "save/Const:0",      
                          	models_dir+model_filename, False, "")

Hello,

This is a known issue our engineers are working on. Not related to Keras/TRT compatibility. Apologize for the inconvenience, will keep you updated on the fix.

In the meantime, note that the other parts of the graph may be still converted, only this part that as an empty tensor in batch norm gets skipped.

Hello,

You might be using precompiled pip package with TRT4.0. If tensorflow is compiled with trt3 some features are disabled.

Can you check if you see warning “WARNING:tensorflow:TensorRT mismatch. Compiled against version 3.0.4, but loaded 4.0.1. Things may not workk”.

If you do see similar warnings, please try again with some recent tensorflow, 1.11rc for example.

Thanks

Hi, I am running the test via NVIDIA container image nvcr.io/nvidia/tensorflow:18.07-py3. For the model producing the error batch_normalization the entire tensors are skipped. For the secodn model, the output shows “MULTIPLE tensorrt candidate conversion: 50” but all of them are skipped, see below:

2018-09-21 21:17:40.229968: I tensorflow/core/grappler/devices.cc:51] Number of eligible GPUs (core count >= 8): 1
2018-09-21 21:17:40.451425: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:419] MULTIPLE tensorrt candidate conversion: 50
2018-09-21 21:17:40.455415: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:0 due to: "Unimplemented: Not supported constant type, at bn4b_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.458863: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:1 due to: "Unimplemented: Not supported constant type, at bn4f_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.462340: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:2 due to: "Unimplemented: Not supported constant type, at bn4c_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.465798: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:3 due to: "Unimplemented: Not supported constant type, at bn4f_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.469606: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:4 due to: "Unimplemented: Not supported constant type, at bn4e_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.473111: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:5 due to: "Unimplemented: Not supported constant type, at bn5a_branch1_1/Const_5" SKIPPING......( 18 nodes)
2018-09-21 21:17:40.476510: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:6 due to: "Unimplemented: Not supported constant type, at bn4c_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.479934: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:7 due to: "Unimplemented: Not supported constant type, at bn5a_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.483381: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:8 due to: "Unimplemented: Not supported constant type, at bn4e_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.486797: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:9 due to: "Unimplemented: Not supported constant type, at bn4b_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.490225: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:10 due to: "Unimplemented: Not supported constant type, at bn5b_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.493641: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:11 due to: "Unimplemented: Not supported constant type, at bn5b_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.497070: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:12 due to: "Unimplemented: Not supported constant type, at bn2b_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.514016: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3161] Max batch size= 4 max workspace size= 56946819072
2018-09-21 21:17:40.514042: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3165] Using FP16 precision mode
2018-09-21 21:17:40.514047: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3167] starting build engine
2018-09-21 21:17:40.523723: E tensorflow/contrib/tensorrt/log/trt_logger.cc:38] DefaultLogger runtime.cpp (16) - Cuda Error in allocate: 2
2018-09-21 21:17:40.524237: E tensorflow/contrib/tensorrt/log/trt_logger.cc:38] DefaultLogger runtime.cpp (16) - Cuda Error in allocate: 2
2018-09-21 21:17:40.524257: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3172] Built network
2018-09-21 21:17:40.524297: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:13 due to: "Internal: Engine building failure" SKIPPING......( 13 nodes)
2018-09-21 21:17:40.527823: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:14 due to: "Unimplemented: Not supported constant type, at bn2c_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.531320: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:15 due to: "Unimplemented: Not supported constant type, at bn5c_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.534728: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:16 due to: "Unimplemented: Not supported constant type, at bn5b_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.538189: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:17 due to: "Unimplemented: Not supported constant type, at bn2b_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.541611: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:18 due to: "Unimplemented: Not supported constant type, at bn2a_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.547062: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:19 due to: "Unimplemented: Not supported constant type, at bn3d_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.553540: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:20 due to: "Unimplemented: Not supported constant type, at bn4f_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.558958: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:21 due to: "Unimplemented: Not supported constant type, at bn4d_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.562526: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:22 due to: "Unimplemented: Not supported constant type, at bn_conv1_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.566117: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:23 due to: "Unimplemented: Not supported constant type, at bn5a_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.569496: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:24 due to: "Unimplemented: Not supported constant type, at bn5c_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.572700: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:25 due to: "Unimplemented: Not supported constant type, at bn4b_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.576072: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:26 due to: "Unimplemented: Not supported constant type, at bn4a_branch1_1/Const_5" SKIPPING......( 18 nodes)
2018-09-21 21:17:40.579261: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:27 due to: "Unimplemented: Not supported constant type, at bn4a_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.582453: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:28 due to: "Unimplemented: Not supported constant type, at bn4d_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.585618: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:29 due to: "Unimplemented: Not supported constant type, at bn4a_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.588856: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:30 due to: "Unimplemented: Not supported constant type, at bn3a_branch1_1/Const_5" SKIPPING......( 18 nodes)
2018-09-21 21:17:40.592067: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:31 due to: "Unimplemented: Not supported constant type, at bn4c_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.595192: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:32 due to: "Unimplemented: Not supported constant type, at bn3d_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.598327: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:33 due to: "Unimplemented: Not supported constant type, at bn3d_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.601521: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:34 due to: "Unimplemented: Not supported constant type, at bn3b_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.604657: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:35 due to: "Unimplemented: Not supported constant type, at bn3c_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.607836: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:36 due to: "Unimplemented: Not supported constant type, at bn3c_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.611048: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:37 due to: "Unimplemented: Not supported constant type, at bn3c_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.614194: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:38 due to: "Unimplemented: Not supported constant type, at bn5c_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.617324: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:39 due to: "Unimplemented: Not supported constant type, at bn3b_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.620453: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:40 due to: "Unimplemented: Not supported constant type, at bn4d_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.623587: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:41 due to: "Unimplemented: Not supported constant type, at bn3b_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.626710: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:42 due to: "Unimplemented: Not supported constant type, at bn2c_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.630075: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:43 due to: "Unimplemented: Not supported constant type, at bn3a_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.636838: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:44 due to: "Unimplemented: Not supported constant type, at bn2c_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.642187: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:45 due to: "Unimplemented: Not supported constant type, at bn2b_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.646262: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:46 due to: "Unimplemented: Not supported constant type, at bn4e_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.650288: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:47 due to: "Unimplemented: Not supported constant type, at bn3a_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.653738: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:48 due to: "Unimplemented: Not supported constant type, at bn2a_branch1_1/Const_5" SKIPPING......( 18 nodes)
2018-09-21 21:17:40.657166: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:49 due to: "Unimplemented: Not supported constant type, at bn2a_branch2c_1/Const_5" SKIPPING......( 9 nodes)

Hello,

one of my colleagues recommended “Would it be possible to try with something more recent. We might have already fixed the issues in recent commits. … at least trying 18.09.”

nvcr.io/nvidia/tensorflow:18.07-py3 contains TensorFlow 1.8.0.

Let’s retry with nvcr.io/nvidia/tensorflow:18.09-py3, which contains TF 1.10.

Hi all, I solved the issue with the model implemented in TensorFlow (I couldn’t solve the problem either with direct Keras API natively with TensorFlow backend or with Keras in the TensorFlow implementation).

After trained the model with Tensorflow, I exported the inference graph as a SavedModel with the function

export_savedmodel

I read that to serve a model with TensorFlow serving, we must export the trained model with export_savedmodel method.

Here are the links with more info if somebody faces the same issue:

TensorFlow Estimator
https://www.tensorflow.org/api_docs/python/tf/estimator/Estimator#export_savedmodel

Serving Pre-Modeled and Custom Tensorflow Estimator with Tensorflow Serving
https://medium.com/@yuu.ishikawa/serving-pre-modeled-and-custom-tensorflow-estimator-with-tensorflow-serving-12833b4be421