Hi, I am running the test via NVIDIA container image nvcr.io/nvidia/tensorflow:18.07-py3. For the model producing the error batch_normalization the entire tensors are skipped. For the secodn model, the output shows “MULTIPLE tensorrt candidate conversion: 50” but all of them are skipped, see below:
2018-09-21 21:17:40.229968: I tensorflow/core/grappler/devices.cc:51] Number of eligible GPUs (core count >= 8): 1
2018-09-21 21:17:40.451425: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:419] MULTIPLE tensorrt candidate conversion: 50
2018-09-21 21:17:40.455415: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:0 due to: "Unimplemented: Not supported constant type, at bn4b_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.458863: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:1 due to: "Unimplemented: Not supported constant type, at bn4f_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.462340: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:2 due to: "Unimplemented: Not supported constant type, at bn4c_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.465798: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:3 due to: "Unimplemented: Not supported constant type, at bn4f_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.469606: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:4 due to: "Unimplemented: Not supported constant type, at bn4e_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.473111: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:5 due to: "Unimplemented: Not supported constant type, at bn5a_branch1_1/Const_5" SKIPPING......( 18 nodes)
2018-09-21 21:17:40.476510: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:6 due to: "Unimplemented: Not supported constant type, at bn4c_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.479934: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:7 due to: "Unimplemented: Not supported constant type, at bn5a_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.483381: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:8 due to: "Unimplemented: Not supported constant type, at bn4e_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.486797: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:9 due to: "Unimplemented: Not supported constant type, at bn4b_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.490225: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:10 due to: "Unimplemented: Not supported constant type, at bn5b_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.493641: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:11 due to: "Unimplemented: Not supported constant type, at bn5b_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.497070: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:12 due to: "Unimplemented: Not supported constant type, at bn2b_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.514016: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3161] Max batch size= 4 max workspace size= 56946819072
2018-09-21 21:17:40.514042: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3165] Using FP16 precision mode
2018-09-21 21:17:40.514047: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3167] starting build engine
2018-09-21 21:17:40.523723: E tensorflow/contrib/tensorrt/log/trt_logger.cc:38] DefaultLogger runtime.cpp (16) - Cuda Error in allocate: 2
2018-09-21 21:17:40.524237: E tensorflow/contrib/tensorrt/log/trt_logger.cc:38] DefaultLogger runtime.cpp (16) - Cuda Error in allocate: 2
2018-09-21 21:17:40.524257: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3172] Built network
2018-09-21 21:17:40.524297: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:13 due to: "Internal: Engine building failure" SKIPPING......( 13 nodes)
2018-09-21 21:17:40.527823: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:14 due to: "Unimplemented: Not supported constant type, at bn2c_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.531320: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:15 due to: "Unimplemented: Not supported constant type, at bn5c_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.534728: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:16 due to: "Unimplemented: Not supported constant type, at bn5b_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.538189: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:17 due to: "Unimplemented: Not supported constant type, at bn2b_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.541611: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:18 due to: "Unimplemented: Not supported constant type, at bn2a_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.547062: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:19 due to: "Unimplemented: Not supported constant type, at bn3d_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.553540: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:20 due to: "Unimplemented: Not supported constant type, at bn4f_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.558958: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:21 due to: "Unimplemented: Not supported constant type, at bn4d_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.562526: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:22 due to: "Unimplemented: Not supported constant type, at bn_conv1_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.566117: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:23 due to: "Unimplemented: Not supported constant type, at bn5a_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.569496: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:24 due to: "Unimplemented: Not supported constant type, at bn5c_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.572700: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:25 due to: "Unimplemented: Not supported constant type, at bn4b_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.576072: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:26 due to: "Unimplemented: Not supported constant type, at bn4a_branch1_1/Const_5" SKIPPING......( 18 nodes)
2018-09-21 21:17:40.579261: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:27 due to: "Unimplemented: Not supported constant type, at bn4a_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.582453: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:28 due to: "Unimplemented: Not supported constant type, at bn4d_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.585618: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:29 due to: "Unimplemented: Not supported constant type, at bn4a_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.588856: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:30 due to: "Unimplemented: Not supported constant type, at bn3a_branch1_1/Const_5" SKIPPING......( 18 nodes)
2018-09-21 21:17:40.592067: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:31 due to: "Unimplemented: Not supported constant type, at bn4c_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.595192: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:32 due to: "Unimplemented: Not supported constant type, at bn3d_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.598327: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:33 due to: "Unimplemented: Not supported constant type, at bn3d_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.601521: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:34 due to: "Unimplemented: Not supported constant type, at bn3b_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.604657: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:35 due to: "Unimplemented: Not supported constant type, at bn3c_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.607836: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:36 due to: "Unimplemented: Not supported constant type, at bn3c_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.611048: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:37 due to: "Unimplemented: Not supported constant type, at bn3c_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.614194: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:38 due to: "Unimplemented: Not supported constant type, at bn5c_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.617324: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:39 due to: "Unimplemented: Not supported constant type, at bn3b_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.620453: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:40 due to: "Unimplemented: Not supported constant type, at bn4d_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.623587: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:41 due to: "Unimplemented: Not supported constant type, at bn3b_branch2a_1/Const_5" SKIPPING......( 10 nodes)
2018-09-21 21:17:40.626710: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:42 due to: "Unimplemented: Not supported constant type, at bn2c_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.630075: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:43 due to: "Unimplemented: Not supported constant type, at bn3a_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.636838: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:44 due to: "Unimplemented: Not supported constant type, at bn2c_branch2b_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.642187: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:45 due to: "Unimplemented: Not supported constant type, at bn2b_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.646262: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:46 due to: "Unimplemented: Not supported constant type, at bn4e_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.650288: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:47 due to: "Unimplemented: Not supported constant type, at bn3a_branch2c_1/Const_5" SKIPPING......( 9 nodes)
2018-09-21 21:17:40.653738: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:48 due to: "Unimplemented: Not supported constant type, at bn2a_branch1_1/Const_5" SKIPPING......( 18 nodes)
2018-09-21 21:17:40.657166: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:454] subgraph conversion error for subgraph_index:49 due to: "Unimplemented: Not supported constant type, at bn2a_branch2c_1/Const_5" SKIPPING......( 9 nodes)