ValueError: Node *** expects to be colocated with unknown node ***(TensorRT Integration Speeds Up T...

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hello, I wanted to run this test(https://devblogs.nvidia.com/tensorrt-integration-speeds-tensorflow-inference/) to test the TensorFlow-TensorRT integration.

I could normally run this test, but while I tried to modify the code fitting the facenet model(.pb) by davidsandberg, https://github.com/davidsandberg/facenet, I got some problem. ><

how to solve it or what is that mean? Thank u.

Tensorflow version: 1.7.0-rc1
CUDA/cuDNN version: 9.0 / 7.0.5
TensorRT version: 4.0.0.3
GPU model and memory: GTX 980M, 4Gb

...
...
2018-04-19 16:27:23.756945: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:417 due to: "Unimplemented: Require 4 dimensional input. Got 1 InceptionResnetV1/Repeat_1/block17_9/Branch_1/Conv2d_0c_7x1/BatchNorm/cond/AssignMovingAvg/sub/Switch" SKIPPING......( 4 nodes)
2018-04-19 16:27:23.758911: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:418 due to: "Unimplemented: Require 4 dimensional input. Got 1 InceptionResnetV1/Repeat/block35_5/Branch_2/Conv2d_0a_1x1/BatchNorm/cond/AssignMovingAvg/sub/Switch" SKIPPING......( 4 nodes)
2018-04-19 16:27:23.761851: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:419 due to: "Unimplemented: Require 4 dimensional input. Got 1 InceptionResnetV1/Mixed_7a/Branch_2/Conv2d_0b_3x3/BatchNorm/cond/Merge_1" SKIPPING......( 6 nodes)
2018-04-19 16:27:23.764425: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:418] subgraph conversion error for subgraph_index:420 due to: "Unimplemented: Require 4 dimensional input. Got 1 InceptionResnetV1/Repeat_1/block17_8/Branch_1/Conv2d_0b_1x7/BatchNorm/cond/AssignMovingAvg_1/sub/Switch" SKIPPING......( 4 nodes)
INFO:tensorflow:Starting execution
Traceback (most recent call last):
  File "facenet_tftrt.py", line 225, in <module>
    timings,comp,valfp16,mdstats=timeGraph( getFP16(f.batch_size,wsize), f.batch_size, f.num_loops,dummy_input,timelineName)
  File "facenet_tftrt.py", line 88, in timeGraph
    out = tf.import_graph_def( graph_def=gdef, input_map={"input":next_element, "phase_train": False}, return_elements=["embeddings"] )
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 432, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 493, in import_graph_def
    raise ValueError(str(e))
ValueError: Node 'InceptionResnetV1/Conv2d_1a_3x3/BatchNorm/cond/AssignMovingAvg/decay' expects to be colocated with unknown node 'InceptionResnetV1/Conv2d_1a_3x3/BatchNorm/moving_mean'
2

I am also running into similar issue. I use TensorFlow 1.7, TensorRT 3.0.4. Thanks in advance.

3

We created a new “Deep Learning Training and Inference” section in Devtalk to improve the experience for deep learning and accelerated computing, and HPC users:
https://devtalk.nvidia.com/default/board/301/deep-learning-training-and-inference-/

We are moving active deep learning threads to the new section.

URLs for topics will not change with the re-categorization. So your bookmarks and links will continue to work as earlier.

-Siddharth

4

I met the same issue when I was running the trt-tf model. I guess it happened because there are some ops not handled correctly when they cannot be converted?
Any solutions now? THX