[TFTRT 4.0.1.6] TFTRT 4.0.1.6 optimize Inception i3d network failure on FP32 mode

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hi,

I tried to optimize a model with network of inception i3d by TFTRT 4.0.1.6.
Below is the log, and I consider it to be a TRT issue.

2018-09-07 16:45:16.422304: I tensorflow/core/grappler/devices.cc:51] Number of eligible GPUs (core count >= 8): 1
2018-09-07 16:45:16.423845: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-09-07 16:45:16.756320: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:383] MULTIPLE tensorrt candidate conversion: 31
2018-09-07 16:45:17.187103: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:0 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Conv3d_2c_3x3/conv_3d/convolution" SKIPPING......( 5 nodes)
2018-09-07 16:45:17.189426: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:1 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_3b/Branch_1/Conv3d_0a_1x1/conv_3d/convolution" SKIPPING......( 5 nodes)
2018-09-07 16:45:17.191707: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:2 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_3b/Branch_3/Conv3d_0b_1x1/conv_3d/convolution" SKIPPING......( 22 nodes)
2018-09-07 16:45:17.193955: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:3 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_3c/Branch_1/Conv3d_0a_1x1/conv_3d/convolution" SKIPPING......( 5 nodes)
2018-09-07 16:45:17.196193: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:4 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_3c/Branch_2/Conv3d_0a_1x1/conv_3d/convolution" SKIPPING......( 5 nodes)
2018-09-07 16:45:17.198473: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:5 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_4b/Branch_1/Conv3d_0a_1x1/conv_3d/convolution" SKIPPING......( 5 nodes)
2018-09-07 16:45:17.200760: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:6 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_4b/Branch_2/Conv3d_0a_1x1/conv_3d/convolution" SKIPPING......( 5 nodes)
2018-09-07 16:45:17.203015: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:7 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_3b/Branch_2/Conv3d_0a_1x1/conv_3d/convolution" SKIPPING......( 5 nodes)
2018-09-07 16:45:17.205279: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:8 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_4b/Branch_3/Conv3d_0b_1x1/conv_3d/convolution" SKIPPING......( 22 nodes)
2018-09-07 16:45:17.207518: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:9 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Conv3d_2b_1x1/conv_3d/convolution" SKIPPING......( 5 nodes)
2018-09-07 16:45:17.209783: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:10 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_4c/Branch_3/Conv3d_0b_1x1/conv_3d/convolution" SKIPPING......( 22 nodes)
2018-09-07 16:45:17.212047: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:11 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_4f/Branch_3/Conv3d_0b_1x1/conv_3d/convolution" SKIPPING......( 22 nodes)
2018-09-07 16:45:17.214285: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:12 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_4f/Branch_1/Conv3d_0a_1x1/conv_3d/convolution" SKIPPING......( 5 nodes)
2018-09-07 16:45:17.216531: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:13 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_4c/Branch_1/Conv3d_0a_1x1/conv_3d/convolution" SKIPPING......( 5 nodes)
2018-09-07 16:45:17.218798: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:14 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Logits/Conv3d_0c_1x1/conv_3d/convolution" SKIPPING......( 2 nodes)
2018-09-07 16:45:17.221132: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:15 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_5c/Branch_3/Conv3d_0b_1x1/conv_3d/convolution" SKIPPING......( 22 nodes)
2018-09-07 16:45:17.223399: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:16 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_4e/Branch_3/Conv3d_0b_1x1/conv_3d/convolution" SKIPPING......( 22 nodes)
2018-09-07 16:45:17.226049: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:17 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_3c/Branch_3/Conv3d_0b_1x1/conv_3d/convolution" SKIPPING......( 22 nodes)
2018-09-07 16:45:17.228287: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:18 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_4d/Branch_2/Conv3d_0a_1x1/conv_3d/convolution" SKIPPING......( 5 nodes)
2018-09-07 16:45:17.230523: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:19 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_5b/Branch_1/Conv3d_0a_1x1/conv_3d/convolution" SKIPPING......( 5 nodes)
2018-09-07 16:45:17.232786: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:20 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_4d/Branch_3/Conv3d_0b_1x1/conv_3d/convolution" SKIPPING......( 22 nodes)
2018-09-07 16:45:17.235023: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:21 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_5c/Branch_2/Conv3d_0a_1x1/conv_3d/convolution" SKIPPING......( 5 nodes)
2018-09-07 16:45:17.237322: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:22 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_5b/Branch_3/Conv3d_0b_1x1/conv_3d/convolution" SKIPPING......( 22 nodes)
2018-09-07 16:45:17.239577: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:23 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Conv3d_1a_7x7/conv_3d/convolution" SKIPPING......( 5 nodes)
2018-09-07 16:45:17.241903: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:24 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_4f/Branch_2/Conv3d_0a_1x1/conv_3d/convolution" SKIPPING......( 5 nodes)
2018-09-07 16:45:17.244137: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:25 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_5b/Branch_2/Conv3d_0a_1x1/conv_3d/convolution" SKIPPING......( 5 nodes)
2018-09-07 16:45:17.246736: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:26 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_4d/Branch_1/Conv3d_0a_1x1/conv_3d/convolution" SKIPPING......( 5 nodes)
2018-09-07 16:45:17.248969: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:27 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_5c/Branch_1/Conv3d_0a_1x1/conv_3d/convolution" SKIPPING......( 5 nodes)
2018-09-07 16:45:17.251202: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:28 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_4e/Branch_2/Conv3d_0a_1x1/conv_3d/convolution" SKIPPING......( 5 nodes)
2018-09-07 16:45:17.253432: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:29 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_4e/Branch_1/Conv3d_0a_1x1/conv_3d/convolution" SKIPPING......( 5 nodes)
2018-09-07 16:45:17.255669: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:412] subgraph conversion error for subgraph_index:30 due to: "Unimplemented: Require 4 dimensional input. Got 5 RGB/inception_i3d/Mixed_4c/Branch_2/Conv3d_0a_1x1/conv_3d/convolution" SKIPPING......( 5 nodes)
2018-09-07 16:45:19.401136: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 0 with properties: 
name: TITAN Xp major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:84:00.0
totalMemory: 11.91GiB freeMemory: 9.70GiB
2018-09-07 16:45:19.401211: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-09-07 16:45:19.401238: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-09-07 16:45:19.401249: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917]      0 
2018-09-07 16:45:19.401258: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0:   N 
2018-09-07 16:45:19.401488: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9756 MB memory) -> physical GPU (device: 0, name: TITAN Xp, pci bus id: 0000:84:00.0, compute capability: 6.1)
Converted 0 variables to const ops.

Finally 0 variables are converted, so the optimization is actually failed.
Any idea will be welcome.

Thanks,
oscarriddle

2

Hello, can you provide details on the platforms you are using?

Linux distro and version
GPU type
nvidia driver version
CUDA version
CUDNN version
Python version [if using python]
Tensorflow version
TensorRT version

Any pd, uff, and source file you can provide will help us debug too. (If you’d like, you can PM me with links to github or google drive for bigger files)

3

Hi,

Below is the platform details of my experiment.
Linux distro and version: CentOS 7
GPU type : TitanXP 12GB
nvidia driver version: 396.26
CUDA version: CUDA 9.0
CUDNN version: cuDNN 7.0.5
Python version [if using python]: Python 2.7
Tensorflow version : Tensorflow-gpu 1.7.0
TensorRT version: TensorRT 4.0.1.6

Thanks,

4

Hello, is it possible for you to try with TF 1.10? we don’t see this error with TF 1.10.

To answer your question, the errors you are seeing means that those nodes are not converted to TRT because they have 5 dimensional inputs. We’ve made improvement in this area in TRT4 and TF 1.9 .

5

Hi,
Thanks for your reply.
I will try to use TF1.10 and TF1.9 to check about this issue.

Thanks,

6

Hi,
I uninstalled tensorflow-gpu==1.7.0 and installed tensorflow-gpu==1.10
Other dependencies are remained the same. [CUDA9.0, TRT4.0.1.6]
Here is the new error:

$ python tftrt_optimizer.py  -m FP32 -b 16
/home/web_server/dlpy72/dlpy/lib/python2.7/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
-----------------------------
TensorFlow == 1.10.0
Namespace(batch_size=u'16', checkpoint=None, input=None, iteration=None, mode=u'FP32', network=None, nodes=None, output=None)
-----------------------------
./frozen.pb
WARNING:tensorflow:TensorRT mismatch. Compiled against version 3.0.4, but loaded 4.0.1. Things may not work
2018-09-17 10:59:59.165397: I tensorflow/core/grappler/devices.cc:51] Number of eligible GPUs (core count >= 8): 1
2018-09-17 10:59:59.734395: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:756] MULTIPLE tensorrt candidate conversion: 31
2018-09-17 10:59:59.734727: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Logits/Conv3d_0c_1x1/conv_3d/', converted to graph
2018-09-17 10:59:59.746141: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_5c/Branch_1/Conv3d_0a_1x1/', converted to graph
2018-09-17 10:59:59.746523: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_5c/', converted to graph
2018-09-17 10:59:59.747826: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_5b/Branch_1/Conv3d_0a_1x1/', converted to graph
2018-09-17 10:59:59.748273: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_4f/', converted to graph
2018-09-17 10:59:59.748899: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_4e/Branch_2/Conv3d_0a_1x1/', converted to graph
2018-09-17 10:59:59.749981: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_4e/Branch_1/Conv3d_0a_1x1/', converted to graph
2018-09-17 10:59:59.750553: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_5b/Branch_2/Conv3d_0a_1x1/', converted to graph
2018-09-17 10:59:59.751268: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_4e/', converted to graph
2018-09-17 10:59:59.752707: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_4f/Branch_1/Conv3d_0a_1x1/', converted to graph
2018-09-17 10:59:59.753334: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_4d/Branch_2/Conv3d_0a_1x1/', converted to graph
2018-09-17 10:59:59.754069: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_5c/Branch_2/Conv3d_0a_1x1/', converted to graph
2018-09-17 10:59:59.755046: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_3c/Branch_1/Conv3d_0a_1x1/', converted to graph
2018-09-17 10:59:59.755853: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_4d/', converted to graph
2018-09-17 10:59:59.756754: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_4f/Branch_2/Conv3d_0a_1x1/', converted to graph
2018-09-17 10:59:59.757428: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Conv3d_1a_7x7/', converted to graph
2018-09-17 10:59:59.758313: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_3b/', converted to graph
2018-09-17 10:59:59.759289: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_3c/Branch_2/Conv3d_0a_1x1/', converted to graph
2018-09-17 10:59:59.760122: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_3c/', converted to graph
2018-09-17 10:59:59.761386: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_3b/Branch_1/Conv3d_0a_1x1/', converted to graph
2018-09-17 10:59:59.762310: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_3b/Branch_2/Conv3d_0a_1x1/', converted to graph
2018-09-17 10:59:59.763267: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_4c/Branch_2/Conv3d_0a_1x1/', converted to graph
2018-09-17 10:59:59.764155: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Conv3d_2c_3x3/', converted to graph
2018-09-17 10:59:59.765561: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_5b/', converted to graph
2018-09-17 10:59:59.766881: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_4b/', converted to graph
2018-09-17 10:59:59.768280: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Conv3d_2b_1x1/', converted to graph
2018-09-17 10:59:59.769759: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_4b/Branch_1/Conv3d_0a_1x1/', converted to graph
2018-09-17 10:59:59.771405: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_4d/Branch_1/Conv3d_0a_1x1/', converted to graph
2018-09-17 10:59:59.772572: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_4b/Branch_2/Conv3d_0a_1x1/', converted to graph
2018-09-17 10:59:59.773884: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_4c/', converted to graph
2018-09-17 10:59:59.775585: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope 'RGB/inception_i3d/Mixed_4c/Branch_1/Conv3d_0a_1x1/', converted to graph
2018-09-17 10:59:59.777661: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 10:59:59.777687: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.267261: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 0, composed of 2 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Logits/Conv3d_0c_1x1/conv_3d/add. Skipping...
2018-09-17 11:00:00.267354: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.267416: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.269674: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 1, composed of 3 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_5c/Branch_1/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.269726: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.269739: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.271990: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 2, composed of 14 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_5c/Branch_0/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.272043: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.272055: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.274286: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 3, composed of 3 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_5b/Branch_1/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.274337: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.274349: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.276588: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 4, composed of 14 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_4f/Branch_0/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.276639: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.276651: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.278934: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 5, composed of 3 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_4e/Branch_2/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.278984: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.278996: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.281219: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 6, composed of 3 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_4e/Branch_1/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.281270: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.281281: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.283501: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 7, composed of 3 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_5b/Branch_2/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.283555: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.283567: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.286253: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 8, composed of 14 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_4e/Branch_0/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.286305: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.286316: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.288538: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 9, composed of 3 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_4f/Branch_1/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.288591: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.288603: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.290824: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 10, composed of 3 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_4d/Branch_2/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.290874: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.290885: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.293105: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 11, composed of 3 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_5c/Branch_2/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.293155: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.293167: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.295384: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 12, composed of 3 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_3c/Branch_1/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.295434: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.295446: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.297695: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 13, composed of 14 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_4d/Branch_0/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.297746: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.297758: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.300012: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 14, composed of 3 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_4f/Branch_2/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.300062: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.300074: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.302300: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 15, composed of 3 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Conv3d_1a_7x7/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.302350: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.302362: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.304624: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 16, composed of 14 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_3b/Branch_0/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.304675: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.304686: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.307300: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 17, composed of 3 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_3c/Branch_2/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.307351: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.307363: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.309594: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 18, composed of 14 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_3c/Branch_0/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.309645: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.309657: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.311878: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 19, composed of 3 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_3b/Branch_1/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.311928: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.311940: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.314160: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 20, composed of 3 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_3b/Branch_2/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.314210: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.314222: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.316442: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 21, composed of 3 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_4c/Branch_2/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.316492: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.316504: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.318786: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 22, composed of 3 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Conv3d_2c_3x3/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.318836: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.318848: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.321082: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 23, composed of 14 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_5b/Branch_0/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.321137: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.321149: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.323383: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 24, composed of 14 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_4b/Branch_0/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.323437: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.323449: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.326107: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 25, composed of 3 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Conv3d_2b_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.326157: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.326184: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.328390: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 26, composed of 3 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_4b/Branch_1/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.328440: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.328452: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.330676: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 27, composed of 3 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_4d/Branch_1/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.330726: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.330737: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.332957: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 28, composed of 3 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_4b/Branch_2/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.333008: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.333020: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.335252: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 29, composed of 14 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_4c/Branch_0/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:00.335306: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-09-17 11:00:00.335317: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0 
2018-09-17 11:00:00.337546: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 30, composed of 3 nodes failed: Invalid argument: addScale requires tensor with rank 3, RGB/inception_i3d/Mixed_4c/Branch_1/Conv3d_0a_1x1/batch_norm/batch_norm/add_1. Skipping...
2018-09-17 11:00:01.158356: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-09-17 11:00:01.159515: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties: 
name: TITAN Xp major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:84:00.0
totalMemory: 11.91GiB freeMemory: 11.66GiB
2018-09-17 11:00:01.159568: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2018-09-17 11:00:01.159621: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-09-17 11:00:01.159635: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0 
2018-09-17 11:00:01.159653: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N 
2018-09-17 11:00:01.159914: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9756 MB memory) -> physical GPU (device: 0, name: TITAN Xp, pci bus id: 0000:84:00.0, compute capability: 6.1)
7

Hope to hear some advice from official side.

Thanks