While converting my network with TF-TRT I receive the error:
ValueError: Value for attr 'InT' of int32 is not in the list of allowed values: int8, half, float
I am using the new nvcr.io/nvidia/tensorflow:19.02-py3 docker image. The previous 19.01 version worked with no issue. Is the new version of Tensorflow causing the issue?
Hello,
container 19.01 to 19.02 updated tensorflow from 1.12.0 to 1.12.2. This is not expected. To help us debug, can you share a small repro package that contains the conversion code and the model you are attempting to convert?
As requested snippet of code is as follows:
precision_mode = "FP16"
trt_graph = trt.create_inference_graph(
frozen_graph_pb, outputs=output_names,
precision_mode=precision_mode, maximum_cached_engines=20,
minimum_segment_size=1, max_workspace_size_bytes=GiB(4), is_dynamic_op=True
)
My .pb file can be found here: https://drive.google.com/file/d/1QkXxDAsieVD8YeWbFcCY38gj84Rin8pT/view?usp=sharing
Hello,
I’m can successfully parse your frozen graph in both 19.01 and 19.02 container. Attached is the same conversion code i’m using.
---> d20cceea8892
Successfully built d20cceea8892
Successfully tagged repro2527206:latest
================
== TensorFlow ==
================
NVIDIA Release 19.02 (build 5618942)
TensorFlow Version v1.13.0-rc0
Container image Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
Copyright 2017-2018 The TensorFlow Authors. All rights reserved.
Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying project or file.
NOTE: Detected MOFED driver 4.4-2.0.7; version automatically upgraded.
root@4b00104f50d4:/test# python test.py
WARNING:tensorflow:From test.py:12: FastGFile.__init__ (from tensorflow.python.platform.gfile) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.gfile.GFile.
2019-03-06 17:53:46.066487: I tensorflow/core/grappler/devices.cc:51] Number of eligible GPUs (core count >= 8): 16
2019-03-06 17:53:46.066765: I tensorflow/core/grappler/clusters/single_machine.cc:359] Starting new session
2019-03-06 17:53:46.103416: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3100000000 Hz
2019-03-06 17:53:46.108819: I tensorflow/compiler/xla/service/service.cc:161] XLA service 0x327cfe80 executing computations on platform Host. Devices:
2019-03-06 17:53:46.108849: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (0): <undefined>, <undefined>
2019-03-06 17:53:46.111896: I tensorflow/compiler/xla/service/service.cc:161] XLA service 0x3282cbc0 executing computations on platform CUDA. Devices:
2019-03-06 17:53:46.111919: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (0): Tesla V100-SXM3-32GB, Compute Capability 7.0
2019-03-06 17:53:46.111926: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (1): Tesla V100-SXM3-32GB, Compute Capability 7.0
2019-03-06 17:53:46.111932: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (2): Tesla V100-SXM3-32GB, Compute Capability 7.0
2019-03-06 17:53:46.111938: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (3): Tesla V100-SXM3-32GB, Compute Capability 7.0
2019-03-06 17:53:46.111944: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (4): Tesla V100-SXM3-32GB, Compute Capability 7.0
2019-03-06 17:53:46.111950: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (5): Tesla V100-SXM3-32GB, Compute Capability 7.0
2019-03-06 17:53:46.111956: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (6): Tesla V100-SXM3-32GB, Compute Capability 7.0
2019-03-06 17:53:46.111962: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (7): Tesla V100-SXM3-32GB, Compute Capability 7.0
2019-03-06 17:53:46.111968: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (8): Tesla V100-SXM3-32GB, Compute Capability 7.0
2019-03-06 17:53:46.111974: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (9): Tesla V100-SXM3-32GB, Compute Capability 7.0
2019-03-06 17:53:46.111980: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (10): Tesla V100-SXM3-32GB, Compute Capability 7.0
2019-03-06 17:53:46.111985: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (11): Tesla V100-SXM3-32GB, Compute Capability 7.0
2019-03-06 17:53:46.111991: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (12): Tesla V100-SXM3-32GB, Compute Capability 7.0
2019-03-06 17:53:46.111997: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (13): Tesla V100-SXM3-32GB, Compute Capability 7.0
2019-03-06 17:53:46.112003: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (14): Tesla V100-SXM3-32GB, Compute Capability 7.0
2019-03-06 17:53:46.112009: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (15): Tesla V100-SXM3-32GB, Compute Capability 7.0
2019-03-06 17:53:46.114464: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:34:00.0
totalMemory: 31.72GiB freeMemory: 26.92GiB
2019-03-06 17:53:46.114866: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 1 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:36:00.0
totalMemory: 31.72GiB freeMemory: 26.92GiB
2019-03-06 17:53:46.115240: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 2 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:39:00.0
totalMemory: 31.72GiB freeMemory: 26.92GiB
2019-03-06 17:53:46.115605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 3 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:3b:00.0
totalMemory: 31.72GiB freeMemory: 26.92GiB
2019-03-06 17:53:46.115996: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 4 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:57:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:53:46.116387: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 5 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:59:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:53:46.116777: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 6 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:5c:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:53:46.117167: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 7 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:5e:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:53:46.117555: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 8 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:b7:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:53:46.117939: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 9 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:b9:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:53:46.118325: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 10 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:bc:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:53:46.118716: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 11 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:be:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:53:46.119100: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 12 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:e0:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:53:46.119508: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 13 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:e2:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:53:46.119897: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 14 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:e5:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:53:46.120278: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 15 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:e7:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:53:46.120742: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15
2019-03-06 17:53:55.272741: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-03-06 17:53:55.272796: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
2019-03-06 17:53:55.272804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
2019-03-06 17:53:55.272810: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: Y N Y Y Y Y Y Y Y Y Y Y Y Y Y Y
2019-03-06 17:53:55.272815: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: Y Y N Y Y Y Y Y Y Y Y Y Y Y Y Y
2019-03-06 17:53:55.272820: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: Y Y Y N Y Y Y Y Y Y Y Y Y Y Y Y
2019-03-06 17:53:55.272825: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 4: Y Y Y Y N Y Y Y Y Y Y Y Y Y Y Y
2019-03-06 17:53:55.272830: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 5: Y Y Y Y Y N Y Y Y Y Y Y Y Y Y Y
2019-03-06 17:53:55.272839: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 6: Y Y Y Y Y Y N Y Y Y Y Y Y Y Y Y
2019-03-06 17:53:55.272844: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 7: Y Y Y Y Y Y Y N Y Y Y Y Y Y Y Y
2019-03-06 17:53:55.272849: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 8: Y Y Y Y Y Y Y Y N Y Y Y Y Y Y Y
2019-03-06 17:53:55.272855: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 9: Y Y Y Y Y Y Y Y Y N Y Y Y Y Y Y
2019-03-06 17:53:55.272861: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 10: Y Y Y Y Y Y Y Y Y Y N Y Y Y Y Y
2019-03-06 17:53:55.272867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 11: Y Y Y Y Y Y Y Y Y Y Y N Y Y Y Y
2019-03-06 17:53:55.272872: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 12: Y Y Y Y Y Y Y Y Y Y Y Y N Y Y Y
2019-03-06 17:53:55.272894: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 13: Y Y Y Y Y Y Y Y Y Y Y Y Y N Y Y
2019-03-06 17:53:55.272901: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 14: Y Y Y Y Y Y Y Y Y Y Y Y Y Y N Y
2019-03-06 17:53:55.272907: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 15: Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y N
2019-03-06 17:53:55.279062: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 26067 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM3-32GB, pci bus id: 0000:34:00.0, compute capability: 7.0)
2019-03-06 17:53:55.280587: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 26063 MB memory) -> physical GPU (device: 1, name: Tesla V100-SXM3-32GB, pci bus id: 0000:36:00.0, compute capability: 7.0)
2019-03-06 17:53:55.281879: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 26063 MB memory) -> physical GPU (device: 2, name: Tesla V100-SXM3-32GB, pci bus id: 0000:39:00.0, compute capability: 7.0)
2019-03-06 17:53:55.283078: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 26067 MB memory) -> physical GPU (device: 3, name: Tesla V100-SXM3-32GB, pci bus id: 0000:3b:00.0, compute capability: 7.0)
2019-03-06 17:53:55.284266: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:4 with 29565 MB memory) -> physical GPU (device: 4, name: Tesla V100-SXM3-32GB, pci bus id: 0000:57:00.0, compute capability: 7.0)
2019-03-06 17:53:55.285398: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:5 with 29565 MB memory) -> physical GPU (device: 5, name: Tesla V100-SXM3-32GB, pci bus id: 0000:59:00.0, compute capability: 7.0)
2019-03-06 17:53:55.286492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:6 with 29565 MB memory) -> physical GPU (device: 6, name: Tesla V100-SXM3-32GB, pci bus id: 0000:5c:00.0, compute capability: 7.0)
2019-03-06 17:53:55.287662: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:7 with 29565 MB memory) -> physical GPU (device: 7, name: Tesla V100-SXM3-32GB, pci bus id: 0000:5e:00.0, compute capability: 7.0)
2019-03-06 17:53:55.288850: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:8 with 29565 MB memory) -> physical GPU (device: 8, name: Tesla V100-SXM3-32GB, pci bus id: 0000:b7:00.0, compute capability: 7.0)
2019-03-06 17:53:55.289854: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:9 with 29565 MB memory) -> physical GPU (device: 9, name: Tesla V100-SXM3-32GB, pci bus id: 0000:b9:00.0, compute capability: 7.0)
2019-03-06 17:53:55.290900: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:10 with 29565 MB memory) -> physical GPU (device: 10, name: Tesla V100-SXM3-32GB, pci bus id: 0000:bc:00.0, compute capability: 7.0)
2019-03-06 17:53:55.291964: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:11 with 29565 MB memory) -> physical GPU (device: 11, name: Tesla V100-SXM3-32GB, pci bus id: 0000:be:00.0, compute capability: 7.0)
2019-03-06 17:53:55.293011: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:12 with 29565 MB memory) -> physical GPU (device: 12, name: Tesla V100-SXM3-32GB, pci bus id: 0000:e0:00.0, compute capability: 7.0)
2019-03-06 17:53:55.294045: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:13 with 29565 MB memory) -> physical GPU (device: 13, name: Tesla V100-SXM3-32GB, pci bus id: 0000:e2:00.0, compute capability: 7.0)
2019-03-06 17:53:55.295068: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:14 with 29565 MB memory) -> physical GPU (device: 14, name: Tesla V100-SXM3-32GB, pci bus id: 0000:e5:00.0, compute capability: 7.0)
2019-03-06 17:53:55.296107: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:15 with 29565 MB memory) -> physical GPU (device: 15, name: Tesla V100-SXM3-32GB, pci bus id: 0000:e7:00.0, compute capability: 7.0)
2019-03-06 17:53:57.255368: I tensorflow/contrib/tensorrt/segment/segment.cc:461] There are 396 ops of 56 different types in the graph that are not converted to TensorRT: TensorArrayWriteV3, PadV2, TensorArrayGatherV3, TensorArraySizeV3, TensorArrayScatterV3, NextIteration, Exit, ExpandDims, NonMaxSuppressionV3, DataFormatVecPermute, Transpose, GatherV2, LogicalAnd, ConcatV2, Pad, Merge, Split, SparseToDense, NoOp, TensorArrayV3, LoopCond, Enter, ResizeNearestNeighbor, ArgMax, Identity, Shape, Sqrt, Placeholder, TensorArrayReadV3, RealDiv, Add, Pack, Reshape, Mul, Equal, Unique, Const, StopGradient, Exp, Where, Maximum, TopKV2, Round, Sub, GatherNd, Cast, Squeeze, CropAndResize, Minimum, GreaterEqual, Greater, Range, DenseToDenseSetOperation, StridedSlice, Less, Switch, (For more information see https://docs.nvidia.com/deeplearning/dgx/integrate-tf-trt/index.html#support-ops).
2019-03-06 17:53:59.169274: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:928] Number of TensorRT candidate segments: 27
2019-03-06 17:53:59.352235: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node TRTEngineOp_0 added for segment 0 consisting of 13 nodes succeeded.
2019-03-06 17:53:59.352342: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node TRTEngineOp_1 added for segment 1 consisting of 3 nodes succeeded.
2019-03-06 17:53:59.352502: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node TRTEngineOp_2 added for segment 2 consisting of 29 nodes succeeded.
2019-03-06 17:53:59.352619: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node TRTEngineOp_3 added for segment 3 consisting of 6 nodes succeeded.
2019-03-06 17:53:59.352679: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node activation_68/TRTEngineOp_4 added for segment 4 consisting of 1 nodes succeeded.
2019-03-06 17:53:59.352720: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node TRTEngineOp_5 added for segment 5 consisting of 3 nodes succeeded.
2019-03-06 17:53:59.592281: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node TRTEngineOp_6 added for segment 6 consisting of 1093 nodes succeeded.
2019-03-06 17:53:59.604539: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node TRTEngineOp_7 added for segment 7 consisting of 16 nodes succeeded.
2019-03-06 17:53:59.613379: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node TRTEngineOp_8 added for segment 8 consisting of 16 nodes succeeded.
2019-03-06 17:53:59.623016: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node TRTEngineOp_9 added for segment 9 consisting of 16 nodes succeeded.
2019-03-06 17:53:59.623085: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:1036] TensorRT node mrcnn_bbox_fc/TRTEngineOp_10 added for segment 10 consisting of 1 nodes failed: Internal: Segment has no inputs (possible constfold failure). Fallback to TF...
2019-03-06 17:53:59.623233: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node mrcnn_bbox_fc/TRTEngineOp_11 added for segment 11 consisting of 3 nodes succeeded.
2019-03-06 17:53:59.623289: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node mrcnn_class/TRTEngineOp_12 added for segment 12 consisting of 1 nodes succeeded.
2019-03-06 17:53:59.623355: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node mrcnn_class_bn1/TRTEngineOp_13 added for segment 13 consisting of 5 nodes succeeded.
2019-03-06 17:53:59.623430: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node mrcnn_class_bn2/TRTEngineOp_14 added for segment 14 consisting of 5 nodes succeeded.
2019-03-06 17:53:59.693614: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node mrcnn_class_conv1/TRTEngineOp_15 added for segment 15 consisting of 4 nodes succeeded.
2019-03-06 17:53:59.696562: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node mrcnn_class_conv2/TRTEngineOp_16 added for segment 16 consisting of 4 nodes succeeded.
2019-03-06 17:53:59.696603: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:1036] TensorRT node mrcnn_class_logits/TRTEngineOp_17 added for segment 17 consisting of 1 nodes failed: Internal: Segment has no inputs (possible constfold failure). Fallback to TF...
2019-03-06 17:53:59.696697: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node mrcnn_class_logits/TRTEngineOp_18 added for segment 18 consisting of 3 nodes succeeded.
2019-03-06 17:53:59.696751: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node mrcnn_detection/TRTEngineOp_19 added for segment 19 consisting of 2 nodes succeeded.
2019-03-06 17:53:59.696823: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node TRTEngineOp_20 added for segment 20 consisting of 1 nodes succeeded.
2019-03-06 17:53:59.696883: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node TRTEngineOp_21 added for segment 21 consisting of 1 nodes succeeded.
2019-03-06 17:53:59.696932: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node mrcnn_detection/TRTEngineOp_22 added for segment 22 consisting of 2 nodes succeeded.
2019-03-06 17:53:59.696997: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node TRTEngineOp_23 added for segment 23 consisting of 3 nodes succeeded.
2019-03-06 17:53:59.697059: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node TRTEngineOp_24 added for segment 24 consisting of 1 nodes succeeded.
2019-03-06 17:53:59.697135: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node roi_align_classifier/TRTEngineOp_25 added for segment 25 consisting of 4 nodes succeeded.
2019-03-06 17:53:59.697185: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:1030] TensorRT node zero_padding2d_1/TRTEngineOp_26 added for segment 26 consisting of 2 nodes succeeded.
2019-03-06 17:53:59.736174: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:581] Optimization results for grappler item: tf_graph
2019-03-06 17:53:59.736201: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:583] constant folding: Graph size after: 1658 nodes (10), 2221 edges (10), time = 342.68ms.
2019-03-06 17:53:59.736208: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:583] layout: Graph size after: 1702 nodes (44), 2292 edges (71), time = 133.808ms.
2019-03-06 17:53:59.736214: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:583] constant folding: Graph size after: 1702 nodes (0), 2292 edges (0), time = 119.793ms.
2019-03-06 17:53:59.736220: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:583] TensorRTOptimizer: Graph size after: 490 nodes (-1212), 1016 edges (-1276), time = 3070.198ms.
Successfully built 8c8f6f8e3a52
Successfully tagged repro2527206:latest
================
== TensorFlow ==
================
NVIDIA Release 19.01 (build 5238117)
TensorFlow Version 1.12.0+
Container image Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
Copyright 2017-2018 The TensorFlow Authors. All rights reserved.
Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying project or file.
NOTE: Detected MOFED driver 4.4-2.0.7; version automatically upgraded.
root@07db8026c5c0:/test# python test.py
WARNING:tensorflow:From test.py:12: FastGFile.__init__ (from tensorflow.python.platform.gfile) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.gfile.GFile.
2019-03-06 17:57:45.298458: I tensorflow/core/grappler/devices.cc:51] Number of eligible GPUs (core count >= 8): 16
2019-03-06 17:57:45.298625: I tensorflow/core/grappler/clusters/single_machine.cc:359] Starting new session
2019-03-06 17:57:45.311126: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:34:00.0
totalMemory: 31.72GiB freeMemory: 26.92GiB
2019-03-06 17:57:45.311558: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:36:00.0
totalMemory: 31.72GiB freeMemory: 26.92GiB
2019-03-06 17:57:45.311928: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 2 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:39:00.0
totalMemory: 31.72GiB freeMemory: 26.92GiB
2019-03-06 17:57:45.312304: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 3 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:3b:00.0
totalMemory: 31.72GiB freeMemory: 26.92GiB
2019-03-06 17:57:45.312692: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 4 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:57:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:57:45.313081: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 5 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:59:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:57:45.313461: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 6 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:5c:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:57:45.313839: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 7 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:5e:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:57:45.314216: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 8 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:b7:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:57:45.314593: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 9 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:b9:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:57:45.314969: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 10 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:bc:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:57:45.315380: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 11 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:be:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:57:45.315777: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 12 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:e0:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:57:45.316154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 13 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:e2:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:57:45.316531: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 14 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:e5:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:57:45.316900: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 15 with properties:
name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.702
pciBusID: 0000:e7:00.0
totalMemory: 31.72GiB freeMemory: 30.52GiB
2019-03-06 17:57:45.317361: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15
2019-03-06 17:57:53.926392: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-03-06 17:57:53.926441: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
2019-03-06 17:57:53.926449: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
2019-03-06 17:57:53.926455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: Y N Y Y Y Y Y Y Y Y Y Y Y Y Y Y
2019-03-06 17:57:53.926461: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 2: Y Y N Y Y Y Y Y Y Y Y Y Y Y Y Y
2019-03-06 17:57:53.926466: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 3: Y Y Y N Y Y Y Y Y Y Y Y Y Y Y Y
2019-03-06 17:57:53.926471: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 4: Y Y Y Y N Y Y Y Y Y Y Y Y Y Y Y
2019-03-06 17:57:53.926476: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 5: Y Y Y Y Y N Y Y Y Y Y Y Y Y Y Y
2019-03-06 17:57:53.926484: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 6: Y Y Y Y Y Y N Y Y Y Y Y Y Y Y Y
2019-03-06 17:57:53.926488: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 7: Y Y Y Y Y Y Y N Y Y Y Y Y Y Y Y
2019-03-06 17:57:53.926493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 8: Y Y Y Y Y Y Y Y N Y Y Y Y Y Y Y
2019-03-06 17:57:53.926498: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 9: Y Y Y Y Y Y Y Y Y N Y Y Y Y Y Y
2019-03-06 17:57:53.926504: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 10: Y Y Y Y Y Y Y Y Y Y N Y Y Y Y Y
2019-03-06 17:57:53.926510: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 11: Y Y Y Y Y Y Y Y Y Y Y N Y Y Y Y
2019-03-06 17:57:53.926516: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 12: Y Y Y Y Y Y Y Y Y Y Y Y N Y Y Y
2019-03-06 17:57:53.926522: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 13: Y Y Y Y Y Y Y Y Y Y Y Y Y N Y Y
2019-03-06 17:57:53.926544: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 14: Y Y Y Y Y Y Y Y Y Y Y Y Y Y N Y
2019-03-06 17:57:53.926551: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 15: Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y N
2019-03-06 17:57:53.932849: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 26076 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM3-32GB, pci bus id: 0000:34:00.0, compute capability: 7.0)
2019-03-06 17:57:53.933424: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 26073 MB memory) -> physical GPU (device: 1, name: Tesla V100-SXM3-32GB, pci bus id: 0000:36:00.0, compute capability: 7.0)
2019-03-06 17:57:53.933878: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 26073 MB memory) -> physical GPU (device: 2, name: Tesla V100-SXM3-32GB, pci bus id: 0000:39:00.0, compute capability: 7.0)
2019-03-06 17:57:53.934232: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 26076 MB memory) -> physical GPU (device: 3, name: Tesla V100-SXM3-32GB, pci bus id: 0000:3b:00.0, compute capability: 7.0)
2019-03-06 17:57:53.934612: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:4 with 29574 MB memory) -> physical GPU (device: 4, name: Tesla V100-SXM3-32GB, pci bus id: 0000:57:00.0, compute capability: 7.0)
2019-03-06 17:57:53.934981: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:5 with 29574 MB memory) -> physical GPU (device: 5, name: Tesla V100-SXM3-32GB, pci bus id: 0000:59:00.0, compute capability: 7.0)
2019-03-06 17:57:53.935361: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:6 with 29574 MB memory) -> physical GPU (device: 6, name: Tesla V100-SXM3-32GB, pci bus id: 0000:5c:00.0, compute capability: 7.0)
2019-03-06 17:57:53.935744: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:7 with 29574 MB memory) -> physical GPU (device: 7, name: Tesla V100-SXM3-32GB, pci bus id: 0000:5e:00.0, compute capability: 7.0)
2019-03-06 17:57:53.936092: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:8 with 29574 MB memory) -> physical GPU (device: 8, name: Tesla V100-SXM3-32GB, pci bus id: 0000:b7:00.0, compute capability: 7.0)
2019-03-06 17:57:53.936394: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:9 with 29574 MB memory) -> physical GPU (device: 9, name: Tesla V100-SXM3-32GB, pci bus id: 0000:b9:00.0, compute capability: 7.0)
2019-03-06 17:57:53.936879: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:10 with 29574 MB memory) -> physical GPU (device: 10, name: Tesla V100-SXM3-32GB, pci bus id: 0000:bc:00.0, compute capability: 7.0)
2019-03-06 17:57:53.937314: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:11 with 29574 MB memory) -> physical GPU (device: 11, name: Tesla V100-SXM3-32GB, pci bus id: 0000:be:00.0, compute capability: 7.0)
2019-03-06 17:57:53.937646: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:12 with 29574 MB memory) -> physical GPU (device: 12, name: Tesla V100-SXM3-32GB, pci bus id: 0000:e0:00.0, compute capability: 7.0)
2019-03-06 17:57:53.937956: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:13 with 29574 MB memory) -> physical GPU (device: 13, name: Tesla V100-SXM3-32GB, pci bus id: 0000:e2:00.0, compute capability: 7.0)
2019-03-06 17:57:53.938317: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:14 with 29574 MB memory) -> physical GPU (device: 14, name: Tesla V100-SXM3-32GB, pci bus id: 0000:e5:00.0, compute capability: 7.0)
2019-03-06 17:57:53.938688: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:15 with 29574 MB memory) -> physical GPU (device: 15, name: Tesla V100-SXM3-32GB, pci bus id: 0000:e7:00.0, compute capability: 7.0)
2019-03-06 17:57:57.153080: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:868] MULTIPLE tensorrt candidate conversion: 19
2019-03-06 17:57:57.154266: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope '', converted to graph
2019-03-06 17:57:57.154286: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-03-06 17:57:57.159554: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope 'ROI/', converted to graph
2019-03-06 17:57:57.159573: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-03-06 17:57:57.160473: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope 'activation_68/', converted to graph
2019-03-06 17:57:57.160490: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-03-06 17:57:57.160867: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope 'activation_69/', converted to graph
2019-03-06 17:57:57.160883: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-03-06 17:57:57.161242: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope 'mrcnn_bbox_fc/', converted to graph
2019-03-06 17:57:57.161257: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-03-06 17:57:57.161966: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope 'mrcnn_bbox_fc/', converted to graph
2019-03-06 17:57:57.162000: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-03-06 17:57:57.162753: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope 'mrcnn_class/', converted to graph
2019-03-06 17:57:57.162769: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-03-06 17:57:57.163297: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope 'mrcnn_class_bn1/', converted to graph
2019-03-06 17:57:57.163313: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-03-06 17:57:57.163840: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope 'mrcnn_class_bn2/', converted to graph
2019-03-06 17:57:57.163857: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-03-06 17:57:57.164419: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope 'mrcnn_class_conv1/', converted to graph
2019-03-06 17:57:57.164436: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-03-06 17:57:57.165035: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope 'mrcnn_class_conv2/', converted to graph
2019-03-06 17:57:57.165052: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-03-06 17:57:57.165561: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope 'mrcnn_class_logits/', converted to graph
2019-03-06 17:57:57.165577: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-03-06 17:57:57.166151: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope 'mrcnn_class_logits/', converted to graph
2019-03-06 17:57:57.166167: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-03-06 17:57:57.166775: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope '', converted to graph
2019-03-06 17:57:57.166792: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-03-06 17:57:57.167417: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope 'mrcnn_detection/', converted to graph
2019-03-06 17:57:57.167434: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-03-06 17:57:57.168032: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope '', converted to graph
2019-03-06 17:57:57.168049: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-03-06 17:57:57.168712: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope 'mrcnn_detection/', converted to graph
2019-03-06 17:57:57.168729: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-03-06 17:57:57.169406: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope '', converted to graph
2019-03-06 17:57:57.169423: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-03-06 17:57:57.170187: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope '', converted to graph
2019-03-06 17:57:57.170203: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-03-06 17:57:57.170825: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine my_trt_op_0 creation for segment 0, composed of 13 nodes succeeded.
2019-03-06 17:57:57.170896: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine ROI/my_trt_op_1 creation for segment 1, composed of 1 nodes succeeded.
2019-03-06 17:57:57.170935: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine activation_68/my_trt_op_2 creation for segment 2, composed of 1 nodes succeeded.
2019-03-06 17:57:57.170971: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine activation_69/my_trt_op_3 creation for segment 3, composed of 1 nodes succeeded.
2019-03-06 17:57:57.170999: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine mrcnn_bbox_fc/my_trt_op_4 creation for segment 4, composed of 1 nodes succeeded.
2019-03-06 17:57:57.171075: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine mrcnn_bbox_fc/my_trt_op_5 creation for segment 5, composed of 3 nodes succeeded.
2019-03-06 17:57:57.171122: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine mrcnn_class/my_trt_op_6 creation for segment 6, composed of 1 nodes succeeded.
2019-03-06 17:57:57.171188: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine mrcnn_class_bn1/my_trt_op_7 creation for segment 7, composed of 5 nodes succeeded.
2019-03-06 17:57:57.171248: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine mrcnn_class_bn2/my_trt_op_8 creation for segment 8, composed of 5 nodes succeeded.
2019-03-06 17:57:57.240756: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine mrcnn_class_conv1/my_trt_op_9 creation for segment 9, composed of 4 nodes succeeded.
2019-03-06 17:57:57.243605: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine mrcnn_class_conv2/my_trt_op_10 creation for segment 10, composed of 4 nodes succeeded.
2019-03-06 17:57:57.243680: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine mrcnn_class_logits/my_trt_op_11 creation for segment 11, composed of 1 nodes succeeded.
2019-03-06 17:57:57.243749: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine mrcnn_class_logits/my_trt_op_12 creation for segment 12, composed of 3 nodes succeeded.
2019-03-06 17:57:57.243835: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine my_trt_op_13 creation for segment 13, composed of 1 nodes succeeded.
2019-03-06 17:57:57.243881: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine mrcnn_detection/my_trt_op_14 creation for segment 14, composed of 1 nodes succeeded.
2019-03-06 17:57:57.243933: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine my_trt_op_15 creation for segment 15, composed of 1 nodes succeeded.
2019-03-06 17:57:57.243976: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine mrcnn_detection/my_trt_op_16 creation for segment 16, composed of 2 nodes succeeded.
2019-03-06 17:57:57.244041: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine my_trt_op_17 creation for segment 17, composed of 1 nodes succeeded.
2019-03-06 17:57:57.244089: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine my_trt_op_18 creation for segment 18, composed of 1 nodes succeeded.
2019-03-06 17:57:59.561151: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-03-06 17:57:59.563136: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-03-06 17:57:59.573212: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-03-06 17:57:59.573900: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-03-06 17:57:59.582574: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-03-06 17:57:59.583334: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-03-06 17:57:59.591727: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-03-06 17:57:59.592468: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-03-06 17:57:59.601501: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-03-06 17:57:59.602356: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-03-06 17:57:59.611415: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-03-06 17:57:59.612098: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-03-06 17:57:59.620349: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-03-06 17:57:59.621061: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-03-06 17:57:59.631260: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-03-06 17:57:59.631964: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-03-06 17:57:59.640055: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-03-06 17:57:59.640755: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-03-06 17:57:59.649689: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-03-06 17:57:59.650364: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-03-06 17:57:59.730773: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-03-06 17:57:59.766649: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-03-06 17:57:59.772251: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:501] Optimization results for grappler item: tf_graph
2019-03-06 17:57:59.772271: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 1658 nodes (10), 2221 edges (10), time = 328.819ms.
2019-03-06 17:57:59.772276: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] layout: Graph size after: 1702 nodes (44), 2292 edges (71), time = 131.199ms.
2019-03-06 17:57:59.772282: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 1671 nodes (-31), 2257 edges (-35), time = 2126.05688ms.
2019-03-06 17:57:59.772287: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 1671 nodes (0), 2257 edges (0), time = 76.47ms.
2019-03-06 17:57:59.772308: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 1671 nodes (0), 2257 edges (0), time = 2210.08398ms.
2019-03-06 17:57:59.772314: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:501] Optimization results for grappler item: mrcnn_class_conv2/my_trt_op_10_native_segment
2019-03-06 17:57:59.772333: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 5 nodes (0), 4 edges (0), time = 4.159ms.
2019-03-06 17:57:59.772339: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] layout: Graph size after: 5 nodes (0), 4 edges (0), time = 1.684ms.
2019-03-06 17:57:59.772344: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 5 nodes (0), 4 edges (0), time = 0.132ms.
2019-03-06 17:57:59.772350: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 5 nodes (0), 4 edges (0), time = 1.845ms.
2019-03-06 17:57:59.772356: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 5 nodes (0), 4 edges (0), time = 0.15ms.
2019-03-06 17:57:59.772378: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:501] Optimization results for grappler item: my_trt_op_17_native_segment
2019-03-06 17:57:59.772384: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 6 nodes (0), 5 edges (0), time = 2.687ms.
2019-03-06 17:57:59.772390: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] layout: Graph size after: 6 nodes (0), 5 edges (0), time = 0.439ms.
2019-03-06 17:57:59.772395: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 6 nodes (0), 5 edges (0), time = 0.107ms.
2019-03-06 17:57:59.772401: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 6 nodes (0), 5 edges (0), time = 0.573ms.
2019-03-06 17:57:59.772410: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 6 nodes (0), 5 edges (0), time = 0.107ms.
2019-03-06 17:57:59.772416: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:501] Optimization results for grappler item: mrcnn_class_logits/my_trt_op_12_native_segment
2019-03-06 17:57:59.772422: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 4 nodes (0), 3 edges (0), time = 2.566ms.
2019-03-06 17:57:59.772428: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] layout: Graph size after: 4 nodes (0), 3 edges (0), time = 0.428ms.
2019-03-06 17:57:59.772433: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 4 nodes (0), 3 edges (0), time = 0.104ms.
2019-03-06 17:57:59.772439: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 4 nodes (0), 3 edges (0), time = 0.645ms.
2019-03-06 17:57:59.772444: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 4 nodes (0), 3 edges (0), time = 0.111ms.
2019-03-06 17:57:59.772450: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:501] Optimization results for grappler item: mrcnn_bbox_fc/my_trt_op_5_native_segment
2019-03-06 17:57:59.772455: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 4 nodes (0), 3 edges (0), time = 2.546ms.
2019-03-06 17:57:59.772461: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] layout: Graph size after: 4 nodes (0), 3 edges (0), time = 0.446ms.
2019-03-06 17:57:59.772467: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 4 nodes (0), 3 edges (0), time = 0.114ms.
2019-03-06 17:57:59.772472: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 4 nodes (0), 3 edges (0), time = 0.623ms.
2019-03-06 17:57:59.772477: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 4 nodes (0), 3 edges (0), time = 0.104ms.
2019-03-06 17:57:59.772483: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:501] Optimization results for grappler item: my_trt_op_0_native_segment
2019-03-06 17:57:59.772489: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 18 nodes (0), 21 edges (0), time = 2.684ms.
2019-03-06 17:57:59.772496: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] layout: Graph size after: 18 nodes (0), 21 edges (0), time = 0.541ms.
2019-03-06 17:57:59.772501: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 18 nodes (0), 21 edges (0), time = 0.134ms.
2019-03-06 17:57:59.772507: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 18 nodes (0), 21 edges (0), time = 0.712ms.
2019-03-06 17:57:59.772513: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 18 nodes (0), 21 edges (0), time = 0.119ms.
2019-03-06 17:57:59.772518: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:501] Optimization results for grappler item: my_trt_op_15_native_segment
2019-03-06 17:57:59.772524: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 5 nodes (0), 4 edges (0), time = 2.444ms.
2019-03-06 17:57:59.772529: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] layout: Graph size after: 5 nodes (0), 4 edges (0), time = 0.45ms.
2019-03-06 17:57:59.772534: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 5 nodes (0), 4 edges (0), time = 0.116ms.
2019-03-06 17:57:59.772539: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 5 nodes (0), 4 edges (0), time = 0.562ms.
2019-03-06 17:57:59.772545: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 5 nodes (0), 4 edges (0), time = 0.112ms.
2019-03-06 17:57:59.772550: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:501] Optimization results for grappler item: mrcnn_class_bn1/my_trt_op_7_native_segment
2019-03-06 17:57:59.772555: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 6 nodes (0), 5 edges (0), time = 2.582ms.
2019-03-06 17:57:59.772561: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] layout: Graph size after: 6 nodes (0), 5 edges (0), time = 0.496ms.
2019-03-06 17:57:59.772567: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 6 nodes (0), 5 edges (0), time = 0.128ms.
2019-03-06 17:57:59.772573: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 6 nodes (0), 5 edges (0), time = 0.577ms.
2019-03-06 17:57:59.772578: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 6 nodes (0), 5 edges (0), time = 0.109ms.
2019-03-06 17:57:59.772583: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:501] Optimization results for grappler item: my_trt_op_13_native_segment
2019-03-06 17:57:59.772619: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 6 nodes (0), 5 edges (0), time = 2.598ms.
2019-03-06 17:57:59.772624: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] layout: Graph size after: 6 nodes (0), 5 edges (0), time = 0.455ms.
2019-03-06 17:57:59.772630: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 6 nodes (0), 5 edges (0), time = 0.119ms.
2019-03-06 17:57:59.772636: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 6 nodes (0), 5 edges (0), time = 0.58ms.
2019-03-06 17:57:59.772641: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 6 nodes (0), 5 edges (0), time = 0.113ms.
2019-03-06 17:57:59.772647: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:501] Optimization results for grappler item: mrcnn_class_bn2/my_trt_op_8_native_segment
2019-03-06 17:57:59.772652: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 6 nodes (0), 5 edges (0), time = 2.538ms.
2019-03-06 17:57:59.772658: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] layout: Graph size after: 6 nodes (0), 5 edges (0), time = 0.481ms.
2019-03-06 17:57:59.772663: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 6 nodes (0), 5 edges (0), time = 0.122ms.
2019-03-06 17:57:59.772668: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 6 nodes (0), 5 edges (0), time = 0.576ms.
2019-03-06 17:57:59.772674: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 6 nodes (0), 5 edges (0), time = 0.108ms.
2019-03-06 17:57:59.772679: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:501] Optimization results for grappler item: my_trt_op_18_native_segment
2019-03-06 17:57:59.772684: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 4 nodes (0), 3 edges (0), time = 2.589ms.
2019-03-06 17:57:59.772689: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] layout: Graph size after: 4 nodes (0), 3 edges (0), time = 0.417ms.
2019-03-06 17:57:59.772695: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 4 nodes (0), 3 edges (0), time = 0.117ms.
2019-03-06 17:57:59.772709: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 4 nodes (0), 3 edges (0), time = 0.561ms.
2019-03-06 17:57:59.772714: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 4 nodes (0), 3 edges (0), time = 0.108ms.
2019-03-06 17:57:59.772720: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:501] Optimization results for grappler item: mrcnn_class_conv1/my_trt_op_9_native_segment
2019-03-06 17:57:59.772725: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 5 nodes (0), 4 edges (0), time = 39.036ms.
2019-03-06 17:57:59.772731: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] layout: Graph size after: 5 nodes (0), 4 edges (0), time = 36.055ms.
2019-03-06 17:57:59.772736: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 5 nodes (0), 4 edges (0), time = 0.114ms.
2019-03-06 17:57:59.772742: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] constant folding: Graph size after: 5 nodes (0), 4 edges (0), time = 35.755ms.
2019-03-06 17:57:59.772747: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503] TensorRTOptimizer: Graph size after: 5 nodes (0), 4 edges (0), time = 0.133ms.
root@07db8026c5c0:/test#
import os
import tensorflow as tf
import tensorflow.contrib.tensorrt as trt
def GiB(val):
return val * 1 << 30
def get_frozen_graph(graph_file):
"""Read Frozen Graph file from disk."""
with tf.gfile.FastGFile(graph_file, "rb") as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
return graph_def
def main():
frozen_graph_def = get_frozen_graph('mask_rcnn_tf_model.pb')
output_names = ['output_detections']
precision_mode = "FP16"
trt_graph = trt.create_inference_graph(
frozen_graph_def, outputs=output_names,
precision_mode=precision_mode, maximum_cached_engines=20,
minimum_segment_size=1, max_workspace_size_bytes=GiB(4), is_dynamic_op=True
)
if __name__ == '__main__':
main()
That’s interesting. I’ve uploaded another pb file. Could you test this one as well. It use the same link: https://drive.google.com/file/d/1QkXxDAsieVD8YeWbFcCY38gj84Rin8pT/view?usp=sharing
the new pb has the same size as previous and also parses fine with both 19.01 and .02.
I just reread your stack trace. I must have described my problem incorrectly. When I run the trt graph with Tensorflow and it dynamically builds i get the error.
Like so:
with tf.Session(tf.import_graph_def(trt_graph)) as sess_trt:
sess_trt.run(None) # do something, it will fail on load
I’m getting the exact same error.I am loading my graph which was created by set is_dynamic_op=True then below error happens:
tensorflow.python.framework.errors_impl.InvalidArgumentError: Value for attr ‘OutT’ of int32 is not in the list of allowed values: int8, half, float