TF-TRT INT8 Failing to convert due to no calibration

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hello I am trying to quantize a Tensorflow pb file to INT8. However I get the following error:

Check failed: tensorflow.python.framework.errors_impl.FailedPreconditionError: Need to run graph with calibration data first!

I am running the calibration process but it does not seem to work. I am using the tensorflow:19.01-py3 docker image provided by Nvidia. I have also tried the tensorflow nightly build image, but that produces the same error. This PB sucessfully converts to FP16, just not INT8. My hardware is a V100 on AWS.

A snippet of code is as follows:

def calibrate_graph(calib_graph, num_calib_images, config):
    with tf.Session(tf.import_graph_def(calib_graph)) as sess:

        sess.run(tf.global_variables_initializer()) # this was needed otherwise odd error was thrown

        tf.summary.FileWriter('/project/output/calibration/', graph=sess.graph)

        # get layers
        img_input_tensor = sess.graph.get_tensor_by_name('input_image:0')
        img_anchors_input_tensor = sess.graph.get_tensor_by_name('input_anchors:0')
        img_meta_input_tensor = sess.graph.get_tensor_by_name('input_image_meta:0')
        detections_input_tensor = sess.graph.get_tensor_by_name('output_detections:0')

        # dummy input data
        molded_image = [np.zeros((1024, 1024, 3))]
        image_meta = [np.array(
            [0] +
            [1024, 1024, 3] +
            [1024, 1024, 3] +
            [0, 0, 1024, 1024] +
            [500] +
            [0, 1, 2, 3, 4, 5]
        )]
        image_anchor = [[[1, 1, 1, 1]]]

        # run net once to dummy calibrate
        detections = sess.run([detections_input_tensor],
                     feed_dict={img_input_tensor: molded_image,
                                img_meta_input_tensor: image_meta,
                                img_anchors_input_tensor: image_anchor}
                                  )

    trt_graph = trt.calib_graph_to_infer_graph(calib_graph, is_dynamic_op=True)

    return trt_graph

precision_mode = "INT8"

trt_graph = trt.create_inference_graph(
    frozen_graph_pb, outputs=output_names, max_batch_size=10000,
    precision_mode=precision_mode, maximum_cached_engines=20,
    minimum_segment_size=3, max_workspace_size_bytes=GiB(4), is_dynamic_op=True
    )

if precision_mode is "INT8":
    # TF-TRT calibration
    trt_graph = calibrate_graph(trt_graph, 10, export_config)

The batch size is currently 1.

2

Full Stack Trace:

================
== TensorFlow ==
================

NVIDIA Release 19.01 (build 5238117)
TensorFlow Version 1.12.0+

Container image Copyright (c) 2018, NVIDIA CORPORATION.  All rights reserved.
Copyright 2017-2018 The TensorFlow Authors.  All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION.  All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying project or file.

NOTE: MOFED driver for multi-node communication was not detected.
      Multi-node communication performance may be reduced.

NOTE: The SHMEM allocation limit is set to the default of 64MB.  This may be
   insufficient for TensorFlow.  NVIDIA recommends the use of the following flags:
   nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 ...

Using TensorFlow backend.
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/sparse_ops.py:1165: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.
Loading weights from  /project/data/mask_rcnn_coco.h5
2019-02-21 19:04:29.096303: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:957] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-02-21 19:04:29.096799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: 
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:00:1e.0
totalMemory: 15.75GiB freeMemory: 15.34GiB
2019-02-21 19:04:29.096832: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-02-21 19:04:29.674542: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-21 19:04:29.674598: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 
2019-02-21 19:04:29.674618: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N 
2019-02-21 19:04:29.674921: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14813 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2019-02-21 19:04:34.688438: I tensorflow/tools/graph_transforms/transform_graph.cc:317] Applying remove_nodes
2019-02-21 19:04:34.878530: I tensorflow/tools/graph_transforms/remove_nodes.cc:101] Skipping replacement for output_detections
2019-02-21 19:04:34.915015: I tensorflow/tools/graph_transforms/remove_nodes.cc:101] Skipping replacement for output_detections
2019-02-21 19:04:34.979162: I tensorflow/tools/graph_transforms/transform_graph.cc:317] Applying merge_duplicate_nodes
2019-02-21 19:04:36.083750: I tensorflow/tools/graph_transforms/transform_graph.cc:317] Applying strip_unused_nodes
2019-02-21 19:04:36.108374: I tensorflow/tools/graph_transforms/transform_graph.cc:317] Applying sort_by_execution_order
2019-02-21 19:04:37.671266: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-02-21 19:04:37.671335: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-21 19:04:37.671352: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 
2019-02-21 19:04:37.671370: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N 
2019-02-21 19:04:37.671504: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14813 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2019-02-21 19:04:39.800550: I tensorflow/core/grappler/devices.cc:51] Number of eligible GPUs (core count >= 8): 1
2019-02-21 19:04:39.803830: I tensorflow/core/grappler/clusters/single_machine.cc:359] Starting new session
2019-02-21 19:04:39.804305: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-02-21 19:04:39.804350: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-21 19:04:39.804367: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 
2019-02-21 19:04:39.804381: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N 
2019-02-21 19:04:39.804555: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14813 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2019-02-21 19:04:44.111935: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:868] MULTIPLE tensorrt candidate conversion: 7
2019-02-21 19:04:44.113342: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope '', converted to graph
2019-02-21 19:04:44.113369: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-02-21 19:04:44.119276: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope 'mrcnn_bbox_fc/', converted to graph
2019-02-21 19:04:44.119306: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-02-21 19:04:44.120517: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope 'mrcnn_class_bn1/', converted to graph
2019-02-21 19:04:44.120547: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-02-21 19:04:44.121234: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope 'mrcnn_class_bn2/', converted to graph
2019-02-21 19:04:44.121264: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-02-21 19:04:44.121977: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope 'mrcnn_class_conv1/', converted to graph
2019-02-21 19:04:44.122005: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-02-21 19:04:44.122749: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope 'mrcnn_class_conv2/', converted to graph
2019-02-21 19:04:44.122777: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-02-21 19:04:44.123587: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:3058] Segment @scope 'mrcnn_class_logits/', converted to graph
2019-02-21 19:04:44.123616: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:433] Can't find a device placement for the op!
2019-02-21 19:04:45.718920: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine my_trt_op_0 creation for segment 0, composed of 13 nodes succeeded.
2019-02-21 19:04:45.748861: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine mrcnn_bbox_fc/my_trt_op_1 creation for segment 1, composed of 3 nodes succeeded.
2019-02-21 19:04:45.758227: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine mrcnn_class_bn1/my_trt_op_2 creation for segment 2, composed of 5 nodes succeeded.
2019-02-21 19:04:45.767588: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine mrcnn_class_bn2/my_trt_op_3 creation for segment 3, composed of 5 nodes succeeded.
2019-02-21 19:04:48.736998: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine mrcnn_class_conv1/my_trt_op_4 creation for segment 4, composed of 4 nodes succeeded.
2019-02-21 19:04:49.039008: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine mrcnn_class_conv2/my_trt_op_5 creation for segment 5, composed of 4 nodes succeeded.
2019-02-21 19:04:49.069280: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:967] Engine mrcnn_class_logits/my_trt_op_6 creation for segment 6, composed of 3 nodes succeeded.
2019-02-21 19:04:52.594103: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-02-21 19:04:52.595202: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-02-21 19:04:52.598171: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-02-21 19:04:52.599088: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-02-21 19:04:52.601973: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-02-21 19:04:52.602916: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-02-21 19:04:52.678802: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-02-21 19:04:52.716191: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-02-21 19:04:52.719048: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-02-21 19:04:52.719936: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-02-21 19:04:52.723917: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-02-21 19:04:52.725680: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-02-21 19:04:52.728663: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-02-21 19:04:52.729597: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-02-21 19:04:52.730983: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:501] Optimization results for grappler item: tf_graph
2019-02-21 19:04:52.731017: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   constant folding: Graph size after: 1658 nodes (10), 2221 edges (10), time = 325.484ms.
2019-02-21 19:04:52.731043: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   layout: Graph size after: 1702 nodes (44), 2292 edges (71), time = 141.584ms.
2019-02-21 19:04:52.731119: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   TensorRTOptimizer: Graph size after: 1672 nodes (-30), 2258 edges (-34), time = 8155.52783ms.
2019-02-21 19:04:52.731138: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   constant folding: Graph size after: 1672 nodes (0), 2258 edges (0), time = 82.431ms.
2019-02-21 19:04:52.731154: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   TensorRTOptimizer: Graph size after: 1672 nodes (0), 2258 edges (0), time = 3417.87109ms.
2019-02-21 19:04:52.731176: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:501] Optimization results for grappler item: my_trt_op_0_native_segment
2019-02-21 19:04:52.731198: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   constant folding: Graph size after: 18 nodes (0), 21 edges (0), time = 1.187ms.
2019-02-21 19:04:52.731215: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   layout: Graph size after: 18 nodes (0), 21 edges (0), time = 0.655ms.
2019-02-21 19:04:52.731228: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   TensorRTOptimizer: Graph size after: 18 nodes (0), 21 edges (0), time = 0.193ms.
2019-02-21 19:04:52.731243: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   constant folding: Graph size after: 18 nodes (0), 21 edges (0), time = 0.896ms.
2019-02-21 19:04:52.731263: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   TensorRTOptimizer: Graph size after: 18 nodes (0), 21 edges (0), time = 0.17ms.
2019-02-21 19:04:52.731288: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:501] Optimization results for grappler item: mrcnn_bbox_fc/my_trt_op_1_native_segment
2019-02-21 19:04:52.731310: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   constant folding: Graph size after: 4 nodes (0), 3 edges (0), time = 1.116ms.
2019-02-21 19:04:52.731332: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   layout: Graph size after: 4 nodes (0), 3 edges (0), time = 0.481ms.
2019-02-21 19:04:52.731348: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   TensorRTOptimizer: Graph size after: 4 nodes (0), 3 edges (0), time = 0.155ms.
2019-02-21 19:04:52.731369: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   constant folding: Graph size after: 4 nodes (0), 3 edges (0), time = 0.748ms.
2019-02-21 19:04:52.731388: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   TensorRTOptimizer: Graph size after: 4 nodes (0), 3 edges (0), time = 0.155ms.
2019-02-21 19:04:52.731407: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:501] Optimization results for grappler item: mrcnn_class_bn1/my_trt_op_2_native_segment
2019-02-21 19:04:52.731427: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   constant folding: Graph size after: 6 nodes (0), 5 edges (0), time = 1.006ms.
2019-02-21 19:04:52.731443: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   layout: Graph size after: 6 nodes (0), 5 edges (0), time = 0.548ms.
2019-02-21 19:04:52.731461: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   TensorRTOptimizer: Graph size after: 6 nodes (0), 5 edges (0), time = 0.169ms.
2019-02-21 19:04:52.731482: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   constant folding: Graph size after: 6 nodes (0), 5 edges (0), time = 0.76ms.
2019-02-21 19:04:52.731501: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   TensorRTOptimizer: Graph size after: 6 nodes (0), 5 edges (0), time = 0.173ms.
2019-02-21 19:04:52.731523: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:501] Optimization results for grappler item: mrcnn_class_conv1/my_trt_op_4_native_segment
2019-02-21 19:04:52.731545: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   constant folding: Graph size after: 5 nodes (0), 4 edges (0), time = 37.795ms.
2019-02-21 19:04:52.731562: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   layout: Graph size after: 5 nodes (0), 4 edges (0), time = 36.76ms.
2019-02-21 19:04:52.731575: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   TensorRTOptimizer: Graph size after: 5 nodes (0), 4 edges (0), time = 0.206ms.
2019-02-21 19:04:52.731597: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   constant folding: Graph size after: 5 nodes (0), 4 edges (0), time = 37.16ms.
2019-02-21 19:04:52.731619: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   TensorRTOptimizer: Graph size after: 5 nodes (0), 4 edges (0), time = 0.201ms.
2019-02-21 19:04:52.731641: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:501] Optimization results for grappler item: mrcnn_class_logits/my_trt_op_6_native_segment
2019-02-21 19:04:52.731659: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   constant folding: Graph size after: 4 nodes (0), 3 edges (0), time = 0.985ms.
2019-02-21 19:04:52.731680: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   layout: Graph size after: 4 nodes (0), 3 edges (0), time = 0.462ms.
2019-02-21 19:04:52.731702: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   TensorRTOptimizer: Graph size after: 4 nodes (0), 3 edges (0), time = 0.152ms.
2019-02-21 19:04:52.731722: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   constant folding: Graph size after: 4 nodes (0), 3 edges (0), time = 0.723ms.
2019-02-21 19:04:52.731746: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   TensorRTOptimizer: Graph size after: 4 nodes (0), 3 edges (0), time = 0.148ms.
2019-02-21 19:04:52.731761: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:501] Optimization results for grappler item: mrcnn_class_conv2/my_trt_op_5_native_segment
2019-02-21 19:04:52.731782: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   constant folding: Graph size after: 5 nodes (0), 4 edges (0), time = 1.92ms.
2019-02-21 19:04:52.731804: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   layout: Graph size after: 5 nodes (0), 4 edges (0), time = 0.83ms.
2019-02-21 19:04:52.731825: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   TensorRTOptimizer: Graph size after: 5 nodes (0), 4 edges (0), time = 0.168ms.
2019-02-21 19:04:52.731847: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   constant folding: Graph size after: 5 nodes (0), 4 edges (0), time = 1.58ms.
2019-02-21 19:04:52.731861: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   TensorRTOptimizer: Graph size after: 5 nodes (0), 4 edges (0), time = 0.156ms.
2019-02-21 19:04:52.731873: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:501] Optimization results for grappler item: mrcnn_class_bn2/my_trt_op_3_native_segment
2019-02-21 19:04:52.731885: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   constant folding: Graph size after: 6 nodes (0), 5 edges (0), time = 1.039ms.
2019-02-21 19:04:52.731968: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   layout: Graph size after: 6 nodes (0), 5 edges (0), time = 0.552ms.
2019-02-21 19:04:52.731976: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   TensorRTOptimizer: Graph size after: 6 nodes (0), 5 edges (0), time = 0.201ms.
2019-02-21 19:04:52.731994: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   constant folding: Graph size after: 6 nodes (0), 5 edges (0), time = 0.749ms.
2019-02-21 19:04:52.732006: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:503]   TensorRTOptimizer: Graph size after: 6 nodes (0), 5 edges (0), time = 0.186ms.
2019-02-21 19:04:54.492720: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-02-21 19:04:54.492791: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-21 19:04:54.492812: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 
2019-02-21 19:04:54.492832: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N 
2019-02-21 19:04:54.492983: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14813 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2019-02-21 19:05:06.692181: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:155] Starting Calib Conversion
2019-02-21 19:05:06.781052: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:172] Could not get Calibration information. Did you run with calibration data?
Traceback (most recent call last):
  File "/project/export/export.py", line 300, in <module>
    trt_graph = calibrate_graph(trt_graph, 10, export_config)
  File "/project/export/export.py", line 187, in calibrate_graph
    trt_graph = trt.calib_graph_to_infer_graph(calib_graph, is_dynamic_op=True)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/tensorrt/python/trt_convert.py", line 370, in calib_graph_to_infer_graph
    int(msg[0]))
tensorflow.python.framework.errors_impl.FailedPreconditionError: Need to run graph with calibration data first!
3

Hello,
Can you give me the permission to the google drive so we can look into the case?

4

Thank you for looking into it; I’ve given you permission to access the file