TensorRT: No conversion function for RealDiv

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hello, I am trying to convert ResNet18 preprocessing from TensorFlow to TensorRT (version 3.0.1-1+cuda8.0 running on Ubuntu 16.04 on an x86-64 machine):

#!/usr/bin/env python

import tensorflow as tf

with tf.Graph().as_default(), tf.Session() as session:
    inputs = tf.placeholder(shape=(None, 256, 256, 3), dtype=tf.float32, name='images')
    output = (inputs / 255.0 - [0.485, 0.456, 0.406]) / [0.229, 0.224, 0.225]
    output_node_names = [output.name[:-len(':0')]]
    graph_def = session.graph.as_graph_def()
    graph_def = tf.graph_util.convert_variables_to_constants(session, graph_def, output_node_names)

import uff
uff_model = uff.from_tensorflow(graph_def, output_node_names)

This fails with:

2018-01-11 13:41:10.015188: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2 FMA
2018-01-11 13:41:10.463045: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:83:00.0
totalMemory: 10.91GiB freeMemory: 10.75GiB
2018-01-11 13:41:10.463116: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:83:00.0, compute capability: 6.1)
Converted 0 variables to const ops.
Using output node div_1
Converting to UFF graph
Warning: No conversion function registered for layer: RealDiv yet.
Converting as custom op RealDiv div_1
name: "div_1"
op: "RealDiv"
input: "sub"
input: "div_1/y"
attr {
  key: "T"
  value {
    type: DT_FLOAT
  }
}

Traceback (most recent call last):
  File "./minimal_example.py", line 13, in <module>
    uff_model = uff.from_tensorflow(graph_def, output_node_names)
  File "/usr/lib/python2.7/dist-packages/uff/converters/tensorflow/conversion_helpers.py", line 75, in from_tensorflow
    name="main")
  File "/usr/lib/python2.7/dist-packages/uff/converters/tensorflow/converter.py", line 64, in convert_tf2uff_graph
    uff_graph, input_replacements)
  File "/usr/lib/python2.7/dist-packages/uff/converters/tensorflow/converter.py", line 51, in convert_tf2uff_node
    op, name, tf_node, inputs, uff_graph, tf_nodes=tf_nodes)
  File "/usr/lib/python2.7/dist-packages/uff/converters/tensorflow/converter.py", line 28, in convert_layer
    fields = cls.parse_tf_attrs(tf_node.attr)
  File "/usr/lib/python2.7/dist-packages/uff/converters/tensorflow/converter.py", line 177, in parse_tf_attrs
    for key, val in attrs.items()}
  File "/usr/lib/python2.7/dist-packages/uff/converters/tensorflow/converter.py", line 177, in <dictcomp>
    for key, val in attrs.items()}
  File "/usr/lib/python2.7/dist-packages/uff/converters/tensorflow/converter.py", line 172, in parse_tf_attr_value
    return cls.convert_tf2uff_field(code, val)
  File "/usr/lib/python2.7/dist-packages/uff/converters/tensorflow/converter.py", line 146, in convert_tf2uff_field
    return TensorFlowToUFFConverter.convert_tf2numpy_dtype(val)
  File "/usr/lib/python2.7/dist-packages/uff/converters/tensorflow/converter.py", line 74, in convert_tf2numpy_dtype
    return np.dtype(dt[dtype])
TypeError: list indices must be integers, not AttrValue

Do I understand correctly that division operation for floats is not supported by the converter?

Do I understand correctly that this operation is actually supported by TensorRT (the developer’s manual mentions division: http://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#tfops), and the problem is only in the TensorFlow to UFF converter?

2

realdiv is not supported but div is.

3

So, how do I implement this preprocessing in TensorFlow, so that it would convert to UFF successfully?

4

If one replaces divisions with multiplications,

#!/usr/bin/env python

import tensorflow as tf

with tf.Graph().as_default(), tf.Session() as session:
    inputs = tf.placeholder(shape=(None, 256, 256, 3), dtype=tf.float32, name='images')
    output = (inputs * (1 / 255.0) - [0.485, 0.456, 0.406]) * [1 / 0.229, 1 / 0.224, 1 / 0.225]
    output_node_names = [output.name[:-len(':0')]]
    graph_def = session.graph.as_graph_def()
    graph_def = tf.graph_util.convert_variables_to_constants(session, graph_def, output_node_names)

import uff
uff_model = uff.from_tensorflow(graph_def, output_node_names)

import tensorrt as trt
from tensorrt.parsers import uffparser

G_LOGGER = trt.infer.ConsoleLogger(trt.infer.LogSeverity.ERROR)

parser = uffparser.create_uff_parser()
parser.register_input(
    'images',
    (256, 256, 3),
    0 # I guess, means that the input of the model is in NHWC?
)
for output_node_name in output_node_names:
    parser.register_output(output_node_name)

engine = trt.utils.uff_to_trt_engine(
    logger=G_LOGGER,
    stream=uff_model,
    parser=parser,
    max_batch_size=4,
    max_workspace_size=1 << 20 # What is a workspace?
)

, the creation of the TensorRT engine fails with

2018-01-15 16:37:26.737175: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2 FMA
2018-01-15 16:37:27.173673: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:83:00.0
totalMemory: 10.91GiB freeMemory: 10.75GiB
2018-01-15 16:37:27.173729: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:83:00.0, compute capability: 6.1)
Converted 0 variables to const ops.
Using output node mul_1
Converting to UFF graph
No. nodes: 8
[TensorRT] ERROR: UFFParser: Parser error: sub: Invalid scale mode, nbWeights: 3
[TensorRT] ERROR: Failed to parse UFF model stream
  File "/usr/lib/python2.7/dist-packages/tensorrt/utils/_utils.py", line 186, in uff_to_trt_engine
    assert(parser_result)
Traceback (most recent call last):
  File "./minimal_example2.py", line 34, in <module>
    max_workspace_size=1 << 20 # What is a workspace?
  File "/usr/lib/python2.7/dist-packages/tensorrt/utils/_utils.py", line 194, in uff_to_trt_engine
    raise AssertionError('UFF parsing failed on line {} in statement {}'.format(line, text))
AssertionError: UFF parsing failed on line 186 in statement assert(parser_result)

Do I understand correctly that ResNet preprocessing is not implementable in TensorFlow so that it could be converted to UFF and then to TensorRT engine?

5

Did you find any workaround for this ?

6

I have decided to switch to tensorflow.contrib.tensorrt in the end.
It optimizes what it can using TensorRT and leaves the computation of the rest to TensorFlow.
This approach seems to work, although the memory consumption did grow by almost a factor of two in my case (comparing to a graph evaluation purely by TensorFlow).
However, the memory consumption might be improved after TensorRT 4 gets released: TensorRT: Large memory consumption on SSD-like graphs [Feature Request/Discussion] · Issue #19619 · tensorflow/tensorflow · GitHub