Tensorflow RetinaNet Object Detector on Deepstream

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

I have a trained RetinaNet Object Detector network that I have been using for some time with good success. It was trained using Keras-defined models on Tensorflow, so I have the trained network as a frozen Tensorflow pb file, or as a Keras h5 file.

I recently purchased a Jetson Xavier AGX and want to try to run this network with Deepstream. I have the Jetson up and running and have run the samples successfully.

I’ve tried to convert using to TensorRT using the code below. It seems to run through without errors, but when I try to use the model, the results don’t seem right. With that said, I want to first make sure that I’m going through the correct process, so here are my questions:

  1. Is converting my Tensorflow pb -OR- Keras h5 to TensorRT the correct 1st step in preparing my model to run on Deepstream? If not, what is?

  2. Is there a chance that my RetinaNet model cannot be converted to TensorRT? If so, why? Layers not supported?

  3. Is it possible that the Tensorflow->TRT conversion will change the way data is output from the model? I ask this because I’m having trouble interpreting the data I’m getting…just looks like the network is not working.

Any suggestions would be greatly appreciated.

Bryan

import tensorflow.contrib.tensorrt as trt
import tensorflow as tf
from tensorflow.python.framework import graph_io
from keras_retinanet.models import load_model
from keras import backend as K
from keras_retinanet import models

#
# Input_Nodes=['input_1']
# Output_Nodes = ['filtered_detections/map/TensorArrayStack/TensorArrayGatherV3',
#                 'filtered_detections/map/TensorArrayStack_1/TensorArrayGatherV3',
#                 'filtered_detections/map/TensorArrayStack_2/TensorArrayGatherV3']

Keras_h5_File = '/home/bryan/training/RetinaNet/face/snapshots/run1_624.h5'
Output_Directory = '/home/bryan/training/RetinaNet/face/snapshots/'
Output_File = 'trt_face.pb'

save_pb_dir = Output_Directory
model_fname = Keras_h5_File

def freeze_graph(graph, session, output, save_pb_dir='.', save_pb_name='frozen_model.pb', save_pb_as_text=False):
    with graph.as_default():
        graphdef_inf = tf.graph_util.remove_training_nodes(graph.as_graph_def())
        graphdef_frozen = tf.graph_util.convert_variables_to_constants(session, graphdef_inf, output)
        graph_io.write_graph(graphdef_frozen, save_pb_dir, save_pb_name, as_text=save_pb_as_text)
        return graphdef_frozen


# Clear any previous session.
tf.keras.backend.clear_session()


with tf.Session() as sess:

    # This line must be executed before loading Keras model.
    tf.keras.backend.set_learning_phase(0)

    #load keras h5 model (the file created using keras to train it
    #provided by Keras Retinanet Implementation)
    model = load_model(Keras_h5_File, backbone_name='resnet50')

    # convert the h5 file from a training file to an inference file
    #provided by Keras Retinanet Implementation)
    model = models.convert_model(model)

    tf.initialize_all_variables().run()

    input_names = [t.op.name for t in model.inputs]
    output_names = [t.op.name for t in model.outputs]

    # Prints input and output nodes names, just making sure they're what I expect
    print(input_names, output_names)

    frozen_graph = freeze_graph(K.get_session().graph, sess, [out.op.name for out in model.outputs], save_pb_dir=save_pb_dir)

    trt_graph = trt.create_inference_graph(
        input_graph_def=frozen_graph,
        outputs=output_names,
        max_batch_size=1,
        max_workspace_size_bytes=1 << 25,
        precision_mode='FP32',
        minimum_segment_size=50
    )

    graph_io.write_graph(trt_graph, Output_Directory, Output_File, as_text=False)
2

Hi,

May I have the detail information about the script you used for conversion?
The source you shared looks like a TF-TRT code rather than an uff-based pure TensorRT code.

RetinaNet should work correctly with pure TensorRT since we do have a sample for it.
https://github.com/NVIDIA/retinanet-examples

My suggestion is try to follow the steps shared above.
And please let us know if you meet any issue or error with the steps.

Thanks

3

Hi,

Thanks for your quick answer. I had previously looked at the /NVIDIA/retinanet-examples for ideas, but I can’t find any help there in converting my model to TensorRT. The example deals with a Pytorch implementation/model, and I have a Tensorflow/Keras implementation.

The /NVIDIA/retinanet-examples/extras/cppapi/export.cpp code shows how to convert an ONNX model to a TRT model. Outside of that, I don’t see anything that might help me convert either my Keras h5 or Tensorflow frozen pb file. Am I going to have to find a way to convert one of them to ONNX first?

I’m obviously new to this, so could be overlooking the obvious.

Thanks

4

So my next attempt consisted of using the convert-to-uff utility on my frozen model pb file, which resulted in a KeyError 20. From some posts I found, it appears that this indicates that there are layers in my model that the convert-to-uff utility can’t handle. (See output below)

I’m guessing where to go from here…I think my options are:

1 - try to follow the example you cited above (GitHub - NVIDIA/retinanet-examples: Fast and accurate object detection with end-to-end GPU optimization) where they basically have this workflow: Pytorch → ONNX → TensorRT. I’m guessing that I may have the same issues of layer compatibility as my workflow will be: Tensorflow → ONNX → TensorRT.

2 - try to convert my model to TF_TRT and figure out a way to run that in Deepstream. I understand that TF_TRT models are not currently supported in Deepstream.

3 - retrain my object detector using a different architecture (YoloV3??) that is compatible with the convert-to-uff converter. This is my least favorite option.

4 - ??? something I missed?

Any suggestions would be greatly appreciated.

$convert-to-uff frozen_model.pb 
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/uff/converters/tensorflow/conversion_helpers.py:18: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.

Loading frozen_model.pb
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/uff/converters/tensorflow/conversion_helpers.py:227: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/uff/bin/../../graphsurgeon/_utils.py:2: The name tf.NodeDef is deprecated. Please use tf.compat.v1.NodeDef instead.

UFF Version 0.6.5
=== Automatically deduced input nodes ===
[name: "input_1"
op: "Placeholder"
attr {
  key: "dtype"
  value {
    type: DT_FLOAT
  }
}
attr {
  key: "shape"
  value {
    shape {
      dim {
        size: -1
      }
      dim {
        size: -1
      }
      dim {
        size: -1
      }
      dim {
        size: 3
      }
    }
  }
}
]
=========================================

=== Automatically deduced output nodes ===
[name: "filtered_detections/map/TensorArrayStack/TensorArrayGatherV3"
op: "TensorArrayGatherV3"
input: "filtered_detections/map/TensorArray_2"
input: "filtered_detections/map/TensorArrayStack/range"
input: "filtered_detections/map/while/Exit_2"
attr {
  key: "_class"
  value {
    list {
      s: "loc:@filtered_detections/map/TensorArray_2"
    }
  }
}
attr {
  key: "dtype"
  value {
    type: DT_FLOAT
  }
}
attr {
  key: "element_shape"
  value {
    shape {
      dim {
        size: 300
      }
      dim {
        size: 4
      }
    }
  }
}
, name: "filtered_detections/map/TensorArrayStack_1/TensorArrayGatherV3"
op: "TensorArrayGatherV3"
input: "filtered_detections/map/TensorArray_3"
input: "filtered_detections/map/TensorArrayStack_1/range"
input: "filtered_detections/map/while/Exit_3"
attr {
  key: "_class"
  value {
    list {
      s: "loc:@filtered_detections/map/TensorArray_3"
    }
  }
}
attr {
  key: "dtype"
  value {
    type: DT_FLOAT
  }
}
attr {
  key: "element_shape"
  value {
    shape {
      dim {
        size: 300
      }
    }
  }
}
, name: "filtered_detections/map/TensorArrayStack_2/TensorArrayGatherV3"
op: "TensorArrayGatherV3"
input: "filtered_detections/map/TensorArray_4"
input: "filtered_detections/map/TensorArrayStack_2/range"
input: "filtered_detections/map/while/Exit_4"
attr {
  key: "_class"
  value {
    list {
      s: "loc:@filtered_detections/map/TensorArray_4"
    }
  }
}
attr {
  key: "dtype"
  value {
    type: DT_INT32
  }
}
attr {
  key: "element_shape"
  value {
    shape {
      dim {
        size: 300
      }
    }
  }
}
]
==========================================

Using output node filtered_detections/map/TensorArrayStack/TensorArrayGatherV3
Using output node filtered_detections/map/TensorArrayStack_1/TensorArrayGatherV3
Using output node filtered_detections/map/TensorArrayStack_2/TensorArrayGatherV3
Converting to UFF graph
Warning: No conversion function registered for layer: TensorArrayGatherV3 yet.
Converting filtered_detections/map/TensorArrayStack_2/TensorArrayGatherV3 as custom op: TensorArrayGatherV3
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py:179: The name tf.AttrValue is deprecated. Please use tf.compat.v1.AttrValue instead.

Warning: No conversion function registered for layer: Exit yet.
Converting filtered_detections/map/while/Exit_4 as custom op: Exit
Warning: No conversion function registered for layer: Switch yet.
Converting filtered_detections/map/while/Switch_4 as custom op: Switch
Warning: No conversion function registered for layer: LoopCond yet.
Converting filtered_detections/map/while/LoopCond as custom op: LoopCond
Warning: No conversion function registered for layer: LogicalAnd yet.
Converting filtered_detections/map/while/LogicalAnd as custom op: LogicalAnd
Warning: No conversion function registered for layer: Less yet.
Converting filtered_detections/map/while/Less_1 as custom op: Less
Warning: No conversion function registered for layer: Enter yet.
Converting filtered_detections/map/while/Less/Enter as custom op: Enter
Warning: No conversion function registered for layer: Cast yet.
Converting clipped_boxes/Cast as custom op: Cast
Warning: No conversion function registered for layer: Tile yet.
Converting anchors_4/Tile as custom op: Tile
Warning: No conversion function registered for layer: Fill yet.
Converting anchors_4/meshgrid/ones as custom op: Fill
Warning: No conversion function registered for layer: Size yet.
Converting anchors_4/meshgrid/Size as custom op: Size
Warning: No conversion function registered for layer: Cast yet.
Converting anchors_4/Cast as custom op: Cast
Warning: No conversion function registered for layer: Range yet.
Converting anchors_4/arange as custom op: Range
Warning: No conversion function registered for layer: Size yet.
Converting anchors_4/meshgrid/Size_1 as custom op: Size
Warning: No conversion function registered for layer: Cast yet.
Converting anchors_4/Cast_1 as custom op: Cast
Warning: No conversion function registered for layer: Range yet.
Converting anchors_4/arange_1 as custom op: Range
Warning: No conversion function registered for layer: Tile yet.
Converting anchors_3/Tile as custom op: Tile
Warning: No conversion function registered for layer: Fill yet.
Converting anchors_3/meshgrid/ones as custom op: Fill
Warning: No conversion function registered for layer: Size yet.
Converting anchors_3/meshgrid/Size as custom op: Size
Warning: No conversion function registered for layer: Cast yet.
Converting anchors_3/Cast as custom op: Cast
Warning: No conversion function registered for layer: Range yet.
Converting anchors_3/arange as custom op: Range
Warning: No conversion function registered for layer: Size yet.
Converting anchors_3/meshgrid/Size_1 as custom op: Size
Warning: No conversion function registered for layer: Cast yet.
Converting anchors_3/Cast_1 as custom op: Cast
Warning: No conversion function registered for layer: Range yet.
Converting anchors_3/arange_1 as custom op: Range
Warning: No conversion function registered for layer: Tile yet.
Converting anchors_2/Tile as custom op: Tile
Warning: No conversion function registered for layer: Fill yet.
Converting anchors_2/meshgrid/ones as custom op: Fill
Warning: No conversion function registered for layer: Size yet.
Converting anchors_2/meshgrid/Size as custom op: Size
Warning: No conversion function registered for layer: Cast yet.
Converting anchors_2/Cast as custom op: Cast
Warning: No conversion function registered for layer: Range yet.
Converting anchors_2/arange as custom op: Range
Warning: No conversion function registered for layer: Size yet.
Converting anchors_2/meshgrid/Size_1 as custom op: Size
Warning: No conversion function registered for layer: Cast yet.
Converting anchors_2/Cast_1 as custom op: Cast
Warning: No conversion function registered for layer: Range yet.
Converting anchors_2/arange_1 as custom op: Range
Warning: No conversion function registered for layer: Tile yet.
Converting anchors_1/Tile as custom op: Tile
Warning: No conversion function registered for layer: ResizeNearestNeighbor yet.
Converting P5_upsampled/resize/ResizeNearestNeighbor as custom op: ResizeNearestNeighbor
Warning: No conversion function registered for layer: Fill yet.
Converting anchors_1/meshgrid/ones as custom op: Fill
Warning: No conversion function registered for layer: Size yet.
Converting anchors_1/meshgrid/Size as custom op: Size
Warning: No conversion function registered for layer: Cast yet.
Converting anchors_1/Cast as custom op: Cast
Warning: No conversion function registered for layer: Range yet.
Converting anchors_1/arange as custom op: Range
Warning: No conversion function registered for layer: Size yet.
Converting anchors_1/meshgrid/Size_1 as custom op: Size
Warning: No conversion function registered for layer: Cast yet.
Converting anchors_1/Cast_1 as custom op: Cast
Warning: No conversion function registered for layer: Range yet.
Converting anchors_1/arange_1 as custom op: Range
Warning: No conversion function registered for layer: Tile yet.
Converting anchors_0/Tile as custom op: Tile
Warning: No conversion function registered for layer: ResizeNearestNeighbor yet.
Converting P4_upsampled/resize/ResizeNearestNeighbor as custom op: ResizeNearestNeighbor
Warning: No conversion function registered for layer: Fill yet.
Converting anchors_0/meshgrid/ones as custom op: Fill
Warning: No conversion function registered for layer: Size yet.
Converting anchors_0/meshgrid/Size as custom op: Size
Warning: No conversion function registered for layer: Cast yet.
Converting anchors_0/Cast as custom op: Cast
Warning: No conversion function registered for layer: Range yet.
Converting anchors_0/arange as custom op: Range
Warning: No conversion function registered for layer: Size yet.
Converting anchors_0/meshgrid/Size_1 as custom op: Size
Warning: No conversion function registered for layer: Cast yet.
Converting anchors_0/Cast_1 as custom op: Cast
Warning: No conversion function registered for layer: Range yet.
Converting anchors_0/arange_1 as custom op: Range
Warning: No conversion function registered for layer: Merge yet.
Converting filtered_detections/map/while/Merge_1 as custom op: Merge
Warning: No conversion function registered for layer: NextIteration yet.
Converting filtered_detections/map/while/NextIteration_1 as custom op: NextIteration
Warning: No conversion function registered for layer: Switch yet.
Converting filtered_detections/map/while/Switch as custom op: Switch
Warning: No conversion function registered for layer: Merge yet.
Converting filtered_detections/map/while/Merge as custom op: Merge
Warning: No conversion function registered for layer: NextIteration yet.
Converting filtered_detections/map/while/NextIteration as custom op: NextIteration
Warning: No conversion function registered for layer: Enter yet.
Converting filtered_detections/map/while/Enter as custom op: Enter
Warning: No conversion function registered for layer: Switch yet.
Converting filtered_detections/map/while/Switch_1 as custom op: Switch
Warning: No conversion function registered for layer: Enter yet.
Converting filtered_detections/map/while/Enter_1 as custom op: Enter
Warning: No conversion function registered for layer: Less yet.
Converting filtered_detections/map/while/Less as custom op: Less
Warning: No conversion function registered for layer: Merge yet.
Converting filtered_detections/map/while/Merge_4 as custom op: Merge
Warning: No conversion function registered for layer: NextIteration yet.
Converting filtered_detections/map/while/NextIteration_4 as custom op: NextIteration
Warning: No conversion function registered for layer: TensorArrayWriteV3 yet.
Converting filtered_detections/map/while/TensorArrayWrite_2/TensorArrayWriteV3 as custom op: TensorArrayWriteV3
Warning: No conversion function registered for layer: Cast yet.
Converting filtered_detections/map/while/Cast as custom op: Cast
Warning: No conversion function registered for layer: PadV2 yet.
Converting filtered_detections/map/while/PadV2_2 as custom op: PadV2
Warning: No conversion function registered for layer: TopKV2 yet.
Converting filtered_detections/map/while/TopKV2 as custom op: TopKV2
Warning: No conversion function registered for layer: GatherNd yet.
Converting filtered_detections/map/while/GatherNd_2 as custom op: GatherNd
Warning: No conversion function registered for layer: GatherNd yet.
Converting filtered_detections/map/while/GatherNd_1 as custom op: GatherNd
Warning: No conversion function registered for layer: NonMaxSuppressionV3 yet.
Converting filtered_detections/map/while/non_max_suppression/NonMaxSuppressionV3 as custom op: NonMaxSuppressionV3
Warning: No conversion function registered for layer: Where yet.
Converting filtered_detections/map/while/Where as custom op: Where
Warning: No conversion function registered for layer: Greater yet.
Converting filtered_detections/map/while/Greater as custom op: Greater
Warning: No conversion function registered for layer: TensorArrayReadV3 yet.
Converting filtered_detections/map/while/TensorArrayReadV3_1 as custom op: TensorArrayReadV3
Warning: No conversion function registered for layer: Enter yet.
Converting filtered_detections/map/while/TensorArrayReadV3_1/Enter_1 as custom op: Enter
Warning: No conversion function registered for layer: TensorArrayScatterV3 yet.
Converting filtered_detections/map/TensorArrayUnstack_1/TensorArrayScatter/TensorArrayScatterV3 as custom op: TensorArrayScatterV3
Warning: No conversion function registered for layer: TensorArrayV3 yet.
Converting filtered_detections/map/TensorArray_1 as custom op: TensorArrayV3
Warning: No conversion function registered for layer: Range yet.
Converting filtered_detections/map/TensorArrayUnstack_1/range as custom op: Range
Warning: No conversion function registered for layer: Enter yet.
Converting filtered_detections/map/while/TensorArrayReadV3_1/Enter as custom op: Enter
Traceback (most recent call last):
  File "/usr/local/bin/convert-to-uff", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/uff/bin/convert_to_uff.py", line 92, in main
    debug_mode=args.debug
  File "/usr/local/lib/python3.6/dist-packages/uff/converters/tensorflow/conversion_helpers.py", line 229, in from_tensorflow_frozen_model
    return from_tensorflow(graphdef, output_nodes, preprocessor, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/uff/converters/tensorflow/conversion_helpers.py", line 178, in from_tensorflow
    debug_mode=debug_mode)
  File "/usr/local/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py", line 94, in convert_tf2uff_graph
    uff_graph, input_replacements, debug_mode=debug_mode)
  File "/usr/local/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py", line 79, in convert_tf2uff_node
    op, name, tf_node, inputs, uff_graph, tf_nodes=tf_nodes, debug_mode=debug_mode)
  File "/usr/local/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py", line 41, in convert_layer
    fields = cls.parse_tf_attrs(tf_node.attr)
  File "/usr/local/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py", line 222, in parse_tf_attrs
    return {key: cls.parse_tf_attr_value(val) for key, val in attrs.items() if val is not None and val.WhichOneof('value') is not None}
  File "/usr/local/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py", line 222, in <dictcomp>
    return {key: cls.parse_tf_attr_value(val) for key, val in attrs.items() if val is not None and val.WhichOneof('value') is not None}
  File "/usr/local/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py", line 218, in parse_tf_attr_value
    return cls.convert_tf2uff_field(code, val)
  File "/usr/local/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py", line 190, in convert_tf2uff_field
    return TensorFlowToUFFConverter.convert_tf2numpy_dtype(val)
  File "/usr/local/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py", line 103, in convert_tf2numpy_dtype
    return tf.as_dtype(dtype).as_numpy_dtype
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py", line 126, in as_numpy_dtype
    return _TF_TO_NP[self._type_enum]
KeyError: 20
5

Hi,

Sorry for the late reply.

TensorFlow support is in Deepstream future plan but not ready yet.
Currently, you will need to go through the following steps to enable Retinanet within Deepstream.
Although it might take some efforts, TensorRT+Deepstream is the optimized solution for Jetson now.

1. Convert .pb file into .uff

$ python3 /usr/lib/python3.6/dist-packages/uff/bin/convert_to_uff.py frozen_inference_graph.pb -O [output/layer/name] -p [path/of/config.py] -o [output/file/name]

In this stage, you may encounter some error caused by TensorFlow operation change or update.
In general, it is related to the shape/format issue and can be fixed via updating the config.py.

If you keep facing error for this, please attach your model and we can check it for you.

2. Create .trt engine from .uff

cd /usr/src/tensor/bin/
./trtexec --uff=[path/to/uff/file] --uffInput=[input/name/and/dim] --output=[outputname]

The most common error occurs in this step is the non-supported operation or layer.
Based on the log you shared in comment#4, it looks like you are not using the latest TensroRT package.
It’s recommended to reflash the Xavier with our latest JetPack installer first.

You can find the support matrix of TensorRT here:

For a non-supported layer, you might need to implement it as plugin.
Samples can be found here: TensorRT/plugin at master · NVIDIA/TensorRT · GitHub

3. Run it with Deepstream
This should be easy. Just follow this sample:
/opt/nvidia/deepstream/deepstream-4.0/sources/objectDetector_SSD/

Update the model path, plugin layer and the network type should be enough.

Thanks.

6

Thanks again for your response!

I’ve tried step 1 above with the output shown below…seems similar as before, so I"m guessing that I need a config.py file to help with the Tensorflow to uff conversion. Unfortunately, I have no idea how to create this config.py file. I’ve searched the internet for an example, but not sure what I’m looking for. I appears that it should contain some GraphSurgeon commands, but not sure how to proceed.

My pb file is about 250MB, and I think the file attachment limit here is 10MB. I’ll try to share it from my Google Drive. Please try this link:

https://drive.google.com/file/d/19H_KpO96AKbC_M1m6Ex3f-l4TH0jtJnA/view?usp=sharing

Even if we get this part to work, I’m a little worried about step 2. Probably have to create a plugin or two for some of the layers. Regardless, first things first…step 1!

Here’s the output I got after executing step 1 above

command I used (note that there are 3 output nodes):

sudo python3 /usr/lib/python3.6/dist-packages/uff/bin/convert_to_uff.py /home/bryan/face/frozen_model.pb -O filtered_detections/map/TensorArrayStack/TensorArrayGatherV3 filtered_detections/map/TensorArrayStack_1/TensorArrayGatherV3 filtered_detections/map/TensorArrayStack_2/TensorArrayGatherV3 -o /home/bryan/face/model.uff

Output:

2019-12-18 13:13:18.690970: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
/home/bryan/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/bryan/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/bryan/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/bryan/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/bryan/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/bryan/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/bryan/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/bryan/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/bryan/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/bryan/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/bryan/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/bryan/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
WARNING:tensorflow:From /usr/lib/python3.6/dist-packages/uff/converters/tensorflow/conversion_helpers.py:18: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.

Loading /home/bryan/face/frozen_model.pb
WARNING:tensorflow:From /usr/lib/python3.6/dist-packages/uff/converters/tensorflow/conversion_helpers.py:231: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

NOTE: UFF has been tested with TensorFlow 1.12.0. Other versions are not guaranteed to work
WARNING: The version of TensorFlow installed on this system is not guaranteed to work with UFF.
WARNING:tensorflow:From /usr/lib/python3.6/dist-packages/graphsurgeon/_utils.py:2: The name tf.NodeDef is deprecated. Please use tf.compat.v1.NodeDef instead.

UFF Version 0.6.3
=== Automatically deduced input nodes ===
[name: "input_1"
op: "Placeholder"
attr {
  key: "dtype"
  value {
    type: DT_FLOAT
  }
}
attr {
  key: "shape"
  value {
    shape {
      dim {
        size: -1
      }
      dim {
        size: -1
      }
      dim {
        size: -1
      }
      dim {
        size: 3
      }
    }
  }
}
]
=========================================

Using output node filtered_detections/map/TensorArrayStack/TensorArrayGatherV3
Converting to UFF graph
Warning: No conversion function registered for layer: TensorArrayGatherV3 yet.
Converting filtered_detections/map/TensorArrayStack/TensorArrayGatherV3 as custom op: TensorArrayGatherV3
WARNING:tensorflow:From /usr/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py:179: The name tf.AttrValue is deprecated. Please use tf.compat.v1.AttrValue instead.

Warning: No conversion function registered for layer: Exit yet.
Converting filtered_detections/map/while/Exit_2 as custom op: Exit
Warning: No conversion function registered for layer: Switch yet.
Converting filtered_detections/map/while/Switch_2 as custom op: Switch
Warning: No conversion function registered for layer: LoopCond yet.
Converting filtered_detections/map/while/LoopCond as custom op: LoopCond
Warning: No conversion function registered for layer: LogicalAnd yet.
Converting filtered_detections/map/while/LogicalAnd as custom op: LogicalAnd
Warning: No conversion function registered for layer: Less yet.
Converting filtered_detections/map/while/Less_1 as custom op: Less
Warning: No conversion function registered for layer: Enter yet.
Converting filtered_detections/map/while/Less/Enter as custom op: Enter
Warning: No conversion function registered for layer: Cast yet.
Converting clipped_boxes/Cast as custom op: Cast
Warning: No conversion function registered for layer: Tile yet.
Converting anchors_4/Tile as custom op: Tile
Warning: No conversion function registered for layer: Fill yet.
Converting anchors_4/meshgrid/ones as custom op: Fill
Warning: No conversion function registered for layer: Size yet.
Converting anchors_4/meshgrid/Size as custom op: Size
Warning: No conversion function registered for layer: Cast yet.
Converting anchors_4/Cast as custom op: Cast
Warning: No conversion function registered for layer: Range yet.
Converting anchors_4/arange as custom op: Range
Warning: No conversion function registered for layer: Size yet.
Converting anchors_4/meshgrid/Size_1 as custom op: Size
Warning: No conversion function registered for layer: Cast yet.
Converting anchors_4/Cast_1 as custom op: Cast
Warning: No conversion function registered for layer: Range yet.
Converting anchors_4/arange_1 as custom op: Range
Warning: No conversion function registered for layer: Tile yet.
Converting anchors_3/Tile as custom op: Tile
Warning: No conversion function registered for layer: Fill yet.
Converting anchors_3/meshgrid/ones as custom op: Fill
Warning: No conversion function registered for layer: Size yet.
Converting anchors_3/meshgrid/Size as custom op: Size
Warning: No conversion function registered for layer: Cast yet.
Converting anchors_3/Cast as custom op: Cast
Warning: No conversion function registered for layer: Range yet.
Converting anchors_3/arange as custom op: Range
Warning: No conversion function registered for layer: Size yet.
Converting anchors_3/meshgrid/Size_1 as custom op: Size
Warning: No conversion function registered for layer: Cast yet.
Converting anchors_3/Cast_1 as custom op: Cast
Warning: No conversion function registered for layer: Range yet.
Converting anchors_3/arange_1 as custom op: Range
Warning: No conversion function registered for layer: Tile yet.
Converting anchors_2/Tile as custom op: Tile
Warning: No conversion function registered for layer: Fill yet.
Converting anchors_2/meshgrid/ones as custom op: Fill
Warning: No conversion function registered for layer: Size yet.
Converting anchors_2/meshgrid/Size as custom op: Size
Warning: No conversion function registered for layer: Cast yet.
Converting anchors_2/Cast as custom op: Cast
Warning: No conversion function registered for layer: Range yet.
Converting anchors_2/arange as custom op: Range
Warning: No conversion function registered for layer: Size yet.
Converting anchors_2/meshgrid/Size_1 as custom op: Size
Warning: No conversion function registered for layer: Cast yet.
Converting anchors_2/Cast_1 as custom op: Cast
Warning: No conversion function registered for layer: Range yet.
Converting anchors_2/arange_1 as custom op: Range
Warning: No conversion function registered for layer: Tile yet.
Converting anchors_1/Tile as custom op: Tile
Warning: No conversion function registered for layer: ResizeNearestNeighbor yet.
Converting P5_upsampled/resize/ResizeNearestNeighbor as custom op: ResizeNearestNeighbor
Warning: No conversion function registered for layer: Fill yet.
Converting anchors_1/meshgrid/ones as custom op: Fill
Warning: No conversion function registered for layer: Size yet.
Converting anchors_1/meshgrid/Size as custom op: Size
Warning: No conversion function registered for layer: Cast yet.
Converting anchors_1/Cast as custom op: Cast
Warning: No conversion function registered for layer: Range yet.
Converting anchors_1/arange as custom op: Range
Warning: No conversion function registered for layer: Size yet.
Converting anchors_1/meshgrid/Size_1 as custom op: Size
Warning: No conversion function registered for layer: Cast yet.
Converting anchors_1/Cast_1 as custom op: Cast
Warning: No conversion function registered for layer: Range yet.
Converting anchors_1/arange_1 as custom op: Range
Warning: No conversion function registered for layer: Tile yet.
Converting anchors_0/Tile as custom op: Tile
Warning: No conversion function registered for layer: ResizeNearestNeighbor yet.
Converting P4_upsampled/resize/ResizeNearestNeighbor as custom op: ResizeNearestNeighbor
Warning: No conversion function registered for layer: Fill yet.
Converting anchors_0/meshgrid/ones as custom op: Fill
Warning: No conversion function registered for layer: Size yet.
Converting anchors_0/meshgrid/Size as custom op: Size
Warning: No conversion function registered for layer: Cast yet.
Converting anchors_0/Cast as custom op: Cast
Warning: No conversion function registered for layer: Range yet.
Converting anchors_0/arange as custom op: Range
Warning: No conversion function registered for layer: Size yet.
Converting anchors_0/meshgrid/Size_1 as custom op: Size
Warning: No conversion function registered for layer: Cast yet.
Converting anchors_0/Cast_1 as custom op: Cast
Warning: No conversion function registered for layer: Range yet.
Converting anchors_0/arange_1 as custom op: Range
Warning: No conversion function registered for layer: Merge yet.
Converting filtered_detections/map/while/Merge_1 as custom op: Merge
Warning: No conversion function registered for layer: NextIteration yet.
Converting filtered_detections/map/while/NextIteration_1 as custom op: NextIteration
Warning: No conversion function registered for layer: Switch yet.
Converting filtered_detections/map/while/Switch as custom op: Switch
Warning: No conversion function registered for layer: Merge yet.
Converting filtered_detections/map/while/Merge as custom op: Merge
Warning: No conversion function registered for layer: NextIteration yet.
Converting filtered_detections/map/while/NextIteration as custom op: NextIteration
Warning: No conversion function registered for layer: Enter yet.
Converting filtered_detections/map/while/Enter as custom op: Enter
Warning: No conversion function registered for layer: Switch yet.
Converting filtered_detections/map/while/Switch_1 as custom op: Switch
Warning: No conversion function registered for layer: Enter yet.
Converting filtered_detections/map/while/Enter_1 as custom op: Enter
Warning: No conversion function registered for layer: Less yet.
Converting filtered_detections/map/while/Less as custom op: Less
Warning: No conversion function registered for layer: Merge yet.
Converting filtered_detections/map/while/Merge_2 as custom op: Merge
Warning: No conversion function registered for layer: NextIteration yet.
Converting filtered_detections/map/while/NextIteration_2 as custom op: NextIteration
Warning: No conversion function registered for layer: TensorArrayWriteV3 yet.
Converting filtered_detections/map/while/TensorArrayWrite/TensorArrayWriteV3 as custom op: TensorArrayWriteV3
Warning: No conversion function registered for layer: PadV2 yet.
Converting filtered_detections/map/while/PadV2 as custom op: PadV2
Warning: No conversion function registered for layer: TopKV2 yet.
Converting filtered_detections/map/while/TopKV2 as custom op: TopKV2
Warning: No conversion function registered for layer: GatherNd yet.
Converting filtered_detections/map/while/GatherNd_2 as custom op: GatherNd
Warning: No conversion function registered for layer: GatherNd yet.
Converting filtered_detections/map/while/GatherNd_1 as custom op: GatherNd
Warning: No conversion function registered for layer: NonMaxSuppressionV3 yet.
Converting filtered_detections/map/while/non_max_suppression/NonMaxSuppressionV3 as custom op: NonMaxSuppressionV3
Warning: No conversion function registered for layer: Where yet.
Converting filtered_detections/map/while/Where as custom op: Where
Warning: No conversion function registered for layer: Greater yet.
Converting filtered_detections/map/while/Greater as custom op: Greater
Warning: No conversion function registered for layer: TensorArrayReadV3 yet.
Converting filtered_detections/map/while/TensorArrayReadV3_1 as custom op: TensorArrayReadV3
Warning: No conversion function registered for layer: Enter yet.
Converting filtered_detections/map/while/TensorArrayReadV3_1/Enter_1 as custom op: Enter
Warning: No conversion function registered for layer: TensorArrayScatterV3 yet.
Converting filtered_detections/map/TensorArrayUnstack_1/TensorArrayScatter/TensorArrayScatterV3 as custom op: TensorArrayScatterV3
Warning: No conversion function registered for layer: TensorArrayV3 yet.
Converting filtered_detections/map/TensorArray_1 as custom op: TensorArrayV3
Warning: No conversion function registered for layer: Range yet.
Converting filtered_detections/map/TensorArrayUnstack_1/range as custom op: Range
Warning: No conversion function registered for layer: Enter yet.
Converting filtered_detections/map/while/TensorArrayReadV3_1/Enter as custom op: Enter
Traceback (most recent call last):
  File "/usr/lib/python3.6/dist-packages/uff/bin/convert_to_uff.py", line 93, in <module>
    main()
  File "/usr/lib/python3.6/dist-packages/uff/bin/convert_to_uff.py", line 89, in main
    debug_mode=args.debug
  File "/usr/lib/python3.6/dist-packages/uff/converters/tensorflow/conversion_helpers.py", line 233, in from_tensorflow_frozen_model
    return from_tensorflow(graphdef, output_nodes, preprocessor, **kwargs)
  File "/usr/lib/python3.6/dist-packages/uff/converters/tensorflow/conversion_helpers.py", line 181, in from_tensorflow
    debug_mode=debug_mode)
  File "/usr/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py", line 94, in convert_tf2uff_graph
    uff_graph, input_replacements, debug_mode=debug_mode)
  File "/usr/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py", line 79, in convert_tf2uff_node
    op, name, tf_node, inputs, uff_graph, tf_nodes=tf_nodes, debug_mode=debug_mode)
  File "/usr/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py", line 41, in convert_layer
    fields = cls.parse_tf_attrs(tf_node.attr)
  File "/usr/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py", line 222, in parse_tf_attrs
    return {key: cls.parse_tf_attr_value(val) for key, val in attrs.items() if val is not None and val.WhichOneof('value') is not None}
  File "/usr/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py", line 222, in <dictcomp>
    return {key: cls.parse_tf_attr_value(val) for key, val in attrs.items() if val is not None and val.WhichOneof('value') is not None}
  File "/usr/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py", line 218, in parse_tf_attr_value
    return cls.convert_tf2uff_field(code, val)
  File "/usr/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py", line 190, in convert_tf2uff_field
    return TensorFlowToUFFConverter.convert_tf2numpy_dtype(val)
  File "/usr/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py", line 103, in convert_tf2numpy_dtype
    return tf.as_dtype(dtype).as_numpy_dtype
  File "/home/bryan/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py", line 126, in as_numpy_dtype
    return _TF_TO_NP[self._type_enum]
KeyError: 20
7

Hi,

Thanks for the update.

Let me check your model first. Will get back to you asap.

8

Hi,

The error occurs when converting a TensorFlow tensor data into a numpy array.
The invalid type is 20, which indicates tf.resource.
There is no corresponding data type in numpy so it trigger the assertion.

Let me check it further and get back to you later.
Thanks.

9

Hi,

It looks like there are some non-supported operations in the TensorFlow based RetinaNet.

Would you mind to convert the pb into onnx format to see if helps?
The whole procedure can be found in this comment:
https://devtalk.nvidia.com/default/topic/1066445/tensorrt/tensorrt-6-0-1-tensorflow-1-14-no-conversion-function-registered-for-layer-fusedbatchnormv3-yet/post/5403567/#5403567

Thanks and please let us know if you need help.

10

The tf2onnx converter you suggested above appeared to work if I set the opset to 11. I hope that the TensorRT ONNX parser will handle opset 11. Now I need to figure out how to test…I’ll get back to you as soon as I have some info on that.

Thanks!

11

I can’t seem to get the Deepstream onnx parser to read the model. Would it be possible to update the onnx parser? If so, and assuming it might help, would that be re-building the nvinfer plugin from source (with updated onnx parser code)?

Here’s the output I get:

$ LD_PRELOAD=/opt/nvidia/deepstream/deepstream-4.0/samples/configs/deepstream-app/nvdsparsebbox_retinanet_equature.so ./deepstream-app -c /opt/nvidia/deepstream/deepstream-4.0/samples/configs/deepstream-app/source1_usb_dec_infer_resnet_int8.txt

(deepstream-app:7478): GStreamer-WARNING **: 15:28:43.274: Name 'src_cap_filter' is not unique in bin 'src_sub_bin0', not adding
Unknown key 'parse-func' for group [property]
Creating LL OSD context new
0:00:01.561027531  7478     0x263adad0 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:initialize(): Trying to create engine from model files
----------------------------------------------------------------
Input filename:   /opt/nvidia/deepstream/deepstream-4.0/samples/configs/deepstream-app/face.onnx
ONNX IR version:  0.0.6
Opset version:    11
Producer name:    tf2onnx
Producer version: 1.6.0
Domain:           
Model version:    0
Doc string:       
----------------------------------------------------------------
WARNING: ONNX model has a newer ir_version (0.0.6) than this parser was built against (0.0.3).
0:00:02.509181584  7478     0x263adad0 ERROR                nvinfer gstnvinfer.cpp:511:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:log(): Parameter check failed at: ../builder/Network.cpp::addInput::671, condition: isValidDims(dims, hasImplicitBatchDimension())
ERROR: ModelImporter.cpp:80 In function importInput:
[8] Assertion failed: *tensor = importer_ctx->network()->addInput( input.name().c_str(), trt_dtype, trt_dims)
0:00:02.519570691  7478     0x263adad0 ERROR                nvinfer gstnvinfer.cpp:511:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:generateTRTModel(): Failed to parse onnx file
0:00:02.528065826  7478     0x263adad0 ERROR                nvinfer gstnvinfer.cpp:511:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:initialize(): Failed to create engine from model files
0:00:02.528416530  7478     0x263adad0 WARN                 nvinfer gstnvinfer.cpp:692:gst_nvinfer_start:<primary_gie_classifier> error: Failed to create NvDsInferContext instance
0:00:02.528589178  7478     0x263adad0 WARN                 nvinfer gstnvinfer.cpp:692:gst_nvinfer_start:<primary_gie_classifier> error: Config file path: /opt/nvidia/deepstream/deepstream-4.0/samples/configs/deepstream-app/infer_config_batch1_equature.txt, NvDsInfer Error: NVDSINFER_TENSORRT_ERROR
** ERROR: <main:651>: Failed to set pipeline to PAUSED
Quitting
ERROR from primary_gie_classifier: Failed to create NvDsInferContext instance
Debug info: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(692): gst_nvinfer_start (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie_classifier:
Config file path: /opt/nvidia/deepstream/deepstream-4.0/samples/configs/deepstream-app/infer_config_batch1_equature.txt, NvDsInfer Error: NVDSINFER_TENSORRT_ERROR
App run failed

Here is my main config file:

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=1
rows=1
columns=1
width=1280
height=720

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=1
camera-width=1280
camera-height=720
camera-fps-n=30
camera-fps-d=1
camera-v4l2-dev-node=0

[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming 5=Overlay
type=5
sync=0
display-id=0
offset-x=0
offset-y=0
width=0
height=0
overlay-id=1
source-id=0

[osd]
enable=1
border-width=2
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0

[streammux]
##Boolean property to inform muxer that sources are live
live-source=1
batch-size=1
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
## Set muxer output width and height
width=1280
height=720

# config-file property is mandatory for any gie section.
# Other properties are optional and if set will override the properties set in
# the infer config file.
[primary-gie]
enable=1
#Required to display the PGIE labels, should be added even when using config-file
#property
batch-size=1
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=0
#Required by the app for SGIE, when used along with config-file property
gie-unique-id=1
config-file=retinanet_config_1.txt

[tests]
file-loop=0

Here is config file for network:

[property]
gpu-id=0
net-scale-factor=0.017352074
offsets=123.675;116.28;103.53
labelfile-path=labels_face.txt
batch-size=1
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=2
num-detected-classes=1
interval=0
gie-unique-id=1
#parse-func=0
is-classifier=0
onnx-file=/home/bryan/face/deepstream/face.onnx
parse-bbox-func-name=NvDsInferParseRetinaNet
custom-lib-path=/home/bryan/face/deepstream/nvdsparsebbox_retinanet_equature.so
#enable-dbscan=1

[class-attrs-all]
threshold=0.5
group-threshold=0
## Set eps=0.7 and minBoxes for enable-dbscan=1
#eps=0.2
##minBoxes=3
#roi-top-offset=0
#roi-bottom-offset=0
detected-min-w=4
detected-min-h=4
#detected-max-w=0
#detected-max-h=0

## Per class configuration
#[class-attrs-2]
#threshold=0.6
#eps=0.5
#group-threshold=3
#roi-top-offset=20
#roi-bottom-offset=10
#detected-min-w=40
#detected-min-h=40
#detected-max-w=400
#detected-max-h=800
12

Hi,

Sorry for the late.
Could you share the converted onnx file for us checking?

Thanks.

13

Thanks for your continued help with this!

My converted onnx file can be retrieved using the following link:

https://drive.google.com/open?id=18lLlrxOgEIISB_4DETvtdWxNDZB1DRZ4

I used the conversion tools from this repo:
https://github.com/onnx/tensorflow-onnx.git

This file was built using the following command (and reported success):

python -m tf2onnx.convert --input /home/bryan/training/RetinaNet/face/snapshots/frozen_model.pb --inputs input_1:0 --outputs 'filtered_detections/map/TensorArrayStack/TensorArrayGatherV3:0','filtered_detections/map/TensorArrayStack_1/TensorArrayGatherV3:0','filtered_detections/map/TensorArrayStack_2/TensorArrayGatherV3:0' --output /home/bryan/training/RetinaNet/face/snapshots/face.onnx --verbose --opset 11

Finally, I tried to use the onnx file with code from this repo (as suggested by you earlier):
https://github.com/NVIDIA/retinanet-examples.git

I used the docker container suggested, and ran the command inside the docker container:
$retinanet export face.onnx face.plan

which gave the following:

Traceback (most recent call last):
  File "/opt/conda/bin/retinanet", line 10, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.6/site-packages/retinanet/main.py", line 176, in main
    model, state = load_model(args, verbose=True)
  File "/opt/conda/lib/python3.6/site-packages/retinanet/main.py", line 98, in load_model
    raise RuntimeError('Invalid model format "{}"!'.format(args.ext))
AttributeError: 'Namespace' object has no attribute 'ext'

Thus failing to create the TRT engine

14

Hi,

I can reproduce this issue with TensorRT binary directly.

$ /usr/src/tensorrt/bin/trtexec --onnx=./face.onnx

[00/14/2020-15:46:20] [E] [TRT] Parameter check failed at: …/builder/Network.cpp::addInput::671, condition: isValidDims(dims, hasImplicitBatchDimension())
ERROR: ModelImporter.cpp:80 In function importInput:
[8] Assertion failed: *tensor = importer_ctx->network()->addInput( input.name().c_str(), trt_dtype, trt_dims)

This is a known issue and won’t occur in TensorRT 7.0.
However, TensorRT 7.0 is not available for Jetson user yet.
Please pay attention to announcement and we will let you know once the new release is ready.

Thanks and sorry for the inconvenience.

15

Hi,

I installed TensorRT 7.0 on my Ubuntu 18.04 host machine (not Jetson), and tried trtexec with my onnx file. Looks like I’ve got some sort of datatype issue.

Here is the new error message:

(tensorrt) bryan@Aurora-R7:~/TensorRT/TensorRT-7.0.0.11/bin$ ./trtexec --onnx=/home/bryan/training/RetinaNet/face/snapshots/face.onnx
&&&& RUNNING TensorRT.trtexec # ./trtexec --onnx=/home/bryan/training/RetinaNet/face/snapshots/face.onnx
[01/14/2020-11:28:56] [I] === Model Options ===
[01/14/2020-11:28:56] [I] Format: ONNX
[01/14/2020-11:28:56] [I] Model: /home/bryan/training/RetinaNet/face/snapshots/face.onnx
[01/14/2020-11:28:56] [I] Output:
[01/14/2020-11:28:56] [I] === Build Options ===
[01/14/2020-11:28:56] [I] Max batch: 1
[01/14/2020-11:28:56] [I] Workspace: 16 MB
[01/14/2020-11:28:56] [I] minTiming: 1
[01/14/2020-11:28:56] [I] avgTiming: 8
[01/14/2020-11:28:56] [I] Precision: FP32
[01/14/2020-11:28:56] [I] Calibration: 
[01/14/2020-11:28:56] [I] Safe mode: Disabled
[01/14/2020-11:28:56] [I] Save engine: 
[01/14/2020-11:28:56] [I] Load engine: 
[01/14/2020-11:28:56] [I] Inputs format: fp32:CHW
[01/14/2020-11:28:56] [I] Outputs format: fp32:CHW
[01/14/2020-11:28:56] [I] Input build shapes: model
[01/14/2020-11:28:56] [I] === System Options ===
[01/14/2020-11:28:56] [I] Device: 0
[01/14/2020-11:28:56] [I] DLACore: 
[01/14/2020-11:28:56] [I] Plugins:
[01/14/2020-11:28:56] [I] === Inference Options ===
[01/14/2020-11:28:56] [I] Batch: 1
[01/14/2020-11:28:56] [I] Iterations: 10
[01/14/2020-11:28:56] [I] Duration: 3s (+ 200ms warm up)
[01/14/2020-11:28:56] [I] Sleep time: 0ms
[01/14/2020-11:28:56] [I] Streams: 1
[01/14/2020-11:28:56] [I] ExposeDMA: Disabled
[01/14/2020-11:28:56] [I] Spin-wait: Disabled
[01/14/2020-11:28:56] [I] Multithreading: Disabled
[01/14/2020-11:28:56] [I] CUDA Graph: Disabled
[01/14/2020-11:28:56] [I] Skip inference: Disabled
[01/14/2020-11:28:56] [I] Input inference shapes: model
[01/14/2020-11:28:56] [I] Inputs:
[01/14/2020-11:28:56] [I] === Reporting Options ===
[01/14/2020-11:28:56] [I] Verbose: Disabled
[01/14/2020-11:28:56] [I] Averages: 10 inferences
[01/14/2020-11:28:56] [I] Percentile: 99
[01/14/2020-11:28:56] [I] Dump output: Disabled
[01/14/2020-11:28:56] [I] Profile: Disabled
[01/14/2020-11:28:56] [I] Export timing to JSON file: 
[01/14/2020-11:28:56] [I] Export output to JSON file: 
[01/14/2020-11:28:56] [I] Export profile to JSON file: 
[01/14/2020-11:28:56] [I] 
----------------------------------------------------------------
Input filename:   /home/bryan/training/RetinaNet/face/snapshots/face.onnx
ONNX IR version:  0.0.6
Opset version:    11
Producer name:    tf2onnx
Producer version: 1.6.0
Domain:           
Model version:    0
Doc string:       
----------------------------------------------------------------
[01/14/2020-11:28:57] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[01/14/2020-11:28:57] [W] [TRT] onnx2trt_utils.cpp:222: One or more weights outside the range of INT32 was clamped
ERROR: ModelImporter.cpp:92 In function parseGraph:
[8] Assertion failed: convertOnnxWeights(initializer, &weights, ctx)
[01/14/2020-11:28:57] [E] Failed to parse onnx file
[01/14/2020-11:28:57] [E] Parsing model failed
[01/14/2020-11:28:57] [E] Engine creation failed
[01/14/2020-11:28:57] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec # ./trtexec --onnx=/home/bryan/training/RetinaNet/face/snapshots/face.onnx
16

Thanks for the trying.
Let me check this issue and get back to you soon.

17

Hi,

This error indicates a non-supported node in the onnx model
https://github.com/onnx/onnx-tensorrt/blob/master/ModelImporter.cpp#L92

Could you check if all your onnx operation are list in this supported matrix first:
https://github.com/onnx/onnx-tensorrt/blob/master/operators.md

Thanks.

18

I’ll have to go through the Keras Retinanet implementation and look at all the layer types, or maybe use Netron to look at the layer types. I’ll do that.

In the mean time, I went back and used the nvidia/retinanet-examples repo to build and train (actually fine tune) a retinanet with my data. I just did a quick training cycle to see if I could use the resulting model with Deepstream. This turned out to be successful with the following caveats:

1 - when I run on Deepstream, it runs WAY too slow (< 1 fps). The CPU on the Jetson is running at over 80%, which tells me something weird is going on. Probably a problem with the config files (I used the ones provided in the retinanet-examples repo). A possible hint may be given by this Warning when Deepstream is starting up

WARN: <parse_streammux:526>: Unknown key 'cuda-memory-type' for group [streammux]
** WARN: <parse_sink:1238>: Unknown key 'cuda-memory-type' for group [sink1]

(deepstream-app:11440): GLib-GObject-WARNING **: 16:23:14.402: g_object_set_is_valid_property: object class 'avenc_mpeg4' has no property named 'iframeinterval'

(deepstream-app:11440): GLib-GObject-WARNING **: 16:23:14.402: g_object_set_is_valid_property: object class 'avenc_mpeg4' has no property named 'bufapi-version'
Unknown key 'parse-func' for group [property]

2 - I’m a little worried about the training. There’s not much feedback from the training process. I’m used to monitoring the training process with Tensorboard. I probably just need to experiment with this a bit.

If you have any thoughts on issue 1, I’d appreciate it.

I’ll also get back to you on the layer-type compatibility issue with my original network.

Thanks!!

19

Hi,

Thanks for your trying.

1. The log is a harmless warning.
Except from the CPU, could you also monitor the GPU usage and share the data with us?

$ sudo tegrastats
RAM 1238/7763MB ... <b>GR3D_FREQ 0%@905</b> ...

2.
Suppose you can still monitor the training process with tensorboard for pyTorch.

Thanks.