Unable to convert Keras model to TensorRT runtime

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

I’m trying to convert a Keras YoloV3 model (i.e. [url]https://github.com/qqwweee/keras-yolo3[/url]) to a format that can be loaded with DeepStream’s nvinfer element.

I’ve seen the samples at [url]https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps[/url] as well as [url]https://docs.nvidia.com/deeplearning/sdk/tensorrt-sample-support-guide/index.html#yolov3_onnx[/url], so I know that it is technically possible, but I haven’t yet been successful with the conversion.

I’ve tried permutations of freezing the graph, converting to UFF/ONNX, using [url]https://github.com/onnx/tensorflow-onnx[/url], but every time there was a different error.

What I’d like to know is: what the best approach is to go from a Keras model to inference in DeepStream?

2

Hi,

It’s recommended to use UFF interface rather than ONNX since UFF is our format specified for TensorFlow.
Once having a UFF model, there is a sample in deepstream package for the UFF based model:
{DeepStream_Release}/sources/objectDetector_SSD

To enable YOLO, it is required some plugin implementation since some non-supported layers inside.
We have an plugin implementation for YOLO in deepstream_reference_apps and you can integrate it directly into your application.

Thanks.

3

Solution 1:
Can you refer to https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps/tree/master/yolo to deloy yoloV3 detection? It does not use uff/onnx/caffe parser but use tensorRT API directly to build network.

Solution 2:

  1. Using tensorflow’s graph_utils, graph_io API to convert keras model to .pb.
    If the model is trained in NHWC, we should make sure NCHW architecture could consume the pretrained weights. Generally, most layers could work well directly in NHWC → NCHW conversion except Reshape, Flatten, Dense and Softmax applied to feature map.
    GitHub - amir-abdi/keras_to_tensorflow: General code to convert a trained keras model into an inference tensorflow model

  2. Converting .pb to .uff using uff converter
    And map all the tf nodes which are not supported by TensorRT directly to plugin node.
    Refer to sources/objectDetector_FasterRCNN

1 Like
4

Thanks for the quick answers! Is there a process to figure out which layers in a model are supported by uff and what I need custom plugins for?

5

TensorRT provides convert-to-uff tool which will return error and which custom plugins need to implement.

https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#samplecode3

You can also refer to tensorRT sample sampleUffSSD “README.md” and “config.py”

6

Thank you both for your help!