TensorRT and Tensorflow: convert to uff failed

We are trying to create a trt engine from the saved uff model. We are getting the below error. Any idea what is causing it?

[TensorRT] ERROR: Specified INT8 but no calibrator provided
Traceback (most recent call last):
  File "generate_prune_graph.py", line 50, in <module>
    engine = trt.utils.uff_to_trt_engine(G_LOGGER, uff_model, parser, 1, 1<<20, trt.infer.DataType.INT8)
  File "/usr/lib/python2.7/dist-packages/tensorrt/utils/_utils.py", line 182, in uff_to_trt_engine
    raise AttributeError("Specified INT8 but no calibrator provided")
AttributeError: Specified INT8 but no calibrator provided

If we change the datatype to HALF we get the following error

[TensorRT] ERROR: Specified FP16 but not supported on platform
Traceback (most recent call last):
  File "generate_prune_graph.py", line 50, in <module>
    engine = trt.utils.uff_to_trt_engine(G_LOGGER, uff_model, parser, 1, 1<<20, trt.infer.DataType.HALF)
  File "/usr/lib/python2.7/dist-packages/tensorrt/utils/_utils.py", line 177, in uff_to_trt_engine
    raise AttributeError("Specified FP16 but not supported on platform")
AttributeError: Specified FP16 but not supported on platform

Hi,

1. INT8 feature is only available on GPU architecture=6.1 or 7.x.
It doesn’t support TX2 platform, which is in sm=6.2 design.

2. Python API is also not available on TX2.
To inference a Tensorflow model, please convert it to UFF format first and launch it via C++ interface.

Thanks and sorry for the inconvenience.

Hello,

We are running this on x64 machine. We want to create the engine (on x64), save it to a file, copy that to TX2 and then run it on the TX2. Is that possible? Or we need to create the engine itself on TX2?

-siddarth

Also, the GPU is GTX 1080.

Hi,

Depends on if your model has a non-supported layer.

For a Tensorflow user, the recommended steps to launch the model on Jetson are:
1. Export TensorFlow model to UFF format with python API on the x86-based machine.
2. Launch TensorRT engine from UFF model with C+ interface on Jetson.

On TX2, only float and FP16 mode are available. No INT8 and uff plugin support.

More, we don’t recommend to output the serialized TensorRT PLAN directly.
TensorRT selects an appropriate implementation with GPU architecture information.
To use a PLAN engine compiled with different architecture may yield some issue and not be optimal.

Thanks.

Hello,

I am facing the same issue (error saying) while converting mobilenet’s frozen graph to uff:

AttributeError: ‘RepeatedCompositeFieldContainer’ object has no attribute ‘unknown_rank’

how did you cleared this error? (If you did.)

Any help will be appreciated. Thanks!!

Hi, gautam.patel

This issue is from the pure TensorFlow use case.
It’s recommended to share your issue with the TensorFlow developer for information.

Thanks.

For TensorFlow Object Detection API, is there any possibility to convert the COCO-trained models (including ssd_mobilenet_v1, ssd_mobilenet_v2, ssd_inception, faster_rcnn_resnet50, faster_rcnn_resnet101 from https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md to TensorRT 4?

Is there any way to modify these models to make it work?

If there is any future plan for them, and estimated time?

Thanks a lot.

Hi,

There are lots of new layer in TensorRT 4.
Please check this page to find if your model is fully-supported:
[url]Developer Guide :: NVIDIA Deep Learning TensorRT Documentation

More, UFF plugin API is supported from TensorRT 4.
For any not-supported layer of TensorFlow, you can implement it with Plugin API now.

Thanks.

Thanks. How to use the UFF plugin API for non-supported layers of TF? Could you show an example of converting the Object Detection Models (SSD_mobilenet or faster rcnn_resnet50), or any other latest example?

Hi,

Please check this topic for the sample:
https://devtalk.nvidia.com/default/topic/1032100

Thanks.

Has the plugin been enabled in TensorRT 4?

YES.
Plugin is added into UFF parser from TensorRT4.0.

Currently, TensorRT4.0 is only available for desktop GPU.
Please wait for our announcement for the update of Jetson package.

Thanks.

Hi TensorRT 4.0. is now part of the Jetpack 3.3.
I’m hoping to convert ssd_mobilenet_v2_coco tf model to TrT.

Is there still layers in ssd_mobilenet not supported byt TrT 4.0?

Hi,

Sorry that we didn’t go through ssd_mobilenet_v2_coco in detail.
You can find our supported layers here:
[url]https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#support_op[/url]

Thanks.

Hi, AastaLLL,
Is there any plans for converting some latest TF models to TRT UFF?
Thanks.

Hi,

We don’t focus on a special model but operations.
Our internal team will evaluate current operations and implement them with priority.

For non-supported operation, you can also add it as a plugin layer.
Thanks.

Hi,

TLDR:

  1. Used convert-to-uff to convert ssd_inception_v2_coco (sampleUffSSD) and ssd_mobilenet_v1_coco to uff (x86)
  2. Tried running both the sample_uff_ssd and sample_uff_ssd_debug binaries. Worked for inception sample but got memory issues when trying to run the mobilenet. (Jetson TX2)

I am trying to get ssd_mobilenet_v1_coco (from the tensorflow SSD zoo) parsed in TensorRT. Comparing it the UFF SSD example in TensorRT 4.0, these two models are the same in the plugin support necessary to get them running (at least from what I have found). In fact, the internal operations inside the mobilenet modeled are supported by TensorRT 4.0: DepthwiseConv2dNative, Conv2D, the underlying ‘batchnorm’ ops which are really just (Conv2D, add and mul) are all supported). Comparing their pipeline.config and overall architecture files in Tensorboard, they seem to have the same plugin parameters.

Based on the above, I have reason to believe that I should be able to use the mobilenet model under the same context of the inception uffSSD example in TensorRT (convert-to-uff, run inference). I am able to successfully parse this model into UFF using the same approach as the ssd_inception_v2_coco example. When I run the inception ssd model, it works fine and gives me the correct output. When I try to do the same for mobilenet, I get a core dump following the enquing memory allocation for the concat plugins. Then Tensorboard output suggests that there is a one-to-one mapping between the concat layers for mobilenet and inception.

Running cuda-memcheck yields an interesting error for the mobilenet:
========= CUDA-MEMCHECK
========= Invalid global read of size 16
========= at 0x00000150 in void cuScale::scale<float, float, cuScale::Mode, bool=0, int=4, cuScale::FusedActivationType>(float const , cuScale::scale<float, float, cuScale::Mode, bool=0, int=4, cuScale::FusedActivationType>, cuScale::KernelParameters<cuScale::scale<float, float, cuScale::Mode, bool=0, int=4, cuScale::FusedActivationType>>, nvinfer1::cudnn::reduced_divisor, nvinfer1::cudnn, nvinfer1::cudnn)
========= by thread (63,0,0) in block (11,0,0)
========= Address 0xfc89bd9f8 is misaligned
========= Saved host backtrace up to driver entry point at kernel launch time
========= Host Frame:/usr/lib/aarch64-linux-gnu/tegra/libcuda.so.1 (cuLaunchKernel + 0x1e8) [0x1fe770]
========= Host Frame:/usr/local/cuda-9.0/targets/aarch64-linux/lib/libcudart.so.9.0 [0xc984]

And the error shows up (320 times in total) until it finally crashes.

Hi,

Here is a tutorial of ssd_mobilenet_v1_coco on Jetson for your reference:
[url]GitHub - NVIDIA-AI-IOT/tf_trt_models: TensorFlow models accelerated with NVIDIA TensorRT

Thanks.

Hi AstaLLL,

Thank you for getting back to me. I am trying to run this in C++ on the Jetson rather using the Python API. Is there any specific reason that using the uffSSDSample code for ssd_inception_v2_coco would not work for ssd_mobilenet_v1_coco_01_2018. I am using the Jetson Tx2 which has TensorRT 4.0 and from what I am reading, every layer in the mobilenet model is supported by TensorRT and the plugin support needed for this model is identical to that of the aforementioned inception model.