When will dilated convolution be supported in TensorRT (Python), or SpaceToBatchND / BatchToSpaceND

AI & Data Science Deep Learning (Training & Inference) TensorRT
#1

What is the timeline for dilated convolution support in TensorRT (Python), or SpaceToBatchND / BatchToSpaceND?

My ResNet50-based TensorFlow model has SpaceToBatchND and BatchToSpaceND around its Conv2D nodes, like so: https://ibb.co/mdxrBU . But these are not supported by TensorRT (in Python, not sure about C++).

There is this related post https://devtalk.nvidia.com/default/topic/1027934/jetson-tx2/tensorflows-dilated-convolutions-with-uff-parser-not-supported but no solution. (Also this one https://devtalk.nvidia.com/default/topic/1037732/tensorrt/no-conversion-function-registered-for-layer-slice-yet- asking the same for the Slice operation (but I can solve that in a different way.)

I would like to know when SpaceToBatchND / BatchToSpaceND, and/or dilated convolution, will be supported in the Python API. If this is expected soon, I won’t have to spend time (a lot) to implement a workaround: write Plugin nodes in C++, use GraphSurgeon to insert it, or replacing SpaceToBatchND and BatchToSpaceND do with a combination of Slice, Reshape, Padding, etc. (which is problematic since Slice is not currently supported either).

Please let me know about the timeline for this, and provide advice on which path to take. Thank you very much in advance.

#2

I am also interested in knowing when dilated convolution will be supported in TensorRT on a Jetson TX2. My Mobilenetv2 variant uses dilated convolutions and currently this is the only issue that stops the model from being served via TensorRT.

#3

Did it become available with the release of TensorRT 5 RC (Python API: SpaceToBatchND - BatchToSpaceND / dilated convolution)?

In the release notes it says the following (https://docs.nvidia.com/deeplearning/sdk/tensorrt-release-notes/rel_5-0-RC.html#rel_5-0-RC).

“Python samples using the new API include parser samples for ResNet-50, …”

“The included resnet_v1_152, resnet_v1_50, lenet5, and vgg19 UFF files do not support FP16 mode.”

To me, it seems that version 5 RC should now support what I am looking for, because ResNet is mentioned in the release notes (which uses dilated convolution). Correct?

#4

Has this been confirmed yet?

I’m using a model which applies 8 dilated convolutional layers prior to concatenation as it’s input layer, followed by a few typical convolutional layers.

Without TensorFlow’s SpaceToBatchND/BatchToSpaceND, TensorRT only optimizes a portion of the model’s layers, and although it still provides performance improvements, the performance is severely limited by the lack of these ops. Upon inspection, the Conv2D ops for the dilated convolutions compose ~98% of the inference runtime.

If context is helpful, a detailed description of how dilated convolutions are used (and why they’re crucial to the model) can be found here: https://github.com/SamRagusa/Batch-First#input-layers

#5

I saw that dilated convolutions should be supported by TensorRT 5. However, I still have issues with BatchToSpaceND . Did anyone manage to make it work?

#6

https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#dla_layers
DLA supports dilated convolution layer

#7

To anyone having trouble converting tensorflow models with dilated convolutions, make sure you are using the correct dilated convolution implementation. Tensorflow has https://www.tensorflow.org/api_docs/python/tf/nn/atrous_conv2d and https://www.tensorflow.org/api_docs/python/tf/nn/conv2d The first one will wrap normal(non dilated) conv2d ops with space_to_batch and batch_to_space as a way of implementing the dilation. The second link is the op that can use the cudnn implementations for dilated convolutions and also makes for a graph that converts nicely with the uff parser. That being said, I’m still experiencing some issues on tx2 with dilated convolutions. Even more strange, the same frozen tensorflow graph converted/running on my 1080ti works perfectly…