TensorRT ERROR:Unsupport opperation

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

I just followed the introdution convert a keras vgg19 model and convert it to a trt engine. It worked properly.

However, when I try the same job on keras.applications.resnet50, it raised unsupport opperation error.

No. nodes: 3727
[TensorRT] ERROR: UFFParser: Validator error: bn/cond/FusedBatchNorm/Switch_1: Unsupported operation _Switch
[TensorRT] ERROR: Failed to parse UFF model stream
File “/usr/local/lib/python3.5/dist-packages/tensorrt/utils/_utils.py”, line 255, in uff_to_trt_engine
assert(parser.parse(stream, network, model_datatype))
Traceback (most recent call last):
File “/usr/local/lib/python3.5/dist-packages/tensorrt/utils/_utils.py”, line 255, in uff_to_trt_engine
assert(parser.parse(stream, network, model_datatype))
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/wang/PycharmProjects/Test_VGG_Tensorrt/create_engine.py”, line 44, in
create_and_save_inference_engine()
File “/home/wang/PycharmProjects/Test_VGG_Tensorrt/create_engine.py”, line 29, in create_and_save_inference_engine
trt.infer.DataType.FLOAT)
File “/usr/local/lib/python3.5/dist-packages/tensorrt/utils/_utils.py”, line 263, in uff_to_trt_engine
raise AssertionError(‘UFF parsing failed on line {} in statement {}’.format(line, text))
AssertionError: UFF parsing failed on line 255 in statement assert(parser.parse(stream, network, model_datatype))

Process finished with exit code 1

Could anyone figure it out, thanks alot.

2

Hello, can you provide details on the platforms you are using?

Linux distro and version
GPU type
nvidia driver version
CUDA version
CUDNN version
Python version [if using python]
Tensorflow version
TensorRT version

3

Hi, platforms are Ubuntu 16.0.4LTS , Tensorflow 1.10.0, Keras 2.2.2 cuda9.0 cudnn 7.1.3

looking forward to your replying

4

GPU tesla p100 and GTX-1060, both tried, python 3.5 tensorrt 4.0.1.6

5

Hello,

The UFF Parser does not support Switch operation. The list of supported TensorFlow ops are mentioned on the dev guide, https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#support_op

Recommend the option to implement a plugin layer for the unsupported operation or modify the original network to remove this operation. Could try using graphsurgeon for this.

6

Hi @1006740126 have you solved this error?
if you have resolved it please share the solution.

File “/home/wang/PycharmProjects/Test_VGG_Tensorrt/create_engine.py”, line 44, in
create_and_save_inference_engine()
File “/home/wang/PycharmProjects/Test_VGG_Tensorrt/create_engine.py”, line 29, in create_and_save_inference_engine
trt.infer.DataType.FLOAT)
File “/usr/local/lib/python3.5/dist-packages/tensorrt/utils/_utils.py”, line 263, in uff_to_trt_engine
raise AssertionError(‘UFF parsing failed on line {} in statement {}’.format(line, text))
AssertionError: UFF parsing failed on line 255 in statement assert(parser.parse(stream, network, model_datatype))

Thanks

7

Hi,

I believe that Keras uses the Switch operation to determine whether a BatchNormalization layer is trainable or not. Thus, this could be removed and I believe the solution to this is to use K.set_learning_phase(0) before loading the model and to remove training nodes using tf.graph_util.remove_training_nodes(frozen_graph)

8

Thank you for your answer. Yes, I have used the K.set_learning_phase(0) but still parser error.

9

Thank you for your answer @florandep I have used the K.set_learning_phase(0) but still parser error.

code:

import tensorflow as tf
import tensorrt as trt
from tensorrt.parsers import uffparser
import pycuda.driver as cuda
import pycuda.autoinit
import numpy as np
from random import randint # generate a random test case
from PIL import Image
from matplotlib.pyplot import imshow # To show test case
import time
import os
#import keras as K
import keras.backend.tensorflow_backend as K
import uff
K.set_learning_phase(0)
#tf.graph_util.remove_training_nodes(‘/home/cloud4/Akhtar_Vir_Env/my_new_app/mar$
uff_model = uff.from_tensorflow(’/home/cloud4/Akhtar_Vir_Env/my_new_app/mars-sma$
G_LOGGER = trt.infer.ConsoleLogger(trt.infer.LogSeverity.ERROR)
parser = uffparser.create_uff_parser()
parser.register_input(“images”, (128, 64, 3), 0)
parser.register_output(“features”)
engine = trt.utils.uff_to_trt_engine(G_LOGGER, uff_model, parser, 1, 1 << 20)

it create same Error

Error

File “/usr/local/lib/python3.5/dist-packages/tensorrt/utils/_utils.py”, line 255, in uff_to_trt_engine
assert(parser.parse(stream, network, model_datatype))

AssertionError Traceback (most recent call last)
/usr/local/lib/python3.5/dist-packages/tensorrt/utils/_utils.py in uff_to_trt_engine(logger, stream, parser, max_batch_size, max_workspace_size, datatype, plugin_factory, calibrator)
254 try:
→ 255 assert(parser.parse(stream, network, model_datatype))
256 except AssertionError:

AssertionError:

During handling of the above exception, another exception occurred:

AssertionError Traceback (most recent call last)
in ()
----> 1 engine = trt.utils.uff_to_trt_engine(G_LOGGER, uff_model, parser, 1, 1 << 20)

/usr/local/lib/python3.5/dist-packages/tensorrt/utils/_utils.py in uff_to_trt_engine(logger, stream, parser, max_batch_size, max_workspace_size, datatype, plugin_factory, calibrator)
261 filename, line, func, text = tb_info[-1]
262
→ 263 raise AssertionError(‘UFF parsing failed on line {} in statement {}’.format(line, text))
264
265

AssertionError: UFF parsing failed on line 255 in statement assert(parser.parse(stream, network, model_datatype))

10

@p146103, could you try to convert the Keras model into UFF model somewhat like this?

file_path = 'Keras_model.h5'

sess = tf.Session(config=tf.ConfigProto(device_count={'GPU': 0})) # CPU only
K.set_session(sess)

K.set_learning_phase(0)
model = load_model(file_path, compile=False)
K.set_learning_phase(0)
output_name = model.output.op.name
input_name = model.input.op.name
frozen_graph = tf.graph_util.remove_training_nodes(tf.graph_util.convert_variables_to_constants(sess, sess.graph.as_graph_def(), [output_name]))

# Convert Tensorflow frozen graph to UFF file
uff_model = uff.from_tensorflow(frozen_graph, output_file=file_path.replace('.h5', '.uff'))
11

from keras.models import load_model
file_path = ‘/home/cloud4/Akhtar_Vir_Env/my_new_app/People_Counting/yolo.h5’

sess = tf.Session(config=tf.ConfigProto(device_count={‘GPU’: 0})) # CPU only
K.set_session(sess)

K.set_learning_phase(0)
model = load_model(file_path, compile=False)
K.set_learning_phase(0)
output_name = model.output.op.name
input_name = model.input.op.name
frozen_graph = tf.graph_util.remove_training_nodes(tf.graph_util.convert_variables_to_constants(sess, sess.graph.as_graph_def(), [output_name]))

Convert Tensorflow frozen graph to UFF file

uff_model = uff.from_tensorflow(frozen_graph, output_file=file_path.replace(‘.h5’, ‘.uff’))

Error

AttributeError Traceback (most recent call last)
in ()
8 model = load_model(file_path, compile=False)
9 K.set_learning_phase(0)
—> 10 output_name = model.output.op.name
11 input_name = model.input.op.name
12 frozen_graph = tf.graph_util.remove_training_nodes(tf.graph_util.convert_variables_to_constants(sess, sess.graph.as_graph_def(), [output_name]))

AttributeError: ‘list’ object has no attribute ‘op’
Thanks

12

Can you give any examples about how to use graphsurgeon?

13

Hi
I am also facing the same error as you.I would like to know if you solved this issue or not?
If yes please let me know.

14

when I load a .uff model to tensorrt, which was converted from a keras resnet_unet model, error occured as following:
[E] [TRT] UffParser: Validator error: batch_normalization_3/cond/FusedBatchNorm/Switch: Unsupported operation _Switch
[E] Failure while parsing UFF file
[E] Model load failed

anybody know how to solve this problem?
thank you very much!

it seems that tensorrt cannot recognize Merge, AddV2 etc…

15

I’m having the same problem with my JetsonNano

I trained a classification network on keras with a MobileNetV2 model.

Then I converted the .h5 into .pb following GitHub - mpkuse/cartwheel_train: NetVLAD based weakly supervised learning whole image descriptor with street view data

Then I converted the .pb generated to the .uff file with the script (convert_to_uff.py) found on my Nano at
/usr/lib/python3.6/dist-packages/uff/bin

I called the .py with this parameters:
python3 convert_to_uff.py -t -o output.uff input.pb | tee logs.log

Then using C++ sample I’m trying to run inference with my new uff file, but I got this error:
[E] [TRT] UffParser: Validator error: Conv_1_bn_1/FusedBatchNormV3_1: Unsupported operation _FusedBatchNormV3

Can anybody please help me?

Note: I think it is not a matter of keras training because I can run inference on Windows using Intel Tools and on other linux devices using tensorflowlite

16

I achieve the conversion from my keras model based on MobileNetV2 to .pb using this python script on my Windows PC:

import tensorflow as tf

from keras import backend as K
from keras.models import load_model
import numpy as np


sess = tf.Session(config=tf.ConfigProto(device_count={'GPU': 0})) # CPU only
K.set_session(sess)
K.set_learning_phase(0)
model = load_model(filename_model_keras, compile=False)
K.set_learning_phase(0)

output_name = model.output.op.name
input_name = model.input.op.name

frozen_graph = tf.graph_util.remove_training_nodes(
    tf.graph_util.convert_variables_to_constants(
        sess, sess.graph.as_graph_def(), [output_name]
    )
)

tf.train.write_graph(frozen_graph, "./", filename_model_tf, as_text=False)

then I moved to my JetsonNano and I converted the .pb file to .uff in this way:

sudo python3 convert_to_uff.py -O output/Softmax --output output_folder/out.uff input_folder/tf_model.pb

The conversion seems correct.

Then I tested this model usign C++.
The result is strange. For every image tested, the result is that the most probable class in #1.

Any idea?

PS. I’m reading images with opencv in this way:

int ch_analysis = 3;
int h_analysis = 224;
int w_analysis = 224;

image = cv::imread(image_name, cv::IMREAD_COLOR);
cv::resize(image, image_input_cv, cv::Size(h_analysis, w_analysis));
int volChl = h_analysis * w_analysis;
for (int c = 0; c < ch_analysis; ++c)
{
	for (unsigned j = 0; j < volChl; ++j)
	{
	img_input[c*volChl + j + ch_analysis * h_analysis * w_analysis] =    float(image_input_cv.data[j*ch_analysis + 2 - c]);
	}
}

Can be this the problem?

thanks

17

The problem was the input order. Now with UffInputOrder::kNHWC instead of UffInputOrder::kNCHW it works correctly