Hi all,
I found similar topics in the forum but none was the solution for my problem, I already tried to reshape and transpose the input according to documentation and samples but the output of the model is different to the original one.
Data:
in the file attached you can find the model and the transformations in pb,uff, trt-engine as well as a pickle file containing some sample data, the jupyter notebook connects all that, including the code to execute tranfomrations and inference.
Problem Description:
Following a simple CNN model defined and traine via tf.keras, converted with trt.lite.Engine into a TRT model.
The model predicts the shape of objects either BOX or CYLINDER (one hot encoded)
input is a depth image, therefore greyscale, only one channel HWC(299,299,1), in TRT it is CHW
The trt predictions are different from the TF which are considered as ground truth
convert-to-uff output:
(venv) ml@mltest2:~/tensorflow/venv/data/jupyter$ convert-to-uff tensorflow --input-file saved_model_small_freeze.pb -l
/home/ml/tensorflow/venv/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
return f(*args, **kwds)
Loading saved_model_small_freeze.pb
Automatically deduced output nodes: dense/Softmax
1 Placeholder: “conv2d_input”
2 Const: “conv2d/kernel”
3 Const: “conv2d/bias”
4 Conv2D: “conv2d/Conv2D”
5 BiasAdd: “conv2d/BiasAdd”
6 Relu: “conv2d/Relu”
7 MaxPool: “max_pooling2d/MaxPool”
8 Const: “conv2d_1/kernel”
9 Const: “conv2d_1/bias”
10 Conv2D: “conv2d_1/Conv2D”
11 BiasAdd: “conv2d_1/BiasAdd”
12 Relu: “conv2d_1/Relu”
13 MaxPool: “max_pooling2d_1/MaxPool”
14 Const: “conv2d_2/kernel”
15 Const: “conv2d_2/bias”
16 Conv2D: “conv2d_2/Conv2D”
17 BiasAdd: “conv2d_2/BiasAdd”
18 Relu: “conv2d_2/Relu”
19 MaxPool: “max_pooling2d_2/MaxPool”
20 Shape: “flatten/Shape”
21 Const: “flatten/strided_slice/stack”
22 Const: “flatten/strided_slice/stack_1”
23 Const: “flatten/strided_slice/stack_2”
24 StridedSlice: “flatten/strided_slice”
25 Const: “flatten/Reshape/shape/1”
26 Pack: “flatten/Reshape/shape”
27 Reshape: “flatten/Reshape”
28 Const: “dense/kernel”
29 Const: “dense/bias”
30 MatMul: “dense/MatMul”
31 BiasAdd: “dense/BiasAdd”
32 Softmax: “dense/Softmax”
Keras Model:
Layer (type) Output Shape Param #
conv2d (Conv2D) (None, 295, 295, 32) 832
max_pooling2d (MaxPooling2D) (None, 97, 97, 32) 0
conv2d_1 (Conv2D) (None, 93, 93, 64) 51264
max_pooling2d_1 (MaxPooling2 (None, 46, 46, 64) 0
conv2d_2 (Conv2D) (None, 44, 44, 64) 36928
max_pooling2d_2 (MaxPooling2 (None, 22, 22, 64) 0
flatten (Flatten) (None, 30976) 0
dense (Dense) (None, 2) 61954
Total params: 150,978
Trainable params: 150,978
Non-trainable params: 0
Here the results of the attached pickle samples, as you the predictions are different in TRT and TF
0-4 are real samples (TF is considered ground truth) 5 white image 6 black image 7 grey image
# run test
for i in test_samples:
result_trt = engine_loaded.infer(i.reshape(1,299,299))
#print(result)
print("TRT: %s" % np.asarray(result_trt).reshape(2))
result_tf = model_loaded.predict(i.reshape((1,299,299,1)))
print("TF : %s" % result_tf.reshape(2))
print(np.argmax(result_trt), np.argmax(result_tf))
print("-----------------------------")
TRT: [0.9007671 0.09923294]
TF : [0.99828416 0.00171584]
0 0
TRT: [0.04684561 0.9531544 ]
TF : [1.000000e+00 3.279911e-08]
1 0
TRT: [0.27094778 0.7290522 ]
TF : [9.9988842e-01 1.1151263e-04]
1 0
TRT: [0.4547093 0.5452907]
TF : [9.9962366e-01 3.7629422e-04]
1 0
TRT: [0.11016078 0.88983923]
TF : [0.9982955 0.00170457]
1 0
TRT: [0.49179485 0.50820524]
TF : [0.4918013 0.5081987]
1 1
TRT: [0.49184766 0.50815237]
TF : [0.49184763 0.50815237]
1 1
TRT: [0.49179485 0.50820524]
TF : [0.4918013 0.5081987]
1 1
command for TRT lite engine transformation
engine_lite = trt.lite.Engine(framework="tf",path="saved_model_small_freeze.pb",max_batch_size=1,input_nodes={"conv2d_input":(1,299,299)},output_nodes=["dense/Softmax"])
The prediction in tensorflow via loading the frozen graph and evaluate via session.run is still correct.
The ‘non-lite’ way to create the engine also gives the same wrong results.
Setup:
Tensorflow 1.9
Cuda 9.2
TensorRT 4.0.1.16
Ubuntu 16.0.4
Best,
Johannes