Volume mismatch error at TensorRT engine

I am trying to run inference on Jetson TX2 for an image segmentation DNN. The network was written in TensorFlow. I was able to convert the trained network to UFF file, but when I try to create TensorRT engine following SampleUffMnist example (replacing the UFF file with mine and changing input and output node names), I get this error:

ERROR: UFFParser: Parser error: Reshape_1: Reshape: Volume mismatch
ERROR: sample_uff_mnist: Fail to parse
ERROR: sample_uff_mnist: Model load failed

What does this error mean and how can I resolve it?

Hi,

The volume mismatch error means that the reshape operation you are trying to perform on a tensor does not retain the same volume (as it must).

For example, you can reshape a tensor from 1,3,3 to 9,1,1 (the volume is 9 in both cases) but you will get this error if you try to reshape 1,3,3 to 2,2,2 (volume 9 to volume 8).

We suspect there is a mismatch between the network and your dimension specification.
Please double check it.

Thanks.

Hello AastaLL,

Thank you for your response. I attached a capture of the part of my network contianing Reshape_1 part. It shows input to reshape being 32x28x28x2, and output 25088x2, which seems to be the same volume? My network uses “tf.reshape(tensor,[-1, 2])” to do reshape, which does preserve volume. What else do I need to look at in this case?

I also attached the graph I am using in case that might be helpful.
Reshape_1.png
graph_optimized_test.pb.tar.gz (6.62 MB)

Hi,

We need more information to debug this issue.
Could you try to collect info-level log and share with us?

gLogger = Logger(nvinfer1::ILogger::Severity::kINFO);

Thanks

Here is the full output:

INFO: UFFParser: parsing Reshape
INFO: UFFParser: parsing conv2d/kernel
INFO: UFFParser: parsing cnn_conv_A1c/conv2d/Conv2D
INFO: UFFParser: parsing conv2d/bias
INFO: UFFParser: parsing cnn_conv_A1c/conv2d/BiasAdd
INFO: UFFParser: parsing batch_normalization/gamma
INFO: UFFParser: parsing batch_normalization/beta
INFO: UFFParser: parsing batch_normalization/moving_mean
INFO: UFFParser: parsing batch_normalization/moving_variance
INFO: UFFParser: parsing cnn_conv_A1c/batch_normalization/FusedBatchNorm
INFO: UFFParser: parsing cnn_conv_A1c/Relu
INFO: UFFParser: parsing conv2d_1/kernel
INFO: UFFParser: parsing cnn_conv_A2c/conv2d/Conv2D
INFO: UFFParser: parsing conv2d_1/bias
INFO: UFFParser: parsing cnn_conv_A2c/conv2d/BiasAdd
INFO: UFFParser: parsing batch_normalization_1/gamma
INFO: UFFParser: parsing batch_normalization_1/beta
INFO: UFFParser: parsing batch_normalization_1/moving_mean
INFO: UFFParser: parsing batch_normalization_1/moving_variance
INFO: UFFParser: parsing cnn_conv_A2c/batch_normalization/FusedBatchNorm
INFO: UFFParser: parsing cnn_conv_A2c/Relu
INFO: UFFParser: parsing max_pooling2d/MaxPool
INFO: UFFParser: parsing conv2d_2/kernel
INFO: UFFParser: parsing cnn_conv_B1c/conv2d/Conv2D
INFO: UFFParser: parsing conv2d_2/bias
INFO: UFFParser: parsing cnn_conv_B1c/conv2d/BiasAdd
INFO: UFFParser: parsing batch_normalization_2/gamma
INFO: UFFParser: parsing batch_normalization_2/beta
INFO: UFFParser: parsing batch_normalization_2/moving_mean
INFO: UFFParser: parsing batch_normalization_2/moving_variance
INFO: UFFParser: parsing cnn_conv_B1c/batch_normalization/FusedBatchNorm
INFO: UFFParser: parsing cnn_conv_B1c/Relu
INFO: UFFParser: parsing conv2d_3/kernel
INFO: UFFParser: parsing cnn_conv_B2c/conv2d/Conv2D
INFO: UFFParser: parsing conv2d_3/bias
INFO: UFFParser: parsing cnn_conv_B2c/conv2d/BiasAdd
INFO: UFFParser: parsing batch_normalization_3/gamma
INFO: UFFParser: parsing batch_normalization_3/beta
INFO: UFFParser: parsing batch_normalization_3/moving_mean
INFO: UFFParser: parsing batch_normalization_3/moving_variance
INFO: UFFParser: parsing cnn_conv_B2c/batch_normalization/FusedBatchNorm
INFO: UFFParser: parsing cnn_conv_B2c/Relu
INFO: UFFParser: parsing max_pooling2d_1/MaxPool
INFO: UFFParser: parsing conv2d_4/kernel
INFO: UFFParser: parsing cnn_conv_E1/conv2d/Conv2D
INFO: UFFParser: parsing conv2d_4/bias
INFO: UFFParser: parsing cnn_conv_E1/conv2d/BiasAdd
INFO: UFFParser: parsing batch_normalization_4/gamma
INFO: UFFParser: parsing batch_normalization_4/beta
INFO: UFFParser: parsing batch_normalization_4/moving_mean
INFO: UFFParser: parsing batch_normalization_4/moving_variance
INFO: UFFParser: parsing cnn_conv_E1/batch_normalization/FusedBatchNorm
INFO: UFFParser: parsing cnn_conv_E1/Relu
INFO: UFFParser: parsing conv2d_5/kernel
INFO: UFFParser: parsing cnn_conv_E2/conv2d/Conv2D
INFO: UFFParser: parsing conv2d_5/bias
INFO: UFFParser: parsing cnn_conv_E2/conv2d/BiasAdd
INFO: UFFParser: parsing batch_normalization_5/gamma
INFO: UFFParser: parsing batch_normalization_5/beta
INFO: UFFParser: parsing batch_normalization_5/moving_mean
INFO: UFFParser: parsing batch_normalization_5/moving_variance
INFO: UFFParser: parsing cnn_conv_E2/batch_normalization/FusedBatchNorm
INFO: UFFParser: parsing cnn_conv_E2/Relu
INFO: UFFParser: parsing conv2d_transpose/kernel
INFO: UFFParser: parsing conv2d_transpose/Shape
INFO: UFFParser: parsing conv2d_transpose/strided_slice/stack
INFO: UFFParser: parsing conv2d_transpose/strided_slice/stack_1
INFO: UFFParser: parsing conv2d_transpose/strided_slice/stack_2
INFO: UFFParser: parsing conv2d_transpose/strided_slice
INFO: UFFParser: parsing conv2d_transpose/strided_slice_1/stack
INFO: UFFParser: parsing conv2d_transpose/strided_slice_1/stack_1
INFO: UFFParser: parsing conv2d_transpose/strided_slice_1/stack_2
INFO: UFFParser: parsing conv2d_transpose/strided_slice_1
INFO: UFFParser: parsing conv2d_transpose/mul/y
INFO: UFFParser: parsing conv2d_transpose/mul
INFO: UFFParser: parsing conv2d_transpose/add/y
INFO: UFFParser: parsing conv2d_transpose/add
INFO: UFFParser: parsing conv2d_transpose/strided_slice_2/stack
INFO: UFFParser: parsing conv2d_transpose/strided_slice_2/stack_1
INFO: UFFParser: parsing conv2d_transpose/strided_slice_2/stack_2
INFO: UFFParser: parsing conv2d_transpose/strided_slice_2
INFO: UFFParser: parsing conv2d_transpose/mul_1/y
INFO: UFFParser: parsing conv2d_transpose/mul_1
INFO: UFFParser: parsing conv2d_transpose/add_1/y
INFO: UFFParser: parsing conv2d_transpose/add_1
INFO: UFFParser: parsing conv2d_transpose/stack/3
INFO: UFFParser: parsing conv2d_transpose/stack
INFO: UFFParser: parsing conv2d_transpose/conv2d_transpose
INFO: UFFParser: parsing conv2d_transpose/bias
INFO: UFFParser: parsing conv2d_transpose/BiasAdd
INFO: UFFParser: parsing concat
INFO: UFFParser: parsing conv2d_6/kernel
INFO: UFFParser: parsing cnn_conv_B1e/conv2d/Conv2D
INFO: UFFParser: parsing conv2d_6/bias
INFO: UFFParser: parsing cnn_conv_B1e/conv2d/BiasAdd
INFO: UFFParser: parsing batch_normalization_6/gamma
INFO: UFFParser: parsing batch_normalization_6/beta
INFO: UFFParser: parsing batch_normalization_6/moving_mean
INFO: UFFParser: parsing batch_normalization_6/moving_variance
INFO: UFFParser: parsing cnn_conv_B1e/batch_normalization/FusedBatchNorm
INFO: UFFParser: parsing cnn_conv_B1e/Relu
INFO: UFFParser: parsing conv2d_7/kernel
INFO: UFFParser: parsing cnn_conv_B2e/conv2d/Conv2D
INFO: UFFParser: parsing conv2d_7/bias
INFO: UFFParser: parsing cnn_conv_B2e/conv2d/BiasAdd
INFO: UFFParser: parsing batch_normalization_7/gamma
INFO: UFFParser: parsing batch_normalization_7/beta
INFO: UFFParser: parsing batch_normalization_7/moving_mean
INFO: UFFParser: parsing batch_normalization_7/moving_variance
INFO: UFFParser: parsing cnn_conv_B2e/batch_normalization/FusedBatchNorm
INFO: UFFParser: parsing cnn_conv_B2e/Relu
INFO: UFFParser: parsing conv2d_transpose_1/kernel
INFO: UFFParser: parsing conv2d_transpose_1/Shape
INFO: UFFParser: parsing conv2d_transpose_1/strided_slice/stack
INFO: UFFParser: parsing conv2d_transpose_1/strided_slice/stack_1
INFO: UFFParser: parsing conv2d_transpose_1/strided_slice/stack_2
INFO: UFFParser: parsing conv2d_transpose_1/strided_slice
INFO: UFFParser: parsing conv2d_transpose_1/strided_slice_1/stack
INFO: UFFParser: parsing conv2d_transpose_1/strided_slice_1/stack_1
INFO: UFFParser: parsing conv2d_transpose_1/strided_slice_1/stack_2
INFO: UFFParser: parsing conv2d_transpose_1/strided_slice_1
INFO: UFFParser: parsing conv2d_transpose_1/mul/y
INFO: UFFParser: parsing conv2d_transpose_1/mul
INFO: UFFParser: parsing conv2d_transpose_1/add/y
INFO: UFFParser: parsing conv2d_transpose_1/add
INFO: UFFParser: parsing conv2d_transpose_1/strided_slice_2/stack
INFO: UFFParser: parsing conv2d_transpose_1/strided_slice_2/stack_1
INFO: UFFParser: parsing conv2d_transpose_1/strided_slice_2/stack_2
INFO: UFFParser: parsing conv2d_transpose_1/strided_slice_2
INFO: UFFParser: parsing conv2d_transpose_1/mul_1/y
INFO: UFFParser: parsing conv2d_transpose_1/mul_1
INFO: UFFParser: parsing conv2d_transpose_1/add_1/y
INFO: UFFParser: parsing conv2d_transpose_1/add_1
INFO: UFFParser: parsing conv2d_transpose_1/stack/3
INFO: UFFParser: parsing conv2d_transpose_1/stack
INFO: UFFParser: parsing conv2d_transpose_1/conv2d_transpose
INFO: UFFParser: parsing conv2d_transpose_1/bias
INFO: UFFParser: parsing conv2d_transpose_1/BiasAdd
INFO: UFFParser: parsing concat_1
INFO: UFFParser: parsing conv2d_8/kernel
INFO: UFFParser: parsing cnn_conv_A1e/conv2d/Conv2D
INFO: UFFParser: parsing conv2d_8/bias
INFO: UFFParser: parsing cnn_conv_A1e/conv2d/BiasAdd
INFO: UFFParser: parsing batch_normalization_8/gamma
INFO: UFFParser: parsing batch_normalization_8/beta
INFO: UFFParser: parsing batch_normalization_8/moving_mean
INFO: UFFParser: parsing batch_normalization_8/moving_variance
INFO: UFFParser: parsing cnn_conv_A1e/batch_normalization/FusedBatchNorm
INFO: UFFParser: parsing cnn_conv_A1e/Relu
INFO: UFFParser: parsing conv2d_9/kernel
INFO: UFFParser: parsing cnn_conv_A2e/conv2d/Conv2D
INFO: UFFParser: parsing conv2d_9/bias
INFO: UFFParser: parsing cnn_conv_A2e/conv2d/BiasAdd
INFO: UFFParser: parsing batch_normalization_9/gamma
INFO: UFFParser: parsing batch_normalization_9/beta
INFO: UFFParser: parsing batch_normalization_9/moving_mean
INFO: UFFParser: parsing batch_normalization_9/moving_variance
INFO: UFFParser: parsing cnn_conv_A2e/batch_normalization/FusedBatchNorm
INFO: UFFParser: parsing cnn_conv_A2e/Relu
INFO: UFFParser: parsing conv2d_10/kernel
INFO: UFFParser: parsing output_score/conv2d/Conv2D
INFO: UFFParser: parsing conv2d_10/bias
INFO: UFFParser: parsing output_score/conv2d/BiasAdd
INFO: UFFParser: parsing output_score/output_relu
INFO: UFFParser: parsing Reshape_1/shape
INFO: UFFParser: parsing Reshape_1
ERROR: UFFParser: Parser error: Reshape_1: Reshape: Volume mismatch
ERROR: sample_uff_mnist: Fail to parse
ERROR: sample_uff_mnist: Model load failed

Thank you

Hi,

In TensorRT 3.0, reshape applies only on constant weights.
Tensor reshape is automatically dropped when importing it from UFF to TensorRT engine.

Since TensorRT supports multiple inputs/outputs.
A quick workaround is to use multiple outputs and receive output_score separately

Thanks.

Hello, Aastalll,

Could you give an explanation about how TensorRT engine works when encounter a fully connection layers, which have to use reshape operation to reshape the features to flatten. and it indeed works for tensorRT3.0.

Hi,

Please check our blog for information:
https://devblogs.nvidia.com/deploying-deep-learning-nvidia-tensorrt/

Thanks.