Tf model converted to onnx/trt dimensions missmatch

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

nvcr.io/nvidia/deepstream:5.1-21.02-devel

I am trying to convert model from tf to onnx/trt and deploy on deepstream-app, so I did these steps:

  1. downloaded saved model from mobilenet_v3 | Kaggle
  2. I used examples from nvidia docs and these notebooks: tf → onnx → trt
  3. I have tried using the created intermediate onnx model with this config:
[property]
gpu-id=0
net-scale-factor=1
labelfile-path=../mobilenet/labels.txt
onnx-file=../mobilenet/mob.onnx
input-dims=3;224;224;0
batch-size=1
model-color-format=0
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=2
network-type=1
num-detected-classes=1001
interval=0
classifier-threshold=0.2

And when running with onnx the output is:

Input filename: /opt/nvidia/deepstream/deepstream-
5.1/sources/apps/main/mobilenet/mob.onnx
ONNX IR version: 0.0.4
Opset version: 9
Producer name: tf2onnx
Producer version: 1.9.2
Domain:
Model version: 0
Doc string:
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
INFO: …/nvdsinfer/nvdsinfer_func_utils.cpp:39 [TRT]: Detected 1 inputs and 1 output network tensors.
0:02:04.535115865 144 0x7ff1780022d0 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 2]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1749> [UID = 2]: serialize cuda engine to file: /opt/nvidia/deepstream/deepstream-5.1/sources/apps/main/mobilenet/mob.onnx_b1_gpu0_fp16.engine successfully
ERROR: …/nvdsinfer/nvdsinfer_func_utils.cpp:33 [TRT]: StatefulPartitionedCall/StatefulPartitionedCall/predict/MobilenetV3/Logits/Squeeze: reshaping failed for tensor: StatefulPartitionedCall/StatefulPartitionedCall/predict/MobilenetV3/Logits/Conv2d_1c_1x1/BiasAdd:0
ERROR: …/nvdsinfer/nvdsinfer_func_utils.cpp:33 [TRT]: shapeMachine.cpp (160) - Shape Error in executeReshape: reshape would change volume
ERROR: …/nvdsinfer/nvdsinfer_func_utils.cpp:33 [TRT]: Instruction: RESHAPE{1 1001 1 1} {8 1001}
INFO: …/nvdsinfer/nvdsinfer_model_builder.cpp:685 [Implicit Engine Info]: layers num: 2
0 INPUT kFLOAT inputs 224x224x3
1 OUTPUT kFLOAT logits 1001
0:02:04.539625870 144 0x7ff1780022d0 ERROR nvinfer gstnvinfer.cpp:613:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 2]: Error in NvDsInferContextImpl::preparePreprocess() <nvdsinfer_context_impl.cpp:875> [UID = 2]: RGB/BGR input format specified but network input channels is not 3
ERROR: nvdsinfer_context_impl.cpp:1158 Infer Context prepare preprocessing resource failed., nvinfer error:NVDSINFER_CONFIG_FAILED

And if I use trt generated model as model-engine-file I get something like this

WARNING: nvdsinfer_backend.cpp:162 Backend context bufferIdx(0) request dims:1x3x224x224 is out of range, [min: 8x224x224x3, max: 8x224x224x3]
0:00:00.966294670 193 0x7fbdd80022d0 WARN nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 2]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:1659> [UID = 2]: backend can not support dims:3x224x224

Since the examples that I used for convertion does not cover it, I am looking for some help here, how can I deploy tf savedmodel downloaded from the tfhub, how should I convert and what config to use in deepstream?

3

Hi,

Could you share the setup information with us first?

• Hardware Platform (Jetson / GPU)
• DeepStream Version
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Thanks.

4

I am using nvcr.io/nvidia/deepstream:5.1-21.02-devel docker image on Ubuntu20.04 so my setup is as defined in the docker image:

  • Ubuntu 20.04 (dGPU)
  • Deepstream 5.1
  • TensorRT 7.2.2
  • Nvidia drivers 460.91.03
5

Also I tried changing config to this:

[property]
gpu-id=0
net-scale-factor=1
labelfile-path=../mobilenet/labels.txt
onnx-file=../mobilenet/model7.onnx
#model-engine-file=../mobilenet/resnet_engine.trt
network-input-order=0
input-dims=3;224;224;0
model-color-format=0
network-mode=2
network-type=1
num-detected-classes=1000
interval=0
classifier-threshold=0.2

And I get this:

ERROR: ../nvdsinfer/nvdsinfer_func_utils.cpp:33 [TRT]: inputs: for dimension number 1 in profile 0 does not match network definition (got min=3, opt=3, max=3), expected min=opt=max=224).

Then I tried changing tensorrt to like this in the config:

#onnx-file=../mobilenet/model7.onnx
model-engine-file=../mobilenet/resnet_engine.trt

And the output is:

0:00:00.947075537   612 0x7f52140022d0 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 2]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1702> [UID = 2]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-5.1/sources/apps/main/mobilenet/resnet_engine.trt
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:685 [Implicit Engine Info]: layers num: 2
0   INPUT  kFLOAT inputs          224x224x3       
1   OUTPUT kFLOAT logits          1001            

WARNING: nvdsinfer_backend.cpp:162 Backend context bufferIdx(0) request dims:8x3x224x224 is out of range, [min: 8x224x224x3, max: 8x224x224x3]
0:00:00.947192727   612 0x7f52140022d0 WARN                 nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 2]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:1659> [UID = 2]: backend can not support dims:3x224x224
0:00:00.947199340   612 0x7f52140022d0 WARN                 nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<secondary_gie_0> NvDsInferContext[UID 2]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1814> [UID = 2]: deserialized backend context :/opt/nvidia/deepstream/deepstream-5.1/sources/apps/main/mobilenet/resnet_engine.trt failed to match config params, trying rebuild

No matter how I try to convert I cant get it to work. I tried changing:
network-input-order=0 to 1
and
input-dims=3;224;224;0 to 3;224;224;1

To try out a different input order but seems to have no effect.

6

Hi,

inputs: for dimension number 1 in profile 0 does not match network definition (got min=3, opt=3, max=3), expected min=opt=max=224).

Based on the above message, it seems that your onnx model is in the NHWC format.
Could you verify this first?

You can visualize the model with this website: https://netron.app/

Thanks.

7

Yes, the model is in NHCW format.
image

I converted the model like this (from the docs):

model_path = 'models/mob/'
subprocess.run(['python', '-m', 'tf2onnx.convert', '--saved-model', model_path, '--output', 'temp.onnx'])

onnx_model = onnx.load_model('temp.onnx')

BATCH_SIZE = 8
inputs = onnx_model.graph.input
for input in inputs:
    dim1 = input.type.tensor_type.shape.dim[0]
    dim1.dim_value = BATCH_SIZE


model_name = "model7.onnx"
onnx.save_model(onnx_model, model_name)
8

Hi,

Based on your snapshot, the model is NHWC format.
(C indicates channel and N is batchsize)

Since Deepstream expects NCHW format, could you convert the model into NCHW instead?
Please try this with the --inputs_as_nchw config in tf2onnx.

Thanks.

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.