Input shape axis 0 must equal 8, got shape [5,600,1024,3]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) T4
• DeepStream Version container image nvcr.io/nvidia/deepstream:5.0.1-20.09-triton
• JetPack Version (valid for Jetson only) n/a
• TensorRT Version 7.0
• NVIDIA GPU Driver Version (valid for GPU only) 450.51.06
**• TensorFlow version (use command below): 1.15.4
**• Python version: 3.6|
**• CUDA/cuDNN version: 10.2 / 7.6.5 |
**• TensorFlow installed from (source or binary): container image nvcr.io/nvidia/tensorflow:20.02-tf1-py3

• Issue Type( questions, new requirements, bugs)

I got input shape axis issues deploying an optimized model to INT8 with TF-TRT (TF object detection faster_rcnn_inception_v2_coco) on DS-Trion.

Describe the current behavior

**• How to reproduce the issue ?
Step# 1: Optimize the model to INT8
To optimize the model I used the object_detection script, although the test was completed successfully I got the below warning about incomplete shapes:

$ python3 object_detection.py --input_saved_model_dir /workspace/faster_rcnn_in ception_v2_coco_2018_01_28/saved_model --output_saved_model_dir /workspace/ --optimize_offline --data_dir /workspace/data/coco/val2017 --annotation_path /workspace/ data/coco/annotations/instances_val2017.json --calib_data_dir /workspace/data/coco/val2017/ --batch_size 8 --use_trt --mode 'benchmark' --precision INT8 --input_size 600 --mini mum_segment_size 3 --num_calib_inputs 800

Tracelog:

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/profiler/internal/flops_registry.py:142: tensor_shape_from_node_def_name (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.tensor_shape_from_node_def_name`
**1752 ops no flops stats due to incomplete shapes.
Parsing Inputs...
Incomplete shape.**
.
.
.
orm/default/dso_loader.cc:49] Successfully opened dynamic library libnvinfer.so.7
WARNING:tensorflow:INT8 precision mode with calibration is supported with dynamic TRT ops only. Disregarding is_dynamic_op parameter.
.
.
.
DONE (t=1.91s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.245
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.390
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.265
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.054
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.262
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.446
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.223
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.300
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.302
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.062
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.314
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.553
{
    "avg_latency_ms": 246.9332789430524,
    "avg_throughput_fps": 32.397415343295854,
    "map": 0.24546921626010776
}
ASSERTION PASSED: statistics['map'] > (0.243 - 0.01)

The model was optimized successfully; however, when I tried to deploy the model with Nvidia DeepStream I got the below input shape errors, even though the engine was created with max batch size 8, it seems the object_detection.py script didn’t assign the parameter input_shape for some operations

Step #2: Deploy the optimized model on DS-Triton
$ deepstream-app -c source1_primary_faster_rcnn_inception_v2.txt
Tracelog Error:

Runtime commands:
        h: Print this help
        q: Quit

        p: Pause
        r: Resume


**PERF:  FPS 0 (Avg)
**PERF:  0.00 (0.00)
** INFO: <bus_callback:181>: Pipeline ready

** INFO: <bus_callback:167>: Pipeline running

ERROR: infer_trtis_server.cpp:276 TRTIS: failed to get response status, trtis_err_str:INTERNAL, err_msg:2 root error(s) found.
  (0) Invalid argument: Input shape axis 0 must equal 8, got shape [5,600,1024,3]
         [[{{node Preprocessor/unstack}}]]
         [[detection_scores/_349]]
  (1) Invalid argument: Input shape axis 0 must equal 8, got shape [5,600,1024,3]
         [[{{node Preprocessor/unstack}}]]
0 successful operations.
0 derived errors ignored.
ERROR: infer_trtis_backend.cpp:532 TRTIS server failed to parse response with request-id:1 model:
0:01:51.458210397   508 0x7f85780022a0 WARN           nvinferserver gstnvinferserver.cpp:519:gst_nvinfer_server_push_buffer:<primary_gie> error: inference failed with unique-id:1
ERROR from primary_gie: inference failed with unique-id:1
Debug info: gstnvinferserver.cpp(519): gst_nvinfer_server_push_buffer (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInferServer:primary_gie
Quitting
ERROR: infer_trtis_server.cpp:276 TRTIS: failed to get response status, trtis_err_str:INTERNAL, err_msg:2 root error(s) found.
  (0) Invalid argument: Input shape axis 0 must equal 8, got shape [5,600,1024,3]
         [[{{node Preprocessor/unstack}}]]
         [[SecondStagePostprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/ClipToWindow/TRTEngineOp_10/_289]]
  (1) Invalid argument: Input shape axis 0 must equal 8, got shape [5,600,1024,3]
         [[{{node Preprocessor/unstack}}]]
0 successful operations.
0 derived errors ignored.
ERROR: infer_trtis_backend.cpp:532 TRTIS server failed to parse response with request-id:2 model:
ERROR from qtdemux0: Internal data stream error.
Debug info: qtdemux.c(6073): gst_qtdemux_loop (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstURIDecodeBin:src_elem/GstDecodeBin:decodebin0/GstQTDemux:qtdemux0:
streaming stopped, reason custom-error (-112)
I0301 21:00:58.391764 508 model_repository_manager.cc:708] unloading: faster_rcnn_inception_v2:21
I0301 21:00:58.733155 508 model_repository_manager.cc:816] successfully unloaded 'faster_rcnn_inception_v2' version 21
I0301 21:00:58.733799 508 server.cc:179] Waiting for in-flight inferences to complete.
I0301 21:00:58.733814 508 server.cc:194] Timeout 30: Found 0 live models and 0 in-flight requests
App run failed

Could you please suggest how to solve this input shapes issue?

Sorry for the late response, it this still an issue to support?

Hi @kayccc, the same problem as reported in the issue here. Is there some sample to convert a yolov model to TensoRT int8 mode (extra int8 calibration needed) with dynamic shape?

Hi @virsg
Sorry for long delay!
Not sure if it’;s possible to share the model to repo?

Thanks!

Hi @mchi, the repo model is in TensorFlow Model Zoo here

Hi @virsg ,
DS 5.0 nvinferserver does not support dynamic batch, but DS5.1 supports.
So, could you upgrade to DS5.1 to take a try?

Thanks!

Hi @mchi, I have already tried DS5.1 with dynamic batching as well as with static batching (BS>1), and is it not working either, could you please take a look at the new issue reported here ?

Ok, will take a try in these days