Error in inferencing using a onnx faster rcnn model

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
• DeepStream Version
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Hardware Platform - T4 GPU (AWS Deep Learning AMI)
DeepStream Version - 5.1 (with Triton)
GPU version - 11.1
Issue Type - Error

I have been trying to run a model that I converted from pytorch to onnx on triton in Deepstream SDK. But I was hit by the following error:

I0317 18:31:58.856597 84 model_repository_manager.cc:810] loading: faster_rcnn_inception_v2:1
I0317 18:31:58.888504 84 onnxruntime.cc:1712] TRITONBACKEND_Initialize: onnxruntime
I0317 18:31:58.888532 84 onnxruntime.cc:1725] Triton TRITONBACKEND API version: 1.0
I0317 18:31:58.888547 84 onnxruntime.cc:1731] 'onnxruntime' TRITONBACKEND API version: 1.0
I0317 18:31:58.896717 84 onnxruntime.cc:1773] TRITONBACKEND_ModelInitialize: faster_rcnn_inception_v2 (version 1)
WARNING: Since openmp is enabled in this build, this API cannot be used to configure intra op num threads. Please use the openmp environment variables to control the number of threads.
I0317 18:31:58.897804 84 onnxruntime.cc:372] skipping model configuration auto-complete for 'faster_rcnn_inception_v2': max_batch_size, inupts or outputs already specified
I0317 18:31:58.898338 84 onnxruntime.cc:1817] TRITONBACKEND_ModelInstanceInitialize: faster_rcnn_inception_v2_0 (GPU device 0)
I0317 18:32:00.833758 84 onnxruntime.cc:1848] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0317 18:32:00.833808 84 onnxruntime.cc:1793] TRITONBACKEND_ModelFinalize: delete model state
E0317 18:32:00.834200 84 model_repository_manager.cc:986] failed to load 'faster_rcnn_inception_v2' version 1: Invalid argument: unexpected inference input 'image_tensor', allowed inputs are: image
ERROR: infer_trtis_server.cpp:1044 Triton: failed to load model faster_rcnn_inception_v2, triton_err_str:Invalid argument, err_msg:load failed for model 'faster_rcnn_inception_v2': version 1: Invalid argument: unexpected inference input 'image_tensor', allowed inputs are: image;

ERROR: infer_trtis_backend.cpp:45 failed to load model: faster_rcnn_inception_v2, nvinfer error:NVDSINFER_TRTIS_ERROR
ERROR: infer_trtis_backend.cpp:184 failed to initialize backend while ensuring model:faster_rcnn_inception_v2 ready, nvinfer error:NVDSINFER_TRTIS_ERROR
0:00:05.932771919    84 0x55d893c13ea0 ERROR          nvinferserver gstnvinferserver.cpp:362:gst_nvinfer_server_logger:<primary_gie> nvinferserver[UID 1]: Error in createNNBackend() <infer_trtis_context.cpp:246> [UID = 1]: failed to initialize trtis backend for model:faster_rcnn_inception_v2, nvinfer error:NVDSINFER_TRTIS_ERROR
I0317 18:32:00.834392 84 server.cc:280] Waiting for in-flight requests to complete.
I0317 18:32:00.834410 84 server.cc:295] Timeout 30: Found 0 live models and 0 in-flight non-inference requests
0:00:05.932895100    84 0x55d893c13ea0 ERROR          nvinferserver gstnvinferserver.cpp:362:gst_nvinfer_server_logger:<primary_gie> nvinferserver[UID 1]: Error in initialize() <infer_base_context.cpp:81> [UID = 1]: create nn-backend failed, check config file settings, nvinfer error:NVDSINFER_TRTIS_ERROR
0:00:05.932909055    84 0x55d893c13ea0 WARN           nvinferserver gstnvinferserver_impl.cpp:439:start:<primary_gie> error: Failed to initialize InferTrtIsContext
0:00:05.932919355    84 0x55d893c13ea0 WARN           nvinferserver gstnvinferserver_impl.cpp:439:start:<primary_gie> error: Config file path: /opt/nvidia/deepstream/deepstream-5.1/samples/configs/deepstream-app-trtis/config_infer_primary_detector_faster_rcnn.txt
0:00:05.937387478    84 0x55d893c13ea0 WARN           nvinferserver gstnvinferserver.cpp:460:gst_nvinfer_server_start:<primary_gie> error: gstnvinferserver_impl start failed
** ERROR: <main:655>: Failed to set pipeline to PAUSED
Quitting
ERROR from primary_gie: Failed to initialize InferTrtIsContext
Debug info: gstnvinferserver_impl.cpp(439): start (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInferServer:primary_gie:
Config file path: /opt/nvidia/deepstream/deepstream-5.1/samples/configs/deepstream-app-trtis/config_infer_primary_detector_faster_rcnn.txt
ERROR from primary_gie: gstnvinferserver_impl start failed
Debug info: gstnvinferserver.cpp(460): gst_nvinfer_server_start (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInferServer:primary_gie
App run failed

I get the same error regardless I set strict_model_config as true or false. What could this error mean?

Could you check the input of your model?

Hey I was able to fix the problem above but I have a new problem now after fixing this:

I0321 18:00:36.647739 308 model_repository_manager.cc:810] loading: faster_rcnn_inception_v2:1
I0321 18:00:36.681473 308 onnxruntime.cc:1712] TRITONBACKEND_Initialize: onnxruntime
I0321 18:00:36.681495 308 onnxruntime.cc:1725] Triton TRITONBACKEND API version: 1.0
I0321 18:00:36.681508 308 onnxruntime.cc:1731] 'onnxruntime' TRITONBACKEND API version: 1.0
I0321 18:00:36.689881 308 onnxruntime.cc:1773] TRITONBACKEND_ModelInitialize: faster_rcnn_inception_v2 (version 1)
WARNING: Since openmp is enabled in this build, this API cannot be used to configure intra op num threads. Please use the openmp environment variables to control the number of threads.
I0321 18:00:38.076074 308 onnxruntime.cc:1817] TRITONBACKEND_ModelInstanceInitialize: faster_rcnn_inception_v2 (GPU device 0)
I0321 18:00:39.883826 308 model_repository_manager.cc:983] successfully loaded 'faster_rcnn_inception_v2' version 1
INFO: infer_trtis_backend.cpp:206 TrtISBackend id:1 initialized model: faster_rcnn_inception_v2
0:00:06.621378191   308 0x56078fbb2cf0 ERROR          nvinferserver gstnvinferserver.cpp:362:gst_nvinfer_server_logger:<primary_gie> nvinferserver[UID 1]: Error in specifyBackendDims() <infer_trtis_context.cpp:143> [UID = 1]: failed to create trtis backend on model:faster_rcnn_inception_v2 when specify input:image with wrong dims:1x3x-1x-1
0:00:06.621408038   308 0x56078fbb2cf0 ERROR          nvinferserver gstnvinferserver.cpp:362:gst_nvinfer_server_logger:<primary_gie> nvinferserver[UID 1]: Error in createNNBackend() <infer_trtis_context.cpp:251> [UID = 1]: failed to specify trtis backend input dims for model:faster_rcnn_inception_v2, nvinfer error:NVDSINFER_CONFIG_FAILED
I0321 18:00:39.887523 308 model_repository_manager.cc:837] unloading: faster_rcnn_inception_v2:1
I0321 18:00:39.887756 308 onnxruntime.cc:1848] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0321 18:00:39.887870 308 server.cc:280] Waiting for in-flight requests to complete.
I0321 18:00:39.887940 308 server.cc:295] Timeout 30: Found 1 live models and 0 in-flight non-inference requests
I0321 18:00:39.898447 308 onnxruntime.cc:1793] TRITONBACKEND_ModelFinalize: delete model state
I0321 18:00:39.902313 308 model_repository_manager.cc:966] successfully unloaded 'faster_rcnn_inception_v2' version 1
I0321 18:00:40.888046 308 server.cc:295] Timeout 29: Found 0 live models and 0 in-flight non-inference requests
0:00:07.622104433   308 0x56078fbb2cf0 ERROR          nvinferserver gstnvinferserver.cpp:362:gst_nvinfer_server_logger:<primary_gie> nvinferserver[UID 1]: Error in initialize() <infer_base_context.cpp:81> [UID = 1]: create nn-backend failed, check config file settings, nvinfer error:NVDSINFER_CONFIG_FAILED
0:00:07.622134604   308 0x56078fbb2cf0 WARN           nvinferserver gstnvinferserver_impl.cpp:439:start:<primary_gie> error: Failed to initialize InferTrtIsContext
0:00:07.622176305   308 0x56078fbb2cf0 WARN           nvinferserver gstnvinferserver_impl.cpp:439:start:<primary_gie> error: Config file path: /opt/nvidia/deepstream/deepstream-5.1/samples/configs/deepstream-app-trtis/config_infer_primary_detector_faster_rcnn.txt
0:00:07.622637690   308 0x56078fbb2cf0 WARN           nvinferserver gstnvinferserver.cpp:460:gst_nvinfer_server_start:<primary_gie> error: gstnvinferserver_impl start failed
** ERROR: <main:655>: Failed to set pipeline to PAUSED
Quitting
ERROR from primary_gie: Failed to initialize InferTrtIsContext
Debug info: gstnvinferserver_impl.cpp(439): start (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInferServer:primary_gie:
Config file path: /opt/nvidia/deepstream/deepstream-5.1/samples/configs/deepstream-app-trtis/config_infer_primary_detector_faster_rcnn.txt
ERROR from primary_gie: gstnvinferserver_impl start failed
Debug info: gstnvinferserver.cpp(460): gst_nvinfer_server_start (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInferServer:primary_gie
App run failed

Here is the config file:

name: "faster_rcnn_inception_v2"
platform: "onnxruntime_onnx" 
input [
  {
    name: "image"
    data_type: TYPE_FP32 
    dims: [ 1, 3, -1, -1 ] 
  }
]
output [
  {
    name: "boxes" 
    data_type: TYPE_FP32
    dims: [ -1, 5 ]  
   
  },
  {
    name: "labels" 
    data_type: TYPE_INT64
    dims: [ -1 ]
  }
]
version_policy: { specific {versions: 1}}
instance_group [
  {
    kind: KIND_GPU
    count: 1
    gpus: [ 0 ]
  }
]

The input that I am using is a 720x1280 video. The model apparently supports variable sized input, if I give it any other dimension it throws an error that the model expects [ 1, 3, -1, -1] as input. This config file was as a result of fixing my model’s requirements yet I get

gstnvinferserver.cpp:362:gst_nvinfer_server_logger:<primary_gie> nvinferserver[UID 1]: Error in specifyBackendDims() <infer_trtis_context.cpp:143> [UID = 1]: failed to create trtis backend on model:faster_rcnn_inception_v2 when specify input:image with wrong dims:1x3x-1x-1

in the above error.

Hi @yash.mandilwar ,
Could you try changing above “dims: [1,3,-1,-1]” to “dims: [ 1, 3, 720, 1280]” and see what error it will report?

Thanks!

Hi!

I did the changes you mentioned. Here is the error:

I0325 11:47:40.938002 84 model_repository_manager.cc:810] loading: faster_rcnn_inception_v2:1
I0325 11:47:41.086747 84 onnxruntime.cc:1712] TRITONBACKEND_Initialize: onnxruntime
I0325 11:47:41.086770 84 onnxruntime.cc:1725] Triton TRITONBACKEND API version: 1.0
I0325 11:47:41.086781 84 onnxruntime.cc:1731] 'onnxruntime' TRITONBACKEND API version: 1.0
I0325 11:47:41.116852 84 onnxruntime.cc:1773] TRITONBACKEND_ModelInitialize: faster_rcnn_inception_v2 (version 1)
WARNING: Since openmp is enabled in this build, this API cannot be used to configure intra op num threads. Please use the openmp environment variables to control the number of threads.
I0325 11:47:41.119538 84 onnxruntime.cc:1817] TRITONBACKEND_ModelInstanceInitialize: faster_rcnn_inception_v2_0 (GPU device 0)
I0325 11:47:46.816732 84 model_repository_manager.cc:983] successfully loaded 'faster_rcnn_inception_v2' version 1
INFO: infer_trtis_backend.cpp:206 TrtISBackend id:1 initialized model: faster_rcnn_inception_v2
0:00:22.653728108    84 0x55d74b5fae90 ERROR          nvinferserver gstnvinferserver.cpp:362:gst_nvinfer_server_logger:<primary_gie> nvinferserver[UID 1]: Error in fixateInferenceInfo() <infer_cuda_context.cpp:164> [UID = 1]: InferContext(uid:1) cannot figure out input tensor order, please specify in config file(preprocess.)
0:00:22.653756737    84 0x55d74b5fae90 ERROR          nvinferserver gstnvinferserver.cpp:362:gst_nvinfer_server_logger:<primary_gie> nvinferserver[UID 1]: Error in initialize() <infer_base_context.cpp:86> [UID = 1]: Infer context faied to initialize inference information, nvinfer error:NVDSINFER_CONFIG_FAILED
0:00:22.653779812    84 0x55d74b5fae90 WARN           nvinferserver gstnvinferserver_impl.cpp:439:start:<primary_gie> error: Failed to initialize InferTrtIsContext
0:00:22.653792037    84 0x55d74b5fae90 WARN           nvinferserver gstnvinferserver_impl.cpp:439:start:<primary_gie> error: Config file path: /opt/nvidia/deepstream/deepstream-5.1/samples/configs/deepstream-app-trtis/config_infer_primary_detector_faster_rcnn.txt
I0325 11:47:54.725885 84 model_repository_manager.cc:837] unloading: faster_rcnn_inception_v2:1
I0325 11:47:54.726172 84 onnxruntime.cc:1848] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0325 11:47:54.733989 84 server.cc:280] Waiting for in-flight requests to complete.
I0325 11:47:54.734022 84 server.cc:295] Timeout 30: Found 1 live models and 0 in-flight non-inference requests
I0325 11:47:54.744633 84 onnxruntime.cc:1793] TRITONBACKEND_ModelFinalize: delete model state
I0325 11:47:54.756427 84 model_repository_manager.cc:966] successfully unloaded 'faster_rcnn_inception_v2' version 1
I0325 11:47:55.734115 84 server.cc:295] Timeout 29: Found 0 live models and 0 in-flight non-inference requests
0:00:23.662626858    84 0x55d74b5fae90 WARN           nvinferserver gstnvinferserver.cpp:460:gst_nvinfer_server_start:<primary_gie> error: gstnvinferserver_impl start failed
** ERROR: <main:655>: Failed to set pipeline to PAUSED
Quitting
ERROR from primary_gie: Failed to initialize InferTrtIsContext
Debug info: gstnvinferserver_impl.cpp(439): start (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInferServer:primary_gie:
Config file path: /opt/nvidia/deepstream/deepstream-5.1/samples/configs/deepstream-app-trtis/config_infer_primary_detector_faster_rcnn.txt
ERROR from primary_gie: gstnvinferserver_impl start failed
Debug info: gstnvinferserver.cpp(460): gst_nvinfer_server_start (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInferServer:primary_gie
App run failed

This is my preprocess in config_infer_primary_detector:

preprocess {
    network_format: IMAGE_FORMAT_RGB
    tensor_order: TENSOR_ORDER_NONE
    maintain_aspect_ratio: 0
    frame_scaling_hw: FRAME_SCALING_HW_DEFAULT
    frame_scaling_filter: 1
    normalize {
      scale_factor: 1.0
      channel_offsets: [0, 0, 0]
    }
  }

above error is because the input tensor_order is NONE as below, please try TENSOR_ORDER_LINEAR or TENSOR_ORDER_NHWC

On setting tensor_order to TENSOR_ORDER_LINEAR, this happens:

I0325 16:29:29.019518 102 model_repository_manager.cc:810] loading: faster_rcnn_inception_v2:1
I0325 16:29:29.052134 102 onnxruntime.cc:1712] TRITONBACKEND_Initialize: onnxruntime
I0325 16:29:29.052165 102 onnxruntime.cc:1725] Triton TRITONBACKEND API version: 1.0
I0325 16:29:29.052178 102 onnxruntime.cc:1731] 'onnxruntime' TRITONBACKEND API version: 1.0
I0325 16:29:29.060584 102 onnxruntime.cc:1773] TRITONBACKEND_ModelInitialize: faster_rcnn_inception_v2 (version 1)
WARNING: Since openmp is enabled in this build, this API cannot be used to configure intra op num threads. Please use the openmp environment variables to control the number of threads.
I0325 16:29:29.061684 102 onnxruntime.cc:1817] TRITONBACKEND_ModelInstanceInitialize: faster_rcnn_inception_v2_0 (GPU device 0)
I0325 16:29:30.959027 102 model_repository_manager.cc:983] successfully loaded 'faster_rcnn_inception_v2' version 1
INFO: infer_trtis_backend.cpp:206 TrtISBackend id:1 initialized model: faster_rcnn_inception_v2
deepstream-app: infer_cuda_utils.cpp:91: nvdsinferserver::CudaTensorBuf::CudaTensorBuf(const nvdsinferserver::InferDims&, nvdsinferserver::InferDataType, int, const string&, nvdsinferserver::InferMemType, int, bool): Assertion `bufBytes > 0' failed.
Aborted (core dumped)

On setting tensor_order to TENSOR_ORDER_NHWC I get this error:

I0325 16:28:10.440288 82 model_repository_manager.cc:810] loading: faster_rcnn_inception_v2:1
I0325 16:28:10.600126 82 onnxruntime.cc:1712] TRITONBACKEND_Initialize: onnxruntime
I0325 16:28:10.600150 82 onnxruntime.cc:1725] Triton TRITONBACKEND API version: 1.0
I0325 16:28:10.600157 82 onnxruntime.cc:1731] 'onnxruntime' TRITONBACKEND API version: 1.0
I0325 16:28:10.641162 82 onnxruntime.cc:1773] TRITONBACKEND_ModelInitialize: faster_rcnn_inception_v2 (version 1)
WARNING: Since openmp is enabled in this build, this API cannot be used to configure intra op num threads. Please use the openmp environment variables to control the number of threads.
I0325 16:28:10.644867 82 onnxruntime.cc:1817] TRITONBACKEND_ModelInstanceInitialize: faster_rcnn_inception_v2_0 (GPU device 0)
I0325 16:28:16.443462 82 model_repository_manager.cc:983] successfully loaded 'faster_rcnn_inception_v2' version 1
INFO: infer_trtis_backend.cpp:206 TrtISBackend id:1 initialized model: faster_rcnn_inception_v2
ERROR: infer_cuda_context.cpp:492 failed to set preprocess scalor and offsets.
0:00:22.697703707    82 0x55d3f19e8a90 ERROR          nvinferserver gstnvinferserver.cpp:362:gst_nvinfer_server_logger:<primary_gie> nvinferserver[UID 1]: Error in initialize() <infer_base_context.cpp:91> [UID = 1]: Infer Context failed to create preprocessors., nvinfer error:NVDSINFER_CONFIG_FAILED
0:00:22.697726439    82 0x55d3f19e8a90 WARN           nvinferserver gstnvinferserver_impl.cpp:439:start:<primary_gie> error: Failed to initialize InferTrtIsContext
0:00:22.697735151    82 0x55d3f19e8a90 WARN           nvinferserver gstnvinferserver_impl.cpp:439:start:<primary_gie> error: Config file path: /opt/nvidia/deepstream/deepstream-5.1/samples/configs/deepstream-app-trtis/config_infer_primary_detector_faster_rcnn.txt
I0325 16:28:24.435100 82 model_repository_manager.cc:837] unloading: faster_rcnn_inception_v2:1
I0325 16:28:24.435340 82 onnxruntime.cc:1848] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0325 16:28:24.435375 82 server.cc:280] Waiting for in-flight requests to complete.
I0325 16:28:24.435398 82 server.cc:295] Timeout 30: Found 1 live models and 0 in-flight non-inference requests
I0325 16:28:24.453803 82 onnxruntime.cc:1793] TRITONBACKEND_ModelFinalize: delete model state
I0325 16:28:24.466048 82 model_repository_manager.cc:966] successfully unloaded 'faster_rcnn_inception_v2' version 1
I0325 16:28:25.435491 82 server.cc:295] Timeout 29: Found 0 live models and 0 in-flight non-inference requests
0:00:23.698827973    82 0x55d3f19e8a90 WARN           nvinferserver gstnvinferserver.cpp:460:gst_nvinfer_server_start:<primary_gie> error: gstnvinferserver_impl start failed
** ERROR: <main:655>: Failed to set pipeline to PAUSED
Quitting
ERROR from primary_gie: Failed to initialize InferTrtIsContext
Debug info: gstnvinferserver_impl.cpp(439): start (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInferServer:primary_gie:
Config file path: /opt/nvidia/deepstream/deepstream-5.1/samples/configs/deepstream-app-trtis/config_infer_primary_detector_faster_rcnn.txt
ERROR from primary_gie: gstnvinferserver_impl start failed
Debug info: gstnvinferserver.cpp(460): gst_nvinfer_server_start (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInferServer:primary_gie
App run failed

remove “channel_offsets: [0, 0, 0]” and try again with TENSOR_ORDER_NHWC

I get this:

I0329 15:32:37.200861 103 model_repository_manager.cc:810] loading: faster_rcnn_inception_v2:1
I0329 15:32:37.233798 103 onnxruntime.cc:1712] TRITONBACKEND_Initialize: onnxruntime
I0329 15:32:37.233817 103 onnxruntime.cc:1725] Triton TRITONBACKEND API version: 1.0
I0329 15:32:37.233829 103 onnxruntime.cc:1731] 'onnxruntime' TRITONBACKEND API version: 1.0
I0329 15:32:37.242312 103 onnxruntime.cc:1773] TRITONBACKEND_ModelInitialize: faster_rcnn_inception_v2 (version 1)
WARNING: Since openmp is enabled in this build, this API cannot be used to configure intra op num threads. Please use the openmp environment variables to control the number of threads.
I0329 15:32:37.243353 103 onnxruntime.cc:1817] TRITONBACKEND_ModelInstanceInitialize: faster_rcnn_inception_v2_0 (GPU device 0)
I0329 15:32:39.166122 103 model_repository_manager.cc:983] successfully loaded 'faster_rcnn_inception_v2' version 1
INFO: infer_trtis_backend.cpp:206 TrtISBackend id:1 initialized model: faster_rcnn_inception_v2
deepstream-app: infer_cuda_utils.cpp:91: nvdsinferserver::CudaTensorBuf::CudaTensorBuf(const nvdsinferserver::InferDims&, nvdsinferserver::InferDataType, int, const string&, nvdsinferserver::InferMemType, int, bool): Assertion `bufBytes > 0' failed.
Aborted (core dumped)

Hi @yash.mandilwar ,
Sorry for the delay!

Could you take a look YOLOV4- DS-TRITON | Configuration specified max-batch 4 but TensorRT engine only supports max-batch 1 - #9 by bcao to see if this topic can help your issue?

Thanks!