Adding multi task classification as sgie after peoplenet as pgie

Accelerated Computing Intelligent Video Analytics DeepStream SDK
, , , , , , , ,
1

Please provide complete information as applicable to your setup.

• Hardware Platform : GPU
• DeepStream Version : 6.1.1
• TensorRT Version : 8.4.1.5
• NVIDIA GPU Driver Version (valid for GPU only) : 525.60.13
• Issue Type : questions, new requirements, bugs

I have a triton server running in a docker container which has two models

  1. Peoplenet
  2. Multitask classification

From deepstream app (in a different docker container) I am able to make grpc calls to peoplenet model which is my pgie. After this pgie I have a tracker and after the tracker I want to make grpc calls to Multitask classification model which will be my sgie.

pgie.link(tracker)
tracker.link(sgie1)

How should my config file for sgie look like?

2

this is how my config for peoplenet serving as my pgie looks like

infer_config {
  unique_id: 1
  gpu_ids: [0]
  max_batch_size: 16
  backend {
    inputs
    {
      dims: [16, 960, 544, 3]
    }
    triton {
      model_name: "peoplenet_tao"
      version: -1
      grpc {
        url: "172.17.0.2:8001"
        enable_cuda_buffer_sharing: true
      }
    }
  }

  preprocess {
    network_format: MEDIA_FORMAT_NONE
    tensor_order: TENSOR_ORDER_LINEAR
    tensor_name: "input_1"
    maintain_aspect_ratio: 0
    frame_scaling_hw: FRAME_SCALING_HW_DEFAULT
    frame_scaling_filter: 1
    normalize {
      scale_factor: 0.0039215697906911373
      channel_offsets: [0, 0, 0]
    }
  }
  postprocess {
    labelfile_path: "labels_peoplenet.txt"
    detection {
      num_detected_classes: 3
      per_class_params {
        key: 0
        value { pre_threshold: 0.4 }
      }
      nms {
        confidence_threshold:0.2
        topk:20
        iou_threshold:0.5
      }
    }
  }

  extra {
    copy_input_to_host_buffers: false
    output_buffer_pool_size: 2
  }
}
input_control {
  process_mode: PROCESS_MODE_FULL_FRAME
  operate_on_gie_id: -1
  interval: 0
}
4

please refer to /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app-triton/source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt
it is a nvinferserver pgie detection + sgie classification sample.

1 Like
5

Thanks for the response @fanzh , i decided to use vehicletypenet instead as sgie. Based on example shared. this is my config for the same

infer_config {
  unique_id: 2
  gpu_ids: [0]
  max_batch_size: 16
  backend {
    inputs
    {
      dims: [224, 224, 3]
    }
    triton {
      model_name: "vehicletypenet_tao"
      version: -1
      grpc {
        url: "172.17.0.2:8001"
        enable_cuda_buffer_sharing: true
      }
    }
  }

  preprocess {
    network_format: IMAGE_FORMAT_BGR
    tensor_order: TENSOR_ORDER_LINEAR
    maintain_aspect_ratio: 0
    frame_scaling_hw: FRAME_SCALING_HW_DEFAULT
    frame_scaling_filter: 1
    normalize {
      scale_factor: 1
      mean_file: "vehicletype_mean.ppm"
    }
  }
  postprocess {
    labelfile_path: "labels_vehicletype.txt"
    classification {
      threshold: 0.51
    }
  }
}
input_control {
  process_mode: PROCESS_MODE_CLIP_OBJECTS
  operate_on_gie_id: 1
  interval: 0
  async_mode: true
}

but I get the error

INFO: infer_grpc_backend.cpp:169 TritonGrpcBackend id:2 initialized for model: vehicletypenet_tao
0:00:00.781236567 179 0x39daaa0 ERROR nvinferserver gstnvinferserver.cpp:375:gst_nvinfer_server_logger: nvinferserver[UID 2]: Error in specifyBackendDims() <infer_grpc_context.cpp:140> [UID = 2]: input tensor: is not found in model:vehicletypenet_tao from config file
ERROR: infer_grpc_client.cpp:427 inference failed with error: inference request batch-size must be <= 1 for ‘vehicletypenet_tao’
ERROR: infer_trtis_backend.cpp:359 failed to specify dims after running inference failed on model:vehicletypenet_tao, nvinfer error:NVDSINFER_TRITON_ERROR
0:00:00.783526842 179 0x39daaa0 ERROR nvinferserver gstnvinferserver.cpp:375:gst_nvinfer_server_logger: nvinferserver[UID 2]: Error in specifyBackendDims() <infer_grpc_context.cpp:154> [UID = 2]: failed to specify input dims triton backend for model:vehicletypenet_tao, nvinfer error:NVDSINFER_TRITON_ERROR
0:00:00.783549241 179 0x39daaa0 ERROR nvinferserver gstnvinferserver.cpp:375:gst_nvinfer_server_logger: nvinferserver[UID 2]: Error in createNNBackend() <infer_grpc_context.cpp:210> [UID = 2]: failed to specify triton backend input dims for model:vehicletypenet_tao, nvinfer error:NVDSINFER_TRITON_ERROR
0:00:00.783644736 179 0x39daaa0 ERROR nvinferserver gstnvinferserver.cpp:375:gst_nvinfer_server_logger: nvinferserver[UID 2]: Error in initialize() <infer_base_context.cpp:79> [UID = 2]: create nn-backend failed, check config file settings, nvinfer error:NVDSINFER_TRITON_ERROR
0:00:00.783663235 179 0x39daaa0 WARN nvinferserver gstnvinferserver_impl.cpp:547:start: error: Failed to initialize InferTrtIsContext
0:00:00.783673135 179 0x39daaa0 WARN nvinferserver gstnvinferserver_impl.cpp:547:start: error: Config file path: configs/config_triton_grpc_infer_secondary_model1.txt
0:00:00.783712032 179 0x39daaa0 WARN nvinferserver gstnvinferserver.cpp:473:gst_nvinfer_server_start: error: gstnvinferserver_impl start failed
Error: gst-resource-error-quark: Failed to initialize InferTrtIsContext (1): gstnvinferserver_impl.cpp(547): start (): /GstPipeline:pipeline0/GstNvInferServer:secondary-inference-1:
Config file path: configs/config_triton_grpc_infer_secondary_model1.txt
Exiting app

6

could you share model’s config.pbtxt?
is vehicletypenet fixed 16 batch?

1 Like
7

@fanzh I have updated the config and errors above

I am not sure about vehicletypenet_tao’s config.pbtxt or batch size since I am using it starightaway from here tao-toolkit-triton-apps/scripts at main · NVIDIA-AI-IOT/tao-toolkit-triton-apps · GitHub

8

from the link tao-toolkit-triton-apps, this vehicletypenet_tao is 1 batch, please correct config.pbtxt and nvinferserver configuration file.

1 Like
9

@fanzh i updated sgie config as follows

infer_config {
  unique_id: 2
  gpu_ids: [0]
  max_batch_size: 16
  backend {
    inputs
    {
      dims: [1, 224, 224, 3]
    }
    triton {
      model_name: "vehicletypenet_tao"
      version: -1
      grpc {
        url: "172.17.0.2:8001"
        enable_cuda_buffer_sharing: true
      }
    }
  }

  preprocess {
    network_format: IMAGE_FORMAT_BGR
    tensor_order: TENSOR_ORDER_LINEAR
    maintain_aspect_ratio: 0
    frame_scaling_hw: FRAME_SCALING_HW_DEFAULT
    frame_scaling_filter: 1
    normalize {
      scale_factor: 1
      mean_file: "vehicletype_mean.ppm"
    }
  }
  postprocess {
    labelfile_path: "labels_vehicletype.txt"
    classification {
      threshold: 0.51
    }
  }
}
input_control {
  process_mode: PROCESS_MODE_CLIP_OBJECTS
  operate_on_gie_id: 1
  interval: 0
  async_mode: true
}

but now i get the following error in logs

INFO: infer_grpc_backend.cpp:169 TritonGrpcBackend id:2 initialized for model: vehicletypenet_tao
0:00:00.793125695 179 0x352e0a0 ERROR nvinferserver gstnvinferserver.cpp:375:gst_nvinfer_server_logger: nvinferserver[UID 2]: Error in specifyBackendDims() <infer_grpc_context.cpp:140> [UID = 2]: input tensor: is not found in model:vehicletypenet_tao from config file
ERROR: infer_grpc_client.cpp:427 inference failed with error: inference request batch-size must be <= 1 for ‘vehicletypenet_tao’
ERROR: infer_trtis_backend.cpp:359 failed to specify dims after running inference failed on model:vehicletypenet_tao, nvinfer error:NVDSINFER_TRITON_ERROR
0:00:00.795119919 179 0x352e0a0 ERROR nvinferserver gstnvinferserver.cpp:375:gst_nvinfer_server_logger: nvinferserver[UID 2]: Error in specifyBackendDims() <infer_grpc_context.cpp:154> [UID = 2]: failed to specify input dims triton backend for model:vehicletypenet_tao, nvinfer error:NVDSINFER_TRITON_ERROR
0:00:00.795141418 179 0x352e0a0 ERROR nvinferserver gstnvinferserver.cpp:375:gst_nvinfer_server_logger: nvinferserver[UID 2]: Error in createNNBackend() <infer_grpc_context.cpp:210> [UID = 2]: failed to specify triton backend input dims for model:vehicletypenet_tao, nvinfer error:NVDSINFER_TRITON_ERROR
0:00:00.795242414 179 0x352e0a0 ERROR nvinferserver gstnvinferserver.cpp:375:gst_nvinfer_server_logger: nvinferserver[UID 2]: Error in initialize() <infer_base_context.cpp:79> [UID = 2]: create nn-backend failed, check config file settings, nvinfer error:NVDSINFER_TRITON_ERROR
0:00:00.795260213 179 0x352e0a0 WARN nvinferserver gstnvinferserver_impl.cpp:547:start: error: Failed to initialize InferTrtIsContext
0:00:00.795269213 179 0x352e0a0 WARN nvinferserver gstnvinferserver_impl.cpp:547:start: error: Config file path: configs/config_triton_grpc_infer_secondary_model1.txt
0:00:00.795309311 179 0x352e0a0 WARN nvinferserver gstnvinferserver.cpp:473:gst_nvinfer_server_start: error: gstnvinferserver_impl start failed
Error: gst-resource-error-quark: Failed to initialize InferTrtIsContext (1): gstnvinferserver_impl.cpp(547): start (): /GstPipeline:pipeline0/GstNvInferServer:secondary-inference-1:
Config file path: configs/config_triton_grpc_infer_secondary_model1.txt
Exiting app

10

please update this line: max_batch_size: 16

1 Like
11

Thanks for the help @fanzh

I have two followup questions

The fix suggested works and makes sense but I also get the following error logs at the very beginning (only once and then it works fine)

WARNING: infer_proto_utils.cpp:271 update max_bath_size to 1 in config:configs/config_triton_grpc_infer_secondary_model1.txt
INFO: infer_grpc_backend.cpp:169 TritonGrpcBackend id:2 initialized for model: vehicletypenet_tao
0:00:00.763948703 179 0x7f39d40022f0 ERROR nvinferserver gstnvinferserver.cpp:375:gst_nvinfer_server_logger: nvinferserver[UID 2]: Error in specifyBackendDims() <infer_grpc_context.cpp:140> [UID = 2]: input tensor: is not found in model:vehicletypenet_tao from config file
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is ON
[NvMultiObjectTracker] Initialized
WARNING: infer_proto_utils.cpp:144 auto-update preprocess.network_format to IMAGE_FORMAT_RGB
INFO: infer_grpc_backend.cpp:169 TritonGrpcBackend id:1 initialized for model: peoplenet_tao
0:00:00.846156938 179 0x7f39d40022f0 ERROR nvinferserver gstnvinferserver.cpp:375:gst_nvinfer_server_logger: nvinferserver[UID 1]: Error in specifyBackendDims() <infer_grpc_context.cpp:140> [UID = 1]: input tensor: is not found in model:peoplenet_tao from config file
Decodebin child added: source

Can you help me understand why do these errors comeup and how do i get rid of them?

Also secondly if at a later point of time for a different model I would want a batch size>1 I will have to make changes on the triton server side and max_batch_size, dims will change in config on the client side - is my understanding correct?

12

it should be that input configuration is not the same with model’s config.pbtxt, you don’t need to configure inputs in nvinferserver’s cfg, please refer to sample opt\nvidia\deepstream\deepstream\samples\configs\deepstream-app-triton-grpc\config_infer_secondary_plan_engine_carmake.txt
nvinferserver is opensource in deepstream 6.2, you can add logs to debug if needed.

1 Like
13

@fanzh as per my understanding input { dims : } that is needed in my case as I am using enable_cuda_buffer_sharing: true . When I have enable_cuda_buffer_sharing: false then I dont need to specify input { dims : }

but the main question is that the app runs just fine and these errors just prop up only once in the beginning. I do not understand still why this must be the case.

15

@fanzh can you please help me in understanding this?

16

/** input tensors settings, optional */
repeated InputLayer inputs = 1;
as the comments shown in nvdsinferserver_config.proto, this setting is optional.
you also might get detailed description of enable_cuda_buffer_sharing in this file.

1 Like
17

@fanzh where can I find this nvdsinferserver_config.proto?

18

in deepstream sdk, \opt\nvidia\deepstream\deepstream-6.0\sources\includes\nvdsinferserver_config.proto

1 Like
19

i understand but @fanzh If i dont specify input dims then I get an error asking me to specify them and the pipeline stops there itself

20

could you share the error content? there are input information in model 's config.pbtxt file, config.pbtxt

21

Thanks for pointing this out. Think I will dig around a bit.

Making changes as you had suggested earlier resolved the main error anyway. Thanks!

22

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.