Need tao_triton video inference

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Please provide the following information when requesting support.

• Hardware (T4/V100/Xavier/Nano/etc) T4
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc) Yolo_v4
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here)
• Training spec file(If have, please share here)
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)

im using above link , i can get the link of video inference , since the above link file tao_client.py

has only Image Inference, as i need Video inference as well .

2

Unfortunately, the triton app does not support video inference.

3

could you provide python3 Parser for yolov4 and TAO models ,
im referring deepstream_python_apps/apps/deepstream-ssd-parser at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub

could you provide relavent to add on top it of instead of SSD parser , i need to replace with python3 yolov4 parser

4

If you are going to run inference against the yolov4 model in deepstream, please refer to official github. GitHub - NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream

If you are going to continue to run inference with trition-app, you can leverage tao-toolkit-triton-apps/yolov3_postprocessor.py at main · NVIDIA-AI-IOT/tao-toolkit-triton-apps · GitHub

5

im using below config , im not able to get the meta-data and im using deepstream


infer_config {
  unique_id: 1
  gpu_ids: [0]
  max_batch_size: 4
  

  backend {
    inputs: [ {
      name: "Input"
    }]
    outputs: [
       {name: "BatchedNMS"},
      {name: "BatchedNMS_1"},
      {name: "BatchedNMS_2"},
      {name: "BatchedNMS_3"} ]
    triton {
      model_name: "Helmet_model"
      version: 1
      grpc {
        url: "localhost:8001"
        #url: "3.141.193.122:8001"
      }
    }
  }

  preprocess {
    network_format: IMAGE_FORMAT_RGB
    tensor_order: TENSOR_ORDER_LINEAR
    maintain_aspect_ratio: 0
    frame_scaling_hw: FRAME_SCALING_HW_DEFAULT
    frame_scaling_filter: 1
    normalize {
      scale_factor: 0.0078125
      channel_offsets: [128, 128, 128]
    }
  }

  

  postprocess {
    labelfile_path: "labels.txt"
    detection {
      num_detected_classes: 2
      custom_parse_bbox_func: "NvDsInferParseCustomYoloV4"
      nms {
        confidence_threshold: 0.3
        iou_threshold: 0.6
        topk : 100
      }

    }
  } 


  

  extra {
    copy_input_to_host_buffers: false
    output_buffer_pool_size: 2
  }

  custom_lib {
  path:"/deepstream_yolov4/nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so"
  
  }
}
input_control {
  process_mode: PROCESS_MODE_FULL_FRAME
  interval: 0
}
output_control {
  output_tensor_meta: true
}

and i get below error

python3: nvdsparsebbox_Yolo.cpp:139: bool NvDsInferParseCustomYoloV4(const std::vector&, const NvDsInferNetworkInfo&, const NvDsInferParseDetectionParams&, std::vector&): Assertion `boxes.inferDims.numDims == 3’ failed.

6

Are you running with official github. GitHub - NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream ?

7

we have python deepstream app , and im using above config and parser generated from

GitHub - NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream

8

Please directly use GitHub - NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream to run inference.

9

we are able to run the TAO apps with normal deepstream app , since we have loaded MODELS IN TRITON SERVER , we need to use the above config and run . But im getting error for custom model.

10

For running TAO model in triton server, refer to GitHub - NVIDIA-AI-IOT/tao-toolkit-triton-apps: Sample app code for deploying TAO Toolkit trained models to Triton

Firstly, you can run the default github to get familiar with the process.

Then, you can replace your etlt model with default one. And then comment out

line72,73,74 of https://github.com/NVIDIA-AI-IOT/tao-toolkit-triton-apps/blob/main/scripts/start_server.sh to let it not download the default lpr model next time when you trigger server.

11

i have tried it . its working with our MODEL . But it doesnt have Video Inferece . Only Image inference exist

12

i need the way take out this parser and push it to deepstream -app

13

i tried with the parser of

GitHub - NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream

but i got below errror now

ERROR: infer_postprocess.cpp:344 Detect-postprocessor failed to init resource because dlsym failed to get func NvDsInferParseCustomBatchedNMSTLT pointer
0:00:00.390680778 20514 0x448e2d0 ERROR nvinferserver gstnvinferserver.cpp:361:gst_nvinfer_server_logger: nvinferserver[UID 1]: Error in allocateResource() <infer_cuda_context.cpp:519> [UID = 1]: failed to allocate resource for postprocessor., nvinfer error:NVDSINFER_CUSTOM_LIB_FAILED
0:00:00.390708228 20514 0x448e2d0 ERROR nvinferserver gstnvinferserver.cpp:361:gst_nvinfer_server_logger: nvinferserver[UID 1]: Error in initialize() <infer_base_context.cpp:109> [UID = 1]: Failed to allocate buffers
0:00:00.390745052 20514 0x448e2d0 WARN nvinferserver gstnvinferserver_impl.cpp:510:start: error: Failed to initialize InferTrtIsContext
0:00:00.390759029 20514 0x448e2d0 WARN nvinferserver gstnvinferserver_impl.cpp:510:start: error: Config file path: config/TripleRiding/config_infer.txt_SoFile
0:00:00.392358949 20514 0x448e2d0 WARN nvinferserver gstnvinferserver.cpp:459:gst_nvinfer_server_start: error: gstnvinferserver_impl start failed
Warning: gst-library-error-quark: Configuration file batch-size reset to: 1 (5): gstnvinferserver_impl.cpp(287): validatePluginConfig (): /GstPipeline:pipeline0/GstNvInferServer:primary

below is the customlib i have used

custom_parse_bbox_func:“NvDsInferParseCustomNMSTLT”

custom_lib {

path:“deepstream_tao_apps/post_processor/libnvds_infercustomparser_tao.so”

}

and i see same error in the lnk

14

Is above error from running deepstream_python_apps?
Actually for tao model, as mentioned in the TAO user guide, the official app is GitHub - NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream . Please use it instead.

15

i have loaded the Model in Triton Server , but the above AI-IOT app is in deepstream . How could i use TAO app where the Model is loaded through triton Server

17

Actually the deeptream tao apps is one of the inference way.
The triton-app is another inference way.

The deeptream tao apps can generate tensorrt engine. You can also deploy .etlt model in it.

18

i have deployed as trt engine model itself , but i need to TAO parser just like SSD parser since im
using below deepstream app as an reference

https://github.com/NVIDIA-AI-IOT/deepstream_python_apps/blob/master/apps/deepstream-ssd-parser/deepstream_ssd_parser.py

i need to modify

https://github.com/NVIDIA-AI-IOT/deepstream_python_apps/blob/master/apps/deepstream-ssd-parser/ssd_parser.py
to parse trt engine model , if u can provide supporting parser as above one that would be good

19

Again, for TAO, it does not maintain/support deepstream_python_apps officially. Officially it supports GitHub - NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream .

20

We need Video Inference as well , GitHub - NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream
above link is only for Image Inference . From the above link im able to use my custom model and do the inference .

But we need for the VIDEO Inference as well .>!!

21

The GitHub - NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream can run inference against video file.