Triton deployment and inference

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Hello.

I’m trying to run inference on Triton Inference Server with faster rcnn model trained with TLT, converted to TensorRT.

In docs I can see:

Note that the models can also be deployed outside of DeepStream using TensorRT but users will need to do image pre-processing and post-process the output Tensor after inference.

What kind of pre-process and post-process should I use for FasterRCNN and other available object detection models?
Currently I have OpenCV’s Mat (8UC3)-> 32FC3 → channel-wise array (3, width, height) (RRRGGGBBB) → pack data into bytestring and send request.

Then I parse output, but NMS_1 always equals zero.

Triton generated this config:
{"name":"frcnn_fp16","versions":["1"],"platform":"tensorrt_plan","inputs":[{"name":"input_image","datatype":"FP32","shape":[-1,3,384,1248]}],"outputs":[{"name":"NMS","datatype":"FP32","shape":[-1,1,100,7]},{"name":"NMS_1","datatype":"FP32","shape":[-1,1,1,1]}]}

2

In TLT 3.0, for post-processing, please refer to https://github.com/NVIDIA-AI-IOT/deepstream_tlt_apps/tree/master/post_processor. For pre-processing, please refer to https://github.com/NVIDIA-AI-IOT/deepstream_tlt_apps/blob/master/configs/frcnn_tlt/pgie_frcnn_tlt_config.txt.

In TLT 2.0, for post-processing, please refer to https://github.com/NVIDIA-AI-IOT/deepstream_tlt_apps/tree/release/tlt2.0.1/nvdsinfer_customparser_frcnn_tlt

Reference topics:
For classification network,

For detectnet_v2 network,

3

Hi to all,
I was able to launch the INCEPTION SSD network plan file via tritonserver and write a python client to query it. However while I can get the correct output for NMS for the bboxes, NMS_1 (Keepcount) always gives me zero value. How is it possible ? The weird thing is that if I run the example script of tensorrt which doesn’t use the triton server I get the correct output of 100. thanks in advance

4

Hi rob91,

Please help to open a new topic if it’s still an issue. Thanks