Serving Peoplenet model using Triton gRPC Inference Server and make calls to it from outside the container

[pulkit]

Please provide complete information as applicable to your setup.
• Hardware Platform : GPU
• DeepStream Version : 6.1.1
• TensorRT Version : 8.4.1.5
• NVIDIA GPU Driver Version (valid for GPU only) : 525.60.13
• Issue Type : questions, new requirements, bugs

Hi, I am able to run deepstream_python_apps/apps/deepstream-test3 at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub as a standalone app with python3 deepstream_test_3.py -i file:///home/ubuntu/video1.mp4 file:///home/ubuntu/video2.mp4

Now I want to serve Peoplenet model using Triton gRPC Inference Server and make calls to it from outside the container.

Can someone please suggest as to how do I go about getting this done?

[fanzh]

deepstream_test_3.py has supported triton gRPC mode, please refer to the README file, especially here deepstream_python_apps/README at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub

[pulkit]

Thanks for the response @fanzh . I did see that but from the instructions in that I was unable to setup the people net model with triton following the instructions in that readme.

So instead of that I followed this one GitHub - NVIDIA-AI-IOT/tao-toolkit-triton-apps: Sample app code for deploying TAO Toolkit trained models to Triton and have

I0202 06:56:34.931328 135 server.cc:626]
±-----------------------------±--------±-------+
| Model | Version | Status |
±-----------------------------±--------±-------+
| dashcamnet_tao | 1 | READY |
| lprnet_tao | 1 | READY |
| multitask_classification_tao | 1 | READY |
| peoplenet_tao | 1 | READY |
| peoplesegnet_tao | 1 | READY |
| pose_classification_tao | 1 | READY |
| re_identification_tao | 1 | READY |
| retinanet_tao | 1 | READY |
| vehicletypenet_tao | 1 | READY |
| yolov3_tao | 1 | READY |
±-----------------------------±--------±-------+

I0202 06:56:34.989151 135 metrics.cc:650] Collecting metrics for GPU 0: Tesla V100-PCIE-16GB
I0202 06:56:34.989922 135 tritonserver.cc:2159]
±---------------------------------±-------------------------------------------------------------------------------+
| Option | Value |
±---------------------------------±-------------------------------------------------------------------------------+
| server_id | triton |
| server_version | 2.23.0 |
| server_extensions | classification sequence model_repository model_repository(unload_dependents) s |
| | chedule_policy model_configuration system_shared_memory cuda_shared_memory bin |
| | ary_tensor_data statistics trace |
| model_repository_path[0] | /model_repository |
| model_control_mode | MODE_NONE |
| strict_model_config | 1 |
| rate_limit | OFF |
| pinned_memory_pool_byte_size | 268435456 |
| cuda_memory_pool_byte_size{0} | 67108864 |
| response_cache_byte_size | 0 |
| min_supported_compute_capability | 6.0 |
| strict_readiness | 1 |
| exit_timeout | 30 |
±---------------------------------±-------------------------------------------------------------------------------+

I0202 06:56:34.991804 135 grpc_server.cc:4587] Started GRPCInferenceService at 0.0.0.0:8001
I0202 06:56:34.992076 135 http_server.cc:3303] Started HTTPService at 0.0.0.0:8000
I0202 06:56:35.034268 135 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002

Can you point out what more do i need to do?

[fanzh]

from your logs, tritonserver started successfully, and peoplenet_tao model was loaded, so server side is ready. you need to set model_name: “peoplenet_tao” in config_triton_grpc_infer_primary_peoplenet.txt because loaded model name is not peoplenet, then start the command according to the readme.

[pulkit]

ok @fanzh so i did ran
python3 deepstream_test_3.py -i file:///home/testA.mp4 --pgie nvinferserver-grpc -c config_triton_grpc_infer_primary_peoplenet.txtafter making the change model_name: “peoplenet_tao”

but i get the error

{‘input’: [‘file:///home/testA.mp4’], ‘configfile’: ‘config_triton_grpc_infer_primary_peoplenet.txt’, ‘pgie’: ‘nvinferserver-grpc’, ‘no_display’: False, ‘file_loop’: False, ‘disable_probe’: False, ‘silent’: False}
Creating Pipeline

Creating streamux

Creating source_bin 0

Creating source bin
source-bin-00
Creating Pgie

Unable to create pgie : nvinferserver-grpc
Creating tiler

Creating nvvidconv

Creating nvosd

Creating EGLSink

Traceback (most recent call last):
File “deepstream_test_3.py”, line 484, in
sys.exit(main(stream_paths, pgie, config, disable_probe))
File “deepstream_test_3.py”, line 317, in main
pgie.set_property(‘config-file-path’, config)
AttributeError: ‘NoneType’ object has no attribute ‘set_property’

[fanzh]

@pulkit

Unable to create pgie : nvinferserver-grpc

as the logs shown, the application created nvinfersever element faild, are you using deepstream triton docker? dose “gst-inspect-1.0 nvinferserver” output “No such element or plugin” ?

[pulkit]

yes apologies

gst-inspect-1.0 nvinferserver
No such element or plugin ‘nvinferserver’

I was under the impression that it is present in deepstream sdk itself. what can i do to fix it? Is it manadatory to use deepstream triton docker

[fanzh]

are you using deepstream devel docker?
deepstream will leverage triton libs to communicate with triton server, we suggest to use deepstream triton docker because you need not to compile triton libs. here is the link: nvcr.io/nvidia/deepstream:6.1.1-triton, please refer to DeepStream | NVIDIA NGC

[pulkit]

I am using docker only to start triton server @fanzh as descibed here GitHub - NVIDIA-AI-IOT/tao-toolkit-triton-apps: Sample app code for deploying TAO Toolkit trained models to Triton which as you have pointed out from the logs seems to work

Earlier I had installed deepstream-sdk as described here Quickstart Guide — DeepStream 6.1.1 Release documentation and was able to run deepstream_python_apps/apps/deepstream-test3 at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub as a standalone app with python3 deepstream_test_3.py -i file:///home/ubuntu/video1.mp4 file:///home/ubuntu/video2.mp4
This was not running inside any container.

So the command python3 deepstream_test_3.py -i file:///home/testA.mp4 --pgie nvinferserver-grpc -c config_triton_grpc_infer_primary_peoplenet.txt after making the change model_name: “peoplenet_tao” should also not need any container right?

[fanzh]

@pulkit

python3 deepstream_test_3.py -i file:///home/ubuntu/video1.mp4 file:///home/ubuntu/video2.mp4
This was not running inside any container.

yes, this way will use nvinfer to do inference, it will not use triton, it can run in or outside docker.

@pulkit

python3 deepstream_test_3.py -i file:///home/testA.mp4 --pgie nvinferserver-grpc -c config_triton_grpc_infer_primary_peoplenet.txt after making the change model_name: “peoplenet_tao”

this method will use nvinferserver to do inference, it will use triton, both client and server need to run in deeptream triton docker because both will use triton libs.

[fanzh]

please refer to this for Triton gRPC setup deepstream_python_apps/README at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub

[pulkit]

ok that makes sense. I will follow that but one quick question @fanzh

I can have two separate docker containers right?
One container for server that is already running and another one for client where I will be running python3 deepstream_test_3.py -i file:///home/testA.mp4 --pgie nvinferserver-grpc -c config_triton_grpc_infer_primary_peoplenet.txt after making the change model_name: “peoplenet_tao”

It should be possible for the two to communicate with each other if I map the ports correctly between host and the 2 containers

[fanzh]

yes, server and client can run in different dockers, you need to use the actual IP, deepstream_python_apps/config_triton_grpc_infer_primary_peoplenet.txt at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub

here is another triton GRPC sample, GitHub - NVIDIA-AI-IOT/deepstream_lpr_app: Sample app code for LPR deployment on DeepStream

[pulkit]

@fanzh thanks this worked. I just had to ensure both stay in same network and had to fix the path to labels file