Deepstream parallell inference failing to get probe data from nvdsmetamux

Accelerated Computing Intelligent Video Analytics DeepStream SDK
, ,
1

Please provide complete information as applicable to your setup.

**• Hardware Platform ** GPU ubuntu 20.04

• DeepStream Version 6.3

• TensorRT Version 8.6.1.6

• Issue Type( questions, new requirements, bugs) questions

Hi, I’m writing some code by C++ for testing deepstream parallel inferencing application.The main purpose is to test the nvdsmetamux plugin.

•aim: I want to implement multi-channel video inference through parallel pipelines.

•questions: unable to dispay video via nvmultistreamtiler, unable to get any data or print any logs in probe function of nvdsmetamux’s src pad(just as described in 273940 this means that the application is hung here). All these problems occurred without any program errors.

• I’ve refered 273940 from this forum.

• I’ve refered [deepstream_parallel_inference_app]GitHub - NVIDIA-AI-IOT/deepstream_parallel_inference_app: A project demonstrating how to use nvmetamux to run multiple models in parallel. repo.

here is the parallel pipeline(dot images) below:

pipe.zip (487.3 KB)

here is metamux.txt configuration file below:

[property]
enable=1
# sink pad name which data will be pass to src pad.
active-pad=sink_0
# default pts-tolerance is 60 ms.
pts-tolerance=60000

[user-configs]

[group-0]
src-ids-model-1=0;1;2;3
src-ids-model-2=0;1;2;3
# src-ids-model-<model unique ID>=<source ids>
# mux all source if don't set it.

the following pipeline runs completely normally.
pipe15.zip (288.8 KB)

here, single pipeline is running completely normally.the src-ids-model-x property of nvdsmetamux here also performs correctly.

Please help me analyze where the problem is?

Thanks,
Charles.

3

Is the demo stuck from the start, or is it stuck after running for a while?
Could you use our demo code without any change to reproduce your issue?

4

It’s stuck from the beginning. okay, I’ll try testing repo demo soon . Thanks.

6

Sorry for the late reply. I tried to test the official demo( deepstream_parallel_inference_app) , but encountered an error after run it. without any change.

GST_DEBUG=3 ./apps/deepstream-parallel-infer/deepstream-parallel-infer -c configs/apps/bodypose_yolo_lpr/source4_1080p_dec_parallel_infer.yml

src_ids:0;1;2
Unknown key enable-batch-process for tracker
Unknown key enable-past-frame for tracker
src_ids:1;2;3
Unknown key enable-batch-process for tracker
Unknown key enable-past-frame for tracker
src_ids:1;2;3
Unknown key enable-batch-process for tracker
Unknown key enable-past-frame for tracker
NVDSMETAMUX_CFG_PARSER: Group 'user-configs' ignored
** INFO: <create_primary_gie_bin:154>: gpu-id: 0 in primary-gie group is ignored, only accept in nvinferserver's config
i:0, src_id_num:3
link_streamdemux_to_streammux, srid:0, mux:0
link_streamdemux_to_streammux, srid:1, mux:0
link_streamdemux_to_streammux, srid:2, mux:0
** INFO: <create_primary_gie_bin:154>: gpu-id: 0 in primary-gie group is ignored, only accept in nvinferserver's config
i:1, src_id_num:3
link_streamdemux_to_streammux, srid:1, mux:1
link_streamdemux_to_streammux, srid:2, mux:1
link_streamdemux_to_streammux, srid:3, mux:1
** INFO: <create_primary_gie_bin:154>: gpu-id: 0 in primary-gie group is ignored, only accept in nvinferserver's config
i:2, src_id_num:3
link_streamdemux_to_streammux, srid:1, mux:2
link_streamdemux_to_streammux, srid:2, mux:2
link_streamdemux_to_streammux, srid:3, mux:2
0:00:00.253203358  6495 0x56095567b4d0 WARN           nvinferserver gstnvinferserver_impl.cpp:360:validatePluginConfig:<primary_gie> warning: Configuration file batch-size reset to: 4
ERROR: infer_trtis_server.cpp:904 Triton: failed to set model repo path: /media/aiui/CharlesShine/ShareFolder/obj_recognize/Project/deepstream_parallel_inference_app/tritonserver/models, triton_err_str:, err_msg:
ERROR: infer_trtis_server.cpp:840 failed to initialize trtserver on repo dir: root: "/media/aiui/CharlesShine/ShareFolder/obj_recognize/Project/deepstream_parallel_inference_app/tritonserver/models"
strict_model_config: true

0:00:00.255034948  6495 0x56095567b4d0 ERROR          nvinferserver gstnvinferserver.cpp:408:gst_nvinfer_server_logger:<primary_gie> nvinferserver[UID 1]: Error in createNNBackend() <infer_trtis_context.cpp:256> [UID = 1]: model:yolov4 get triton server instance failed. repo:root: "/media/aiui/CharlesShine/ShareFolder/obj_recognize/Project/deepstream_parallel_inference_app/tritonserver/models"
strict_model_config: true

0:00:00.255044544  6495 0x56095567b4d0 ERROR          nvinferserver gstnvinferserver.cpp:408:gst_nvinfer_server_logger:<primary_gie> nvinferserver[UID 1]: Error in initialize() <infer_base_context.cpp:79> [UID = 1]: create nn-backend failed, check config file settings, nvinfer error:NVDSINFER_TRITON_ERROR
0:00:00.255050793  6495 0x56095567b4d0 WARN           nvinferserver gstnvinferserver_impl.cpp:592:start:<primary_gie> error: Failed to initialize InferTrtIsContext
0:00:00.255056150  6495 0x56095567b4d0 WARN           nvinferserver gstnvinferserver_impl.cpp:592:start:<primary_gie> error: Config file path: /media/aiui/CharlesShine/ShareFolder/obj_recognize/Project/deepstream_parallel_inference_app/tritonclient/sample/configs/yolov4/config_yolov4_inferserver.txt
0:00:00.255238024  6495 0x56095567b4d0 WARN           nvinferserver gstnvinferserver.cpp:518:gst_nvinfer_server_start:<primary_gie> error: gstnvinferserver_impl start failed
0:00:00.255252755  6495 0x56095567b4d0 WARN                GST_PADS gstpad.c:1142:gst_pad_set_active:<primary_gie:sink> Failed to activate pad
Running...
ERROR from element primary_gie: Failed to initialize InferTrtIsContext
Error details: gstnvinferserver_impl.cpp(592): start (): /GstPipeline:deepstream-tensorrt-openpose-pipeline/GstBin:parallel_infer_bin/GstBin:primary_gie_0_bin/GstNvInferServer:primary_gie:
Config file path: /media/aiui/CharlesShine/ShareFolder/obj_recognize/Project/deepstream_parallel_inference_app/tritonclient/sample/configs/yolov4/config_yolov4_inferserver.txt
Quitting
Returned, stopping playback
Deleting pipeline
App run successful

The build engine log with no errors . Here is its content:
build_engine.log (128.9 KB)

According to the error message, does it seem to be a model problem or model configuration problem?

Thanks,
Charles.

7

It could be your triton environment issue. Are you using our docker based on triton, like nvcr.io/nvidia/deepstream-l4t:7.0-triton-multiarch? Could you try to update your DeepStream to our latest version or there may also be compatibility issues.