Source ID input to the Triton Inference Server from the nvinferserver plugin

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

→ Hardware Platform (Jetson / GPU)** : Deepstream 6.3 docker container on a NVIDIA GeForce MX230
→ DeepStream Version : 6.3
→ JetPack Version (valid for Jetson only) : Nil
→ TensorRT Version** : 8.5.3.1
→ NVIDIA GPU Driver Version (valid for GPU only) : 535.183.01
→ Issue Type( questions, new requirements, bugs) : Problem

The problem is to ensure that every source frame feed has a source ID connected to it throughout the different processing steps in the pipeline. I just want to know what are the steps involved in passing the source ID from the pipeline (nvinferserver plugin) to the Triton Inference Server.

What is known : The frame meta data in the NvBufSurfaces already has source ID.

How can I modify the Triton Inference Server custom implementation code present in /opt/nvidia/deepstream/deepstream-6.3/sources/TritonBackendEnsemble/nvdsinferserver_custom_impl_ensemble/nvdsinferserver_custom_process_ensemble.cpp to acheive the same.

I am pretty new to this toolkit so pardon me if the doubt is too trivial …

3

Thanks for the sharing! why do you need to send source ID to Triton Server? what do you men about “NvBufSurfaces already has source ID”? NvBufSurface has no source ID. what do you want to impletment in nvdsinferserver_custom_process_ensemble.cpp?

4

Thank you for the reply. I might have misinterpreted that the frame buffers in between the plugins have in built source ID property in its metadata.
Getting back to why I need the solution to this problem is to ensure that whenever a Source in a multi-source pipeline malfunctions, we should be able to tell which source is faulty and thus divert our focus on rectifying that particular source. So, the trivial solution is to assign source IDs to the same.
About the cpp file, I believe its the custom implementation for the triton inference server, so is there anyway I can input or accommodate this source ID to the triton inference server, or is it beneficial to even do this for solving my problem?

5

please refer to deepstream-app which is opensource. deepstream-app uses rtspsrc_monitor_probe_func to monitor src’s data receiving, and reconnect rtsp source in watch_source_status when not receiving data in a specific time.

nvinferserver plugin uses Triton client API to let triton do inference. nvinferserver and Triton code are opensource. in TrtServerRequest::setInputs of \opt\nvidia\deepstream\deepstream\sources\libs\nvdsinferserver\infer_trtis_server.cpp， you can see how the API are used. source ID can’t be passed.

That cpp is used to add extraInput, which is used as model input data. source ID is not model input data.

6

So, from your reply I can understand that the Source ID cannot be passed as extraInput to the Triton Inference Server and the only mechanism to monitor the sources is by using rtspsrc_monitor_probe_func.

To not leave any stone unturned, is there any mechanism to maintain source ID for a particular source throughout the pipeline?. Can nvstreammux plugin help in assigning Source ID to each frame’s metadata?

Also is it possible to add the the value of source ID as input to the python backend code (model.py) of the Triton Inference Server?

7

Please refer to NvDsFrameMeta in \opt\nvidia\deepstream\deepstream\sources\includes\nvdsmeta.h. source_id is saved in NvDsFrameMeta.
please refer to my last comment. from the Triton client API, source ID can’t be passed. you can access source ID in nvinferserver plugin.

8

Okay noted.

“The nvinferserver plugin also supports the interface for custom functions for parsing outputs of object detectors, classifiers, and initialisation of non-image input layers in cases where there is more than one input layer. Refer to sources/includes/nvdsinfer_custom_impl.h for the custom method implementations for custom models”

I got this info from this support link : https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinferserver.html

Can this be of any relevance to the above mentioned Source ID assigning problem or what I am trying to achieve?

9

AYK, source id is not the input layer of model. the method above is only for add of non-image input layers.

10

Thank you for your response.
So, just to clarify, you are stating that there is no way we can pass an integer source id that is essential for model param, which specify the colour or features of the video.

We are passing source id as input to the model, so it should be in a layer.

11

if your model have multiple layers and one layer represents source id, please refer to nvdsinferserver_custom_process_ensemble.cpp for how to use extraInputProcess to add extra inputs.

12

I have tried creating a shared object file with the cpp file mentioned above and included it using the config file as specified below :

infer_config {
unique_id: 1
gpu_ids: 0
max_batch_size: 1
backend {
inputs [
{
name: “INPUT0”
dims: [3, 1080, 1920]
},
{
name: “SOURCE_ID”
dims: [1]
}
]
trt_is {
model_name: “centerface”
version: -1
model_repo {
root: “./centerface”
log_level: 1
tf_gpu_memory_fraction: 0.2
tf_disable_soft_placement: 0
}
}
}

preprocess {
network_format: IMAGE_FORMAT_RGB
tensor_order: TENSOR_ORDER_LINEAR
maintain_aspect_ratio: 0
normalize {
scale_factor: 1.0
channel_offsets: [0, 0, 0]
}
}

postprocess {
labelfile_path: “./centerface/centerface/centerface_labels.txt”
other{}
}

custom_lib {
path: “./centerface/libnvdstriton_custom_impl_ensemble.so”
}

extra {
copy_input_to_host_buffers: false
}
}
input_control {
process_mode: PROCESS_MODE_FULL_FRAME
interval: 0
}

output_control {
output_tensor_meta: true
}

As given above, I have specified a tensor Source ID to be given as input.

The query for running the pipeline has been specified as below:

gst-launch-1.0 --gst-debug=rtpjitterbuffer:6,nvstreammux:6,nvstreamdemux:6 videotestsrc ! nvvideoconvert ! “video/x-raw(memory:NVMM), width=640, height=480” ! m.sink_0 nvstreammux name=m width=640 height=480 batch_size=1 ! nvinferserver config-file-path=config.txt ! nvstreamdemux name=d d.src_0 ! nvvideoconvert ! autovideosink sync=0

and these are the error logs I get :

0:00:00.054489923 55981 0x5aa6f4f16210 DEBUG nvstreammux gstnvstreammux.cpp:1539:gst_nvstreammux_request_new_pad: Requesting new sink pad
0:00:00.054765830 55981 0x5aa6f4f16210 DEBUG nvstreamdemux gstnvstreamdemux.cpp:129:gst_nvstreamdemux_request_new_pad: Requesting new src pad
0:00:00.054779426 55981 0x5aa6f4f16210 DEBUG nvstreamdemux gstnvstreamdemux.cpp:153:gst_nvstreamdemux_request_new_pad: Requesting new src pad
Setting pipeline to PAUSED …
WARNING: infer_proto_utils.cpp:201 backend.trt_is is deprecated. updated it to backend.triton
W0902 04:16:25.504173 55981 metrics.cc:512] Unable to get power limit for GPU 0. Status:Success, value:0.000000
W0902 04:16:25.504253 55981 metrics.cc:530] Unable to get power usage for GPU 0. Status:Success, value:0.000000
W0902 04:16:25.504269 55981 metrics.cc:554] Unable to get energy consumption for GPU 0. Status:Success, value:0
Model initialized
Error connecting: [Errno 111] Connection refused
INFO: infer_trtis_backend.cpp:218 TrtISBackend id:1 initialized model: centerface
Pipeline is PREROLLING …
0:00:02.191259811 55981 0x5aa6f4f08240 DEBUG nvstreammux gstnvstreammux.cpp:1256:gst_nvstreammux_sink_event: parse video info from caps video/x-raw(memory:NVMM), width=(int)640, height=(int)480, framerate=(fraction)30/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)NV12, block-linear=(boolean)false, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0
0:00:02.191289412 55981 0x5aa6f4f08240 DEBUG nvstreammux gstnvstreammux.cpp:1295:gst_nvstreammux_sink_event: peer caps video/x-raw(memory:NVMM), format=(string){ NV12, RGBA }, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]
0:00:02.191611653 55981 0x5aa6f4f08240 DEBUG nvstreamdemux gstnvstreamdemux.cpp:443:set_src_pad_caps: caps before = video/x-raw(memory:NVMM), width=(int)640, height=(int)480, framerate=(fraction)30/1, multiview-mode=(string)mono, format=(string)NV12, block-linear=(boolean)false, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0, batch-size=(int)1, num-surfaces-per-frame=(int)1; video/x-raw, width=(int)640, height=(int)480, framerate=(fraction)30/1, multiview-mode=(string)mono, format=(string)NV12, block-linear=(boolean)false, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0, batch-size=(int)1, num-surfaces-per-frame=(int)1

0:00:02.191633984 55981 0x5aa6f4f08240 DEBUG nvstreamdemux gstnvstreamdemux.cpp:453:set_src_pad_caps: caps after = video/x-raw(memory:NVMM), width=(int)640, height=(int)480, framerate=(fraction)30/1, multiview-mode=(string)mono, format=(string)NV12, block-linear=(boolean)false, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0, batch-size=(int)1, num-surfaces-per-frame=(int)1; video/x-raw, width=(int)640, height=(int)480, framerate=(fraction)30/1, multiview-mode=(string)mono, format=(string)NV12, block-linear=(boolean)false, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0, batch-size=(int)1, num-surfaces-per-frame=(int)1

0:00:02.195234493 55981 0x5aa6f4f08240 INFO nvstreammux gstnvstreammux.cpp:1467:gst_nvstreammux_sink_event: mux got segment from src 0 time segment start=0:00:00.000000000, offset=0:00:00.000000000, stop=99:99:99.999999999, rate=1.000000, applied_rate=1.000000, flags=0x00, time=0:00:00.000000000, base=0:00:00.000000000, position 0:00:00.000000000, duration 99:99:99.999999999
W0902 04:16:26.504844 55981 metrics.cc:512] Unable to get power limit for GPU 0. Status:Success, value:0.000000
W0902 04:16:26.504897 55981 metrics.cc:530] Unable to get power usage for GPU 0. Status:Success, value:0.000000
W0902 04:16:26.504910 55981 metrics.cc:554] Unable to get energy consumption for GPU 0. Status:Success, value:0
0:00:02.764540418 55981 0x5aa6f4f08240 DEBUG nvstreammux gstnvstreammux.cpp:521:gst_nvstreammux_chain: Got buffer 0x5aa6f54ccea0 from source 0 pts = 0:00:00.000000000
0:00:02.764706936 55981 0x5aa6f4f08580 DEBUG nvstreammux gstnvstreammux.cpp:2432:gst_nvstreammux_src_collect_buffers: Pad added event sent 0

0:00:02.764895466 55981 0x5aa6f4f08580 DEBUG nvstreammux gstnvstreammux.cpp:2933:gst_nvstreammux_src_push_loop: Pushing buffer 0x5aa6f54cc6c0, batch size 1, PTS 0:00:00.000000000
0:00:02.764909791 55981 0x5aa6f4f08580 DEBUG nvstreammux gstnvstreammux.cpp:2954:gst_nvstreammux_src_push_loop: STREAMMUX OUT BUFFER attached timestamp 0:00:00.000000000
0:00:02.765532390 55981 0x5aa6f4f08240 DEBUG nvstreammux gstnvstreammux.cpp:521:gst_nvstreammux_chain: Got buffer 0x718a8c014000 from source 0 pts = 0:00:00.033333333
0:00:02.765601157 55981 0x5aa6f4f08580 DEBUG nvstreammux gstnvstreammux.cpp:2933:gst_nvstreammux_src_push_loop: Pushing buffer 0x5aa6f54cc7e0, batch size 1, PTS 0:00:00.033333333
0:00:02.765621790 55981 0x5aa6f4f08580 DEBUG nvstreammux gstnvstreammux.cpp:2954:gst_nvstreammux_src_push_loop: STREAMMUX OUT BUFFER attached timestamp 0:00:00.033333333
0:00:02.767095384 55981 0x718a800022a0 WARN nvinferserver gstnvinferserver.cpp:412:gst_nvinfer_server_logger: nvinferserver[UID 1]: Warning from initFixedExtraInputLayers() <infer_cuda_context.cpp:361> [UID = 1]: More than one input layers but custom initialization function not implemented
ERROR: infer_cuda_context.cpp:327 Init fixed extra input tensors failed., nvinfer error:NVDSINFER_CUSTOM_LIB_FAILED
ERROR: infer_base_context.cpp:289 pre-inference on input tensors failed., nvinfer error:NVDSINFER_CUSTOM_LIB_FAILED
0:00:02.767257133 55981 0x718a800022a0 WARN nvinferserver gstnvinferserver.cpp:412:gst_nvinfer_server_logger: nvinferserver[UID 1]: Warning from initFixedExtraInputLayers() <infer_cuda_context.cpp:361> [UID = 1]: More than one input layers but custom initialization function not implemented
ERROR: infer_cuda_context.cpp:327 Init fixed extra input tensors failed., nvinfer error:NVDSINFER_CUSTOM_LIB_FAILED
ERROR: infer_base_context.cpp:289 pre-inference on input tensors failed., nvinfer error:NVDSINFER_CUSTOM_LIB_FAILED
Pipeline is PREROLLED …
Setting pipeline to PLAYING …
0:00:02.769432404 55981 0x5aa6f4f16210 DEBUG nvstreammux gstnvstreammux.cpp:1191:gst_nvstreammux_src_event: latency 0:00:00.000000000
0:00:02.769627474 55981 0x5aa6f4f08240 DEBUG nvstreammux gstnvstreammux.cpp:521:gst_nvstreammux_chain: Got buffer 0x718a8c014120 from source 0 pts = 0:00:00.066666666
0:00:02.769686536 55981 0x5aa6f4f08580 DEBUG nvstreammux gstnvstreammux.cpp:2933:gst_nvstreammux_src_push_loop: Pushing buffer 0x5aa6f54cc900, batch size 1, PTS 0:00:00.066666666
0:00:02.769713112 55981 0x5aa6f4f08580 DEBUG nvstreammux gstnvstreammux.cpp:2954:gst_nvstreammux_src_push_loop: STREAMMUX OUT BUFFER attached timestamp 0:00:00.066666666
New clock: GstSystemClock
0:00:02.770178096 55981 0x718ab00c7f50 WARN nvinferserver gstnvinferserver.cpp:581:gst_nvinfer_server_push_buffer: error: inference failed with unique-id:1
ERROR: from element /GstPipeline:pipeline0/GstNvInferServer:nvinferserver0: inference failed with unique-id:1
Additional debug info:
gstnvinferserver.cpp(581): gst_nvinfer_server_push_buffer (): /GstPipeline:pipeline0/GstNvInferServer:nvinferserver0
Execution ended after 0:00:00.000551316
Setting pipeline to NULL …
0:00:02.772811732 55981 0x718a800022a0 WARN nvinferserver gstnvinferserver.cpp:412:gst_nvinfer_server_logger: nvinferserver[UID 1]: Warning from initFixedExtraInputLayers() <infer_cuda_context.cpp:361> [UID = 1]: More than one input layers but custom initialization function not implemented
ERROR: infer_cuda_context.cpp:327 Init fixed extra input tensors failed., nvinfer error:NVDSINFER_CUSTOM_LIB_FAILED
0:00:02.772834926 55981 0x5aa6f4f08240 DEBUG nvstreammux gstnvstreammux.cpp:521:gst_nvstreammux_chain: Got buffer 0x718a8c014240 from source 0 pts = 0:00:00.100000000
ERROR: infer_base_context.cpp:289 pre-inference on input tensors failed., nvinfer error:NVDSINFER_CUSTOM_LIB_FAILED
0:00:02.772940873 55981 0x5aa6f4f08580 DEBUG nvstreammux gstnvstreammux.cpp:2933:gst_nvstreammux_src_push_loop: Pushing buffer 0x5aa6f54cca20, batch size 1, PTS 0:00:00.099999999
0:00:02.772965876 55981 0x5aa6f4f08580 DEBUG nvstreammux gstnvstreammux.cpp:2954:gst_nvstreammux_src_push_loop: STREAMMUX OUT BUFFER attached timestamp 0:00:00.099999999
0:00:02.775537661 55981 0x5aa6f4f08240 DEBUG nvstreammux gstnvstreammux.cpp:521:gst_nvstreammux_chain: Got buffer 0x5aa6f54ccea0 from source 0 pts = 0:00:00.133333333
0:00:02.775606317 55981 0x5aa6f4f08580 DEBUG nvstreammux gstnvstreammux.cpp:2933:gst_nvstreammux_src_push_loop: Pushing buffer 0x5aa6f54cc6c0, batch size 1, PTS 0:00:00.133333332
0:00:02.775631240 55981 0x5aa6f4f08580 DEBUG nvstreammux gstnvstreammux.cpp:2954:gst_nvstreammux_src_push_loop: STREAMMUX OUT BUFFER attached timestamp 0:00:00.133333332
Error connecting: [Errno 111] Connection refused
W0902 04:16:27.505894 55981 metrics.cc:512] Unable to get power limit for GPU 0. Status:Success, value:0.000000
W0902 04:16:27.505968 55981 metrics.cc:530] Unable to get power usage for GPU 0. Status:Success, value:0.000000
W0902 04:16:27.506001 55981 metrics.cc:554] Unable to get energy consumption for GPU 0. Status:Success, value:0
Cleaning up…
Error connecting: [Errno 111] Connection refused

Am I going on the right path on achieving the solution, could you please specify how these error logs can be useful in this case?

13
  1. as the log “backend.trt_is is deprecated. updated it to backend.triton” shown, please remove the trt_is configurations.
  2. nvinferserver plugin and low-level lib are opensource. the fatal error is “ERROR: infer_cuda_context.cpp:327 Init fixed extra input tensors failed”. it is because “More than one input layers but custom initialization function not implemented”. plese refer to \opt\nvidia\deepstream\deepstream-6.3\sources\TritonOnnxYolo\config_triton_inferserver_primary_yolov3_onnx_custom.txt. the model has multiple input layers. please provide a function in extra to add extra input layer data.
14

This is the updated config.txt file contents (changes made to the ‘extra’ section) :

infer_config {
unique_id: 1
gpu_ids: 0
max_batch_size: 1
backend {
inputs [
{
name: “INPUT0”
dims: [3, 1080, 1920]
},
{
name: “SOURCE_ID”
dims: [1]
}
]
triton {
model_name: “centerface”
version: -1
model_repo {
root: “./centerface”
log_level: 1
tf_gpu_memory_fraction: 0.2
tf_disable_soft_placement: 0
}
}
}

preprocess {
network_format: IMAGE_FORMAT_RGB
tensor_order: TENSOR_ORDER_LINEAR
maintain_aspect_ratio: 0
normalize {
scale_factor: 1.0
channel_offsets: [0, 0, 0]
}
}

postprocess {
labelfile_path: “./centerface/centerface/centerface_labels.txt”
other{}
}

custom_lib {
path: “./centerface/libnvdstriton_custom_impl_ensemble.so”
}

extra {
custom_process_funcion: “CreateInferServerCustomProcess”
}
}
input_control {
process_mode: PROCESS_MODE_FULL_FRAME
interval: 0
}

output_control {
output_tensor_meta: true
}

This is the custom backend function in the cpp file nvdsinferserver_custom_process_ensemble.cpp contents :

class NvInferServerCustomProcess : public dsis::IInferCustomProcessor {
private:
std::map<uint64_t, std::vector> _streamFeedback;
std::mutex _streamMutex;

public:
~NvInferServerCustomProcess() override = default;
/** override function
* Specifies supported extraInputs memtype in extraInputProcess()
*/
void supportInputMemType(dsis::InferMemType& type) override { type = dsis::InferMemType::kCpu; }

/** Override function
 * check whether custom loop process needed.
 * If return True, extraInputProcess() and inferenceDone() runs in order per stream_ids
 * This is usually for LSTM loop purpose. FasterRCNN does not need it.
 * The code for requireInferLoop() conditions just sample when user has
 * a LSTM-like Loop model and requires loop custom processing.
 * */
bool requireInferLoop() const override { return false; }

/**
 * Override function
 * Do custom processing on extra inputs.
 * @primaryInput is already preprocessed. DO NOT update it again.
 * @extraInputs, do custom processing and fill all data according the tensor shape
 * @options, it has most of the common Deepstream metadata along with primary data.
 *           e.g. NvDsBatchMeta, NvDsObjectMeta, NvDsFrameMeta, stream ids...
 *           see infer_ioptions.h to see all the potential key name and structures
 *           in the key-value table.
 */
NvDsInferStatus extraInputProcess(
    const std::vector<dsis::IBatchBuffer*>&
        primaryInputs,  // primary tensor(image) has been processed
    std::vector<dsis::IBatchBuffer*>& extraInputs, const dsis::IOptions* options) override
{
    INFER_ASSERT(primaryInputs.size() > 0);
    INFER_ASSERT(extraInputs.size() >= 1);
    // primary input tensor: INPUT [batch, channel, height, width]
    dsis::InferBufferDescription input0Desc = primaryInputs[0]->getBufDesc();
    // extra input tensor: STREAM_ID [batch, 1]
    // dsis::InferBufferDescription extra1Desc = extraInputs[0]->getBufDesc(); 
    dsis::InferBufferDescription sourceIdDesc = extraInputs[0]->getBufDesc();  

    // INFER_ASSERT(extra1Desc.dataType == dsis::InferDataType::kInt32);
    // INFER_ASSERT(extra1Desc.elementSize == sizeof(int32_t));  // bytes per element

    // INFER_ASSERT(streamIdDesc.dataType == dsis::InferDataType::kInt32);
    INFER_ASSERT(sourceIdDesc.dataType == dsis::InferDataType::kInt32);
    // INFER_ASSERT(streamIdDesc.elementSize == sizeof(int32_t));
    INFER_ASSERT(sourceIdDesc.elementSize == sizeof(int32_t));

    INFER_ASSERT(!strOfBufDesc(input0Desc).empty());
    LOG_DEBUG("extraInputProcess: primary input %s", strOfBufDesc(input0Desc).c_str());
    LOG_DEBUG("extraInputProcess: extra input %s", strOfBufDesc(sourceIdDesc).c_str());

    // batch size must be get from primary input tensor.
    // extra inputs 'image_shape' does not have a batch size in this specific model
    int batchSize = input0Desc.dims.d[0];
    // INFER_ASSERT(extra1Desc.dims.numDims == 2 && extra1Desc.dims.d[0] == batchSize);
    // INFER_ASSERT(streamIdDesc.dims.numDims == 2 && streamIdDesc.dims.d[0] == batchSize);
    INFER_ASSERT(sourceIdDesc.dims.numDims == 2 && sourceIdDesc.dims.d[0] == batchSize);
    INFER_ASSERT(batchSize >= 1);

    if (!options) {
        LOG_ERROR("custom process does not receive IOptions");
        return NVDSINFER_CUSTOM_LIB_FAILED;
    }

    NvDsBatchMeta* batchMeta = nullptr;
    std::vector<NvDsFrameMeta*> frameMetaList;
    NvBufSurface* bufSurf = nullptr;
    std::vector<NvBufSurfaceParams*> surfParamsList;
    std::vector<uint64_t> streamIds;
    int64_t unique_id = 0;
    INFER_ASSERT(options->getValueArray(OPTION_NVDS_SREAM_IDS, streamIds) == NVDSINFER_SUCCESS);
    INFER_ASSERT(streamIds.size() == (uint32_t)batchSize);

    // get NvBufSurface
    if (options->hasValue(OPTION_NVDS_BUF_SURFACE)) {
        INFER_ASSERT(options->getObj(OPTION_NVDS_BUF_SURFACE, bufSurf) == NVDSINFER_SUCCESS);
    }
    INFER_ASSERT(bufSurf);

    // get NvDsBatchMeta
    if (options->hasValue(OPTION_NVDS_BATCH_META)) {
        INFER_ASSERT(options->getObj(OPTION_NVDS_BATCH_META, batchMeta) == NVDSINFER_SUCCESS);
    }
    INFER_ASSERT(batchMeta);

    // get all frame meta list into vector<NvDsFrameMeta*>
    if (options->hasValue(OPTION_NVDS_FRAME_META_LIST)) {
        INFER_ASSERT(
            options->getValueArray(OPTION_NVDS_FRAME_META_LIST, frameMetaList) ==
            NVDSINFER_SUCCESS);
    }

    // get unique_id
    if (options->hasValue(OPTION_NVDS_UNIQUE_ID)) {
        INFER_ASSERT(options->getInt(OPTION_NVDS_UNIQUE_ID, unique_id) == NVDSINFER_SUCCESS);
    }

    // get all surface params list into vector<NvBufSurfaceParams*>
    if (options->hasValue(OPTION_NVDS_BUF_SURFACE_PARAMS_LIST)) {
        INFER_ASSERT(
            options->getValueArray(OPTION_NVDS_BUF_SURFACE_PARAMS_LIST, surfParamsList) ==
            NVDSINFER_SUCCESS);
    }

    // fill extra input tensor "STREAM_ID[-1,1]"
    // int32_t* streamIdTensor = (int32_t*)extraInputs[0]->getBufPtr(0);

    // for (int iBatch = 0; iBatch < batchSize; ++iBatch) {
    //     streamIdTensor[iBatch] = streamIds[iBatch];
    // }

    int32_t* sourceIdTensor = (int32_t*)extraInputs[0]->getBufPtr(0);

    for (int iBatch = 0; iBatch < batchSize; ++iBatch) {
        sourceIdTensor[iBatch] = frameMetaList[iBatch]->source_id;
    }

    return NVDSINFER_SUCCESS;
}

/** Override function
 * Custom processing for inferenced output tensors.
 * output memory types is controlled by gst-nvinferserver config file
 *       infer_config {
 *         backend {  output_mem_type: MEMORY_TYPE_CPU }
 *     }
 * User can even attach parsed metadata into GstBuffer from this function
 */
NvDsInferStatus inferenceDone(
    const dsis::IBatchArray* outputs, const dsis::IOptions* inOptions) override
{
    std::vector<uint64_t> streamIds;
    INFER_ASSERT(
            inOptions->getValueArray(OPTION_NVDS_SREAM_IDS, streamIds) == NVDSINFER_SUCCESS);
    INFER_ASSERT(!streamIds.empty());
    uint32_t batchSize = streamIds.size();

    // add output tensors into map
    std::unordered_map<std::string, const dsis::IBatchBuffer*> tensors;

    for (uint32_t iTensor = 0; iTensor < outputs->getSize(); ++iTensor) {
        const dsis::IBatchBuffer* outTensor = outputs->getBuffer(iTensor);
        INFER_ASSERT(outTensor);
        auto desc = outTensor->getBufDesc();
        LOG_DEBUG("out tensor: %s, desc: %s \n", CSTR(desc.name), strOfBufDesc(desc).c_str());
        tensors.emplace(desc.name, outTensor);
    }

    auto outputBuf = tensors["INFERENCE_COUNT"];
    INFER_ASSERT(outputBuf);
    auto outDesc = outputBuf->getBufDesc();
    INFER_ASSERT(outDesc.dims.numDims == 2 && outDesc.dims.d[1] == 1);
    INFER_ASSERT(outDesc.dims.d[0] == (int32_t)batchSize);

    /* Read the inference count output and print using debug logs. */
    int32_t* outInferenceCounts = (int32_t*)outputBuf->getBufPtr(0);
    for (int idx = 0; idx < outDesc.dims.d[0]; ++idx) {
        LOG_DEBUG("Inference count = %d for batch index %d\n", outInferenceCounts[idx], idx);
        (void)outInferenceCounts; // avoid warning
    }

    /* Parse the classifier outputs. */
    std::vector<const dsis::IBatchBuffer*> classTensorOutput;
    std::vector<std::vector<std::string>> labels;

    classTensorOutput.push_back(tensors["CAR_COLOR"]);
    classTensorOutput.push_back(tensors["CAR_MAKE"]);
    classTensorOutput.push_back(tensors["VEHICLE_TYPE"]);

    labels.push_back(kCarColorLabels);
    labels.push_back(kCarMakeLabels);
    labels.push_back(kVehicleTypeLabels);

    /* Get the number of attributes supported by the classifier. */
    unsigned int numAttributes = classTensorOutput.size();
    /* Iterate through all the output coverage layers of the classifier.  */
    for (unsigned int l = 0; l < numAttributes; l++) {
        const dsis::IBatchBuffer* outputBuf = classTensorOutput[l];
        INFER_ASSERT(outputBuf);
        auto outDesc = outputBuf->getBufDesc();
        INFER_ASSERT(outDesc.dims.numDims == 4 && outDesc.dims.d[1] == (int)labels[l].size());
        INFER_ASSERT(outDesc.dims.d[0] == (int32_t)batchSize);
    }

    for (unsigned int idx = 0; idx < batchSize; ++idx) {
        std::string attrString;
        std::vector<NvDsInferAttribute> attributes;

        for (unsigned int l = 0; l < numAttributes; l++) {
            const dsis::IBatchBuffer* outputBuf = classTensorOutput[l];
            auto outDesc = outputBuf->getBufDesc();

            float* outPtr = (float*)outputBuf->getBufPtr(0) + idx * outDesc.dims.d[1];
            unsigned int numClasses = outDesc.dims.d[1];
            float maxProbability = 0;
            bool attrFound = false;
            NvDsInferAttribute attr;

            /* Iterate through all the probabilities that the object belongs to
             * each class. Find the maximum probability and the corresponding class
             * which meets the minimum threshold. */
            for (unsigned int c = 0; c < numClasses; c++) {
                float probability = outPtr[c];
                if (probability > kClassifierThreshold &&
                        probability > maxProbability) {
                    maxProbability = probability;
                    attrFound = true;
                    attr.attributeIndex = l;
                    attr.attributeValue = c;
                    attr.attributeConfidence = probability;
                }
            }
            if (attrFound) {
                if (labels.size() > attr.attributeIndex &&
                        attr.attributeValue < labels[attr.attributeIndex].size())
                    attr.attributeLabel =
                        strdup(labels[attr.attributeIndex][attr.attributeValue].c_str());
                else
                    attr.attributeLabel = nullptr;
                attributes.push_back(attr);
                if (attr.attributeLabel)
                    attrString.append(attr.attributeLabel).append(" ");
            }
        }

        InferClassificationOutput output;

        for (auto& attr : attributes) {
            InferAttribute safeAttr;
            static_cast<NvDsInferAttribute&>(safeAttr) = attr;
            safeAttr.safeAttributeLabel = safeStr(attr.attributeLabel);
            safeAttr.attributeLabel = (char *) safeAttr.safeAttributeLabel.c_str();
            output.attributes.emplace_back(safeAttr);
            free(attr.attributeLabel);
        }

        INFER_ASSERT(attachClassificationMeta(inOptions, output, idx)
                == NVDSINFER_SUCCESS);

    }

    return NVDSINFER_SUCCESS;
}

/** override function
 * Receiving errors if anything wrong inside lowlevel lib
 */
void notifyError(NvDsInferStatus s) override
{
    std::unique_lock<std::mutex> locker(_streamMutex);
    _streamFeedback.clear();
}

private:
/** function for loop processing only. not requried for fasterRCNN
/
NvDsInferStatus feedbackStreamInput(
const dsis::IBatchArray outputs, const dsis::IOptions* inOptions);

/**
 * attach bounding boxes into NvDsBatchMeta and NvDsFrameMeta
 */
NvDsInferStatus attachClassificationMeta(
    const dsis::IOptions* inOptions, InferClassificationOutput& objInfo,
    uint32_t batchIdx);

};

/** Implementation to create a custom processor for DeepStream Triton

  • plugin (nvinferserver) to do custom extra input preprocess and custom
  • postprocess on triton based models.
    */

extern “C” {
dsis::IInferCustomProcessor*
CreateInferServerCustomProcess(const char* config, uint32_t configLen)
{
return new NvInferServerCustomProcess();
}
}

To be more specific major changes were made in the extraInputProcess to accomodate Source_id in extraInputs layer data

But for the same query as given above, I get the following error logs :

Setting pipeline to PAUSED …
W0903 05:27:56.305753 149199 metrics.cc:512] Unable to get power limit for GPU 0. Status:Success, value:0.000000
W0903 05:27:56.305842 149199 metrics.cc:530] Unable to get power usage for GPU 0. Status:Success, value:0.000000
W0903 05:27:56.305860 149199 metrics.cc:554] Unable to get energy consumption for GPU 0. Status:Success, value:0
Model initialized
Error connecting: [Errno 111] Connection refused
INFO: infer_trtis_backend.cpp:218 TrtISBackend id:1 initialized model: centerface
Pipeline is PREROLLING …
W0903 05:27:57.306389 149199 metrics.cc:512] Unable to get power limit for GPU 0. Status:Success, value:0.000000
W0903 05:27:57.306417 149199 metrics.cc:530] Unable to get power usage for GPU 0. Status:Success, value:0.000000
W0903 05:27:57.306421 149199 metrics.cc:554] Unable to get energy consumption for GPU 0. Status:Success, value:0
nvdsinferserver_custom_process_ensemble.cpp:357 ASSERT(outputBuf)
Error connecting: [Errno 111] Connection refused
Aborted (core dumped)
I0903 05:27:58.455949 149325 pb_stub.cc:1510] Non-graceful termination detected.

Am I going on the right track and can you point out on why these errors would come up?

15
  1. Noticing you are using NVIDIA GeForce MX230, which is Pascal architecture. it is not in the DeepStream supported device list. please refer to this link. can the deepstream-test1 with nvinfer and nvinferserver run well on your device?
  2. what are model’s outputs? from the error “ASSERT(outputBuf)”, this issue is in NvDsInferStatus inferenceDone.
16

It is working in my system, its just that the decoder module doesn’t work and it is not included in the test pipeline query :

gst-launch-1.0 videotestsrc ! nvvideoconvert ! “video/x-raw(memory:NVMM), width=1920, height=1080” ! m.sink_0 nvstreammux name=m width=1920 height=1080 batch_size=3 ! nvinferserver config-file-path=config.txt ! nvstreamdemux name=d d.src_0 ! nvvideoconvert ! autovideosink sync=0

So, the latest update that I did was to bring back the stream ID in the extraInput along with the addition of source ID in the custom input layer implementation function in nvdsinferserver_custom_process_ensemble.cpp which solved the problem present in inferenceDone. It was updated as follows :

NvDsInferStatus extraInputProcess(
const std::vectordsis::IBatchBuffer*&
primaryInputs, // primary tensor(image) has been processed
std::vectordsis::IBatchBuffer*& extraInputs, const dsis::IOptions* options) override
{
INFER_ASSERT(primaryInputs.size() > 0);
INFER_ASSERT(extraInputs.size() >= 1);
// primary input tensor: INPUT [batch, channel, height, width]
dsis::InferBufferDescription input0Desc = primaryInputs[0]->getBufDesc();
// extra input tensor: STREAM_ID [batch, 1]
dsis::InferBufferDescription streamIdDesc = extraInputs[0]->getBufDesc();
dsis::InferBufferDescription sourceIdDesc = extraInputs[1]->getBufDesc();

    INFER_ASSERT(streamIdDesc.dataType == dsis::InferDataType::kInt32);
    INFER_ASSERT(sourceIdDesc.dataType == dsis::InferDataType::kInt32);
    INFER_ASSERT(streamIdDesc.elementSize == sizeof(int32_t));
    INFER_ASSERT(sourceIdDesc.elementSize == sizeof(int32_t));

    INFER_ASSERT(!strOfBufDesc(input0Desc).empty());
    LOG_DEBUG("extraInputProcess: primary input %s", strOfBufDesc(input0Desc).c_str());
    LOG_DEBUG("extraInputProcess: extra input stream ID %s", strOfBufDesc(streamIdDesc).c_str());
    LOG_DEBUG("extraInputProcess: extra input source ID %s", strOfBufDesc(sourceIdDesc).c_str());

    // batch size must be get from primary input tensor.
    // extra inputs 'image_shape' does not have a batch size in this specific model
    int batchSize = input0Desc.dims.d[0];
    INFER_ASSERT(streamIdDesc.dims.numDims == 2 && streamIdDesc.dims.d[0] == batchSize);
    INFER_ASSERT(sourceIdDesc.dims.numDims == 2 && sourceIdDesc.dims.d[0] == batchSize);
    INFER_ASSERT(batchSize >= 1);

    if (!options) {
        LOG_ERROR("custom process does not receive IOptions");
        return NVDSINFER_CUSTOM_LIB_FAILED;
    }

    NvDsBatchMeta* batchMeta = nullptr;
    std::vector<NvDsFrameMeta*> frameMetaList;
    NvBufSurface* bufSurf = nullptr;
    std::vector<NvBufSurfaceParams*> surfParamsList;
    std::vector<uint64_t> streamIds;
    int64_t unique_id = 0;
    INFER_ASSERT(options->getValueArray(OPTION_NVDS_SREAM_IDS, streamIds) == NVDSINFER_SUCCESS);
    INFER_ASSERT(streamIds.size() == (uint32_t)batchSize);

    // get NvBufSurface
    if (options->hasValue(OPTION_NVDS_BUF_SURFACE)) {
        INFER_ASSERT(options->getObj(OPTION_NVDS_BUF_SURFACE, bufSurf) == NVDSINFER_SUCCESS);
    }
    INFER_ASSERT(bufSurf);

    // get NvDsBatchMeta
    if (options->hasValue(OPTION_NVDS_BATCH_META)) {
        INFER_ASSERT(options->getObj(OPTION_NVDS_BATCH_META, batchMeta) == NVDSINFER_SUCCESS);
    }
    INFER_ASSERT(batchMeta);

    // get all frame meta list into vector<NvDsFrameMeta*>
    if (options->hasValue(OPTION_NVDS_FRAME_META_LIST)) {
        INFER_ASSERT(
            options->getValueArray(OPTION_NVDS_FRAME_META_LIST, frameMetaList) ==
            NVDSINFER_SUCCESS);
    }

    // get unique_id
    if (options->hasValue(OPTION_NVDS_UNIQUE_ID)) {
        INFER_ASSERT(options->getInt(OPTION_NVDS_UNIQUE_ID, unique_id) == NVDSINFER_SUCCESS);
    }

    // get all surface params list into vector<NvBufSurfaceParams*>
    if (options->hasValue(OPTION_NVDS_BUF_SURFACE_PARAMS_LIST)) {
        INFER_ASSERT(
            options->getValueArray(OPTION_NVDS_BUF_SURFACE_PARAMS_LIST, surfParamsList) ==
            NVDSINFER_SUCCESS);
    }

    // fill extra input tensor "STREAM_ID[-1,1]"
    int32_t* streamIdTensor = (int32_t*)extraInputs[0]->getBufPtr(0);

    for (int iBatch = 0; iBatch < batchSize; ++iBatch) {
        streamIdTensor[iBatch] = streamIds[iBatch];
    }

    int32_t* sourceIdTensor = (int32_t*)extraInputs[1]->getBufPtr(0);

    for (int iBatch = 0; iBatch < batchSize; ++iBatch) {
        sourceIdTensor[iBatch] = 1321;
    }
    return NVDSINFER_SUCCESS;
}

Note here that I am trying to keep a custom value 1321 for source ID

The config file is unchanged.
But I tried logging the source ID by accessing it in model.py (triton inference server request handling code) using this code snippet :

source_id_tensor = pb_utils.get_input_tensor_by_name(request, “SOURCE_ID”)
print(source_id_tensor.as_numpy())

After running the pipeline, I got the following logs:

Setting pipeline to PAUSED …
W0905 06:00:45.736411 182434 metrics.cc:512] Unable to get power limit for GPU 0. Status:Success, value:0.000000
W0905 06:00:45.736521 182434 metrics.cc:530] Unable to get power usage for GPU 0. Status:Success, value:0.000000
W0905 06:00:45.736557 182434 metrics.cc:554] Unable to get energy consumption for GPU 0. Status:Success, value:0
Model initialized
Error connecting: [Errno 111] Connection refused
INFO: infer_trtis_backend.cpp:218 TrtISBackend id:1 initialized model: centerface
[[0]]
Pipeline is PREROLLING …
W0905 06:00:46.737389 182434 metrics.cc:512] Unable to get power limit for GPU 0. Status:Success, value:0.000000
W0905 06:00:46.737413 182434 metrics.cc:530] Unable to get power usage for GPU 0. Status:Success, value:0.000000
W0905 06:00:46.737416 182434 metrics.cc:554] Unable to get energy consumption for GPU 0. Status:Success, value:0
Caught SIGSEGV
Error connecting: [Errno 111] Connection refused
W0905 06:00:47.737898 182434 metrics.cc:512] Unable to get power limit for GPU 0. Status:Success, value:0.000000
W0905 06:00:47.737933 182434 metrics.cc:530] Unable to get power usage for GPU 0. Status:Success, value:0.000000
W0905 06:00:47.737937 182434 metrics.cc:554] Unable to get energy consumption for GPU 0. Status:Success, value:0

Here the pipeline is running but it is not printing the custom source ID that I provided (1321) but instead just prints [[0]] for the source ID.
Am I doing anything wrong here or is there anyway I can change the value of Source ID that is being printed to a custom value?

17

please refer to this topic.