extraInput to triton-server with multi dimension

• Hardware Platform: GPU
• DeepStream Version: 6.2
• TensorRT Version: 8.5
• NVIDIA GPU Driver Version: 525.105.17
• Issue Type: questions


In our use case, we want to use nvinferserver plugin SGIE into FULL_FRAME_MODE.
Below is our pipeline.
stream_mux->nvinferserver(face detector - PGIE)->queue->tracker->nviferserver(SGIE) → nvosd
We are operating SGIE into FULL_FRAME_MODE and we want to send additional input to triton-server using extraInput along with primary input frame.

Can you guide us on how to send extraInput with multiple dimensions?
extra input shape like [3, 2, 6]. I am able to send extra input with dimensions 6, 6 but I want to send multiple frame inputs to triton.

Are the frames fixed? Will the frames changes with different video frames? Have you read the sample /opt/nvidia/deepstream/deepstream/sources/objectDetector_FasterRCNN/nvdsinfer_custom_impl_fasterRCNN/nvdsinferserver_custom_process.cpp?

We referred to the same example already and it has only a single dynamic dimension implemented which we already have we wanted to have multiple dynamic dimensions to support input image shape (2, 1080, 1920, 3) and bounding boxes shape (2, 4, 4) → (no. of frames, no. bboxes, box size).

There is dimension information in dsis::InferBufferDescription interface. You can copy the data you prepared into the extra input buffers according to the actaul dimension.

The only problem is where you get the no. bboxes and box size? Are they the inferencing results from the last frame? Can you explain more about your model’s input layers?

Our PGIE output is bounding boxes with the shape of (2, 4, 4) → (no. of frames, no. bboxes, box size). And we applied some arithmetic operations and send these bounding boxes to our secondary model which we performed in FULL_FRAME mode and their inputs are:
frames: (2, 1080, 1920, 3) → (no. of frames, height, width, channels)
bounding boxes: (2, 4, 4) → (no. of frames, no. bboxes, box size).

This is our pipeline.

So, the problem is in SGIE, we are receiving input frame shape is (8, 1080, 1920, 3) → (no. of frames, height, width, channels) and the bounding boxes shape is (8, 4) → (no. of boxes, box size). Which is wrong.

It depends on your post-processing. Please check your post-processing for your PGIE.

Hello @Fiona.Chen,

I have said my PGIE is working correctly.

And it gives a correct output.
But my actual problem is when I send PGIE outputs to SGIE’s metadata in PGIE’s custom parser. When I read that data from SGIE at that time input shapes are different.

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

could you share the pgie’s model config.pbtxt and nvinferserver configuration file? how did you know the pgie 's output is (2, 4, 4) not (8, 4) ?

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.