Deepstream 6 osd library draws on wrong surface

[etibon]

NVIDIA Jetson Xavier NX (Developer Kit Version)
L4T 32.6.1 [ JetPack 4.6 ]
Ubuntu 18.04.5 LTS
Kernel Version: 4.9.253-tegra
CUDA 10.2.300
CUDA Architecture: 7.2
OpenCV version: 4.1.1
OpenCV Cuda: NO
CUDNN: 8.2.1.32
TensorRT: 8.0.1.6
Vision Works: 1.6.0.501
VPI: ii libnvvpi1 1.1.12 arm64 NVIDIA Vision Programming Interface library
Vulcan: 1.2.70
Deepstream 6.0
Issue type: Bug

I have written a custom gstreamer/deepstream plugin (based on the gst-example-plugin example supplied with Deepstream) that, among other things, draws some boxes in the frame using nvll_osd_draw_rectangles(). It works fine with a single video source, however when I set up the pipeline with two sources (using GstNvStreamMux and outputting the result using NvMultistreamTiler), the boxes seem to be overlaid on a source at random, resulting in an unstable display with overlay boxes flashing from one tile to another at random.

Exactly the same code works perfectly under DS5.1.

The pipeline looks like this:
UriDecodeBin(source 1)—NvStreamMux - nvinfer - nvtracker - nvvideoconvert - [custom plugin] - NvMultistreamTiler - NvDsOsd - [output bin]
UriDecodeBin(source 2)-|

The NvOSD_FrameRectParams parameter to nvll_osd_draw_rectangles has it’s buf_ptr parameter set to NvBufSurface(surface)->surfaceList[frame_meta->batch_id], where frame_meta is the NvDsFrameMeta* corresponding to the current frame in the batch.

Simplified code follows:

int overlayBoxes(GstApp *myapp, GstBuffer *inbuf) {

NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta(inbuf);

GstMapInfo inmap = GST_MAP_INFO_INIT;

if (!gst_buffer_map(inbuf, &inmap, GST_MAP_READ)) {
    GST_ELEMENT_ERROR (sctrafficdir, RESOURCE, FAILED,
                       ("Unable to map info from buffer"), ("Unable to map info from buffer"));
    return GST_FLOW_ERROR;
}

for (NvDsMetaList *l_frame = batch_meta->frame_meta_list; l_frame != NULL;  l_frame = l_frame->next) {
    NvDsFrameMeta *frame_meta = (NvDsFrameMeta *) (l_frame->data);

    auto source_id = frame_meta->source_id;
    <do other stuff>
    NvBufSurface *surface = NULL;
    surface = (NvBufSurface *) inmap.data;
    NvOSD_FrameRectParams rect_params;
    NvOSD_RectParams rects[x_size * y_size]
   <push box defs to rects incrementing rect_params.num_rects>
   rect_params.rect_params_list = (NvOSD_RectParams *) &rects;
   rect_params.buf_ptr = &surface->surfaceList[frame_meta->batch_id];
   rect_params.mode = MODE_CPU;
   if (nvll_osd_draw_rectangles(sctrafficdir->nvosd_context, &rect_params) == -1) {
       GST_ELEMENT_ERROR (sctrafficdir, RESOURCE, FAILED,
                          ("Unable to draw rectangles"), ("Unable to draw rectangles"));
       return GST_FLOW_ERROR;
   } } // for} // func

Some pointers as to what I could be doing wrong would be greatly appreciated :)

[kesong]

Why not draw it by nvdsosd?

[etibon]

Because I need to draw more than MAX_ELEMENTS_IN_DISPLAY_META and so I thought that it would produce cleaner code doing this way than having to fetch multiple display_meta objects from the pool. It was working fine in DS5.1, but broke in DS 6.

[kesong]

Suggest to fetch multiple display_meta objects from the pool.