Got frame of rtmp source 1 in rtmp source 2 nvdsosd's callback

deepstream docker: nvcr.io/nvidia/deepstream:5.0-20.07-devel
GPU: GeForce GTX 1080 Ti or 2080 Ti

Hi, we have a multi-source application, our deepstream pipeline is showed below:


we set 4 rtmp input, and the osd’s callback function is:

GstPadProbeReturn osd_sink_pad_buffer_probe(GstPad* pad, GstPadProbeInfo* info, gpointer u_data)
{
int rtmpSourceId = (int)u_data;
GstBuffer* buf = (GstBuffer*) info->data;
GstMapInfo map;
if(!gst_buffer_map(buf, &map, GST_MAP_READ))
{
printf(“Gst buffer is empty\n”);
return GST_PAD_PROBE_OK;
}

NvBufSurface* nvsurface = ((NvBufSurface*) map.data);

NvDsBatchMeta* batch_meta = gst_buffer_get_nvds_batch_meta(buf);
for(NvDsMetaList* frame_meta = batch_meta->frame_meta_list; frame_meta != NULL; frame_meta = frame_meta->next)
{
	
    NvDsFrameMeta *frame_meta_data = (NvDsFrameMeta*)(frame_meta->data);

	int imageWidth = frame_meta_data->source_frame_width;
	int imageHeight = frame_meta_data->source_frame_height;
	
	// save frame
    cv::Mat image = nvBufferSurface2CVmat(nvsurface, frame_meta_data->batch_id);
    std::string imagePath = "./" + std::toString(rtmpSourceId) + "_" + std::toString(getCurrentTime()) + ".jpg";
    cv::imwrite(imagePath, image);
}

}

cv::Mat nvBufferSurface2CVmat(NvBufSurface* nvsurface, uint batch_id)
{
#define FRAME_BTYE_SIZE = 1920x1080x4
if(nvsurface->surfaceList[batch_id].dataSize > FRAME_BTYE_SIZE)
{
printf(“nvSurface buffer’s size is beyond common size(%d)\n”, FRAME_BTYE_SIZE);
exit(-1);
}

char* pFrameVideoDataBuffer = nullptr;
if (nvsurface->memType == NvBufSurfaceMemType::NVBUF_MEM_DEFAULT)
{
    pFrameVideoDataBuffer = new char[FRAME_BTYE_SIZE];
	cudaMemcpy(pFrameVideoDataBuffer, nvsurface->surfaceList[batch_id].dataPtr, nvsurface->surfaceList[batch_id].dataSize, cudaMemcpyDeviceToHost);
}
else
{
    printf("nvsurface memory type: %d\n", nvsurface->memType);
    exit(-1);
}

try{
    cv::Mat image = cv::Mat(cv::Size(nvsurface->surfaceList[batch_id].width, nvsurface->surfaceList[batch_id].height), CV_8UC4, \
												(char*)pFrameVideoDataBuffer, cv::Mat::AUTO_STEP);
    delete[] pFrameVideoDataBuffer;
	cvtColor(image, image, cv::COLOR_RGBA2BGR);
}
catch(...)
{
    delete[] pFrameVideoDataBuffer;
	printf("cvtColor exception\n");
	return cv::Mat();
}

return image;

}

/* problem */

we found that some saved pictures were not from corresponding rtmp source, for example, the content of saved picture file 0_1612341697548.jpg is origin from rtmp source 1,
rather than rtmp source 0.

we also found that the occurring frequency is related to the batch size, which can be summarized as follows:

“source num” “batch size” “occurring frequency”
1 1 0
2 2 0
3 2 high
3 3 high
3 4 low
4 3 high
4 3 high
4 8 low

and we also print the address of “nvsurface->surfaceList[batch_id].dataPtr”, every rtmp source has four fixed addresses which are recyclly used.

RtmpSourceId: 0, dataPtr: 0x7fa4b6c00000
RtmpSourceId: 0, dataPtr: 0x7fa4b7400000
RtmpSourceId: 0, dataPtr: 0x7fa4b7c00000
RtmpSourceId: 0, dataPtr: 0x7fa4b8400000

RtmpSourceId: 2, dataPtr: 0x7fa4b8c00000
RtmpSourceId: 2, dataPtr: 0x7fa4b9400000
RtmpSourceId: 2, dataPtr: 0x7fa4b9c00000
RtmpSourceId: 2, dataPtr: 0x7fa4ba400000

RtmpSourceId: 1, dataPtr: 0x7fa4bac00000
RtmpSourceId: 1, dataPtr: 0x7fa4bb400000
RtmpSourceId: 1, dataPtr: 0x7fa4bbc00000
RtmpSourceId: 1, dataPtr: 0x7fa4bc400000

RtmpSourceId: 3, dataPtr: 0x7fa4bcc00000
RtmpSourceId: 3, dataPtr: 0x7fa4bd400000
RtmpSourceId: 3, dataPtr: 0x7fa4bdc00000
RtmpSourceId: 3, dataPtr: 0x7fa4be400000

the addresses are adjacent, and the interval is 0x800000, beyond 0x7e9000 (1920x1080x4).

Thansk!

Please use ‘source_id’ in NvDsFrameMeta to identify the source. _NvDsFrameMeta — Deepstream Deepstream Version: 5.0 documentation

Thanks for your reply!
what not mentioned above is that the NvDsMetaList is correct with the wrong frame,
even source_id used, we also can’t get both right frame and NvDsMetaList at the same time.

Do you mean the batch-size of nvstreammux?

Yes

If replace the fakesink to display sink, can you see correct video?

Because of the warning “libEGL warning : DRI2: failed to authenticate”, the display sink nveglglessink didn’t work in our docker.
we have pushed the frames as rtmp in osd’s callback function, in the output rtmp, the video also showed wrong pictures.

Hi, Fiona.Chen, could you tell me what probably has caused this?

Deepstream can support multiple input and multiple output correctly. So I can not tell you the reason of your problem. Can you use deepstream-app to reproduce the problem? You can input multiple rtmp streams with uri or multiuri type DeepStream Reference Application - deepstream-app — DeepStream DeepStream Version: 5.0 documentation and output multiple streams as encoded video files DeepStream Reference Application - deepstream-app — DeepStream DeepStream Version: 5.0 documentation

Hi, Fiona.Chen, I have given a try with deepstream-app, there were still wrong pictures in the output video files.
source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8_gpu1.txt (6.2 KB)

however, with the tiled-display, the displayed videos is correct without wrong picture.

This is the known bug for deepstream 5.0 version. We have fixed the bug in deepstream 5.0.1. Announcing DeepStream 5.0.1 - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums. Please upgrade to deepstream 5.0.1.

https://ngc.nvidia.com/catalog/containers/nvidia:deepstream

Hi, Fiona.Chen, Thanks for your replay!
Dose deepstream 5.0.1 support the rtx 30xx series?

DeepStream support standard v4l2 driver camera. The vendor and type means nothing.

Hi, Fiona.Chen, I mean Nvidia Geforce RTX 30xx series GPU card.
I have tried to install deepstream 5.0.1 with cuda 11.2.0 and tensorrt 7.2, there were some libs not matched those deepstream 5.0.1 linked.

DeepStreamSDK support CUDA 10.2 now. cuda11.2 + trnsorrt7.2 may not work.