How can I get a gpu RGBA buffer directly by setting a probe on the nvosd sink pad?

The DeepStream 3.0 Plugin Manual states the following:

The plugin accepts an RGBA buffer with attached metadata from the upstream component.

I want to get gpu image data directly and convert it only when needed.

The example shows how to get the metadata.

But I couldn’t find a way to get gpu image data directly.

Can you tell me how to access image information directly?

Will you plan to port to Deepstream 4.0
You can refer to https://devtalk.nvidia.com/default/topic/1061492/deepstream-sdk/deepstream-sdk-4-0-faq/ -> How to get original NV12 frame buffer
And dsexample plugin also has the similar example to get frame buffer.

Thank you, ChrisDing.

It seems to be exactly what I was looking for.

I express my deep gratitude.

Someone else is working on a deepstream model file, version 3.0.

Does it matter if I change to version 4.0?

Ah, This is a question that is out of the question.

When I run the model file he created with the “deepstream test” example,

the event type index does not come out correctly. What could be the problem?

–Does it matter if I change to version 4.0?
No. Suggest port to 4.0 as soon as possible.

– the event type index does not come out correctly.
What’s event? Can you give more details.

You are right. In v3.0, it was not possible to build code.

Checking the plugin documentation,

it is estimated that the RGBA buffer is coming into the nvosd sink buffer.

So I am implementing logic to get GPU RGBA buffer and meta data from sink pad.

Roughly as follows:

GstBuffer *pBuf = (GstBuffer*)pInfo->data;

GstMapInfo mapInfo;
if (pBuf != NULL && gst_buffer_map(pBuf, &mapInfo, GST_MAP_READ)) {
	NvBufSurface *pSurf = (NvBufSurface*)mapInfo.data;
	// get image data
}
gst_buffer_unmap(pBuf, &mapInfo);


while ((pMeta = gst_buffer_iterate_meta(pBuf, &pState)) != NULL) {
	if (gst_meta_api_type_has_tag(pMeta->info->api, _nvdsmeta_quark) == false) continue;
	
	pDSMeta = (NvDsMeta*)pMeta;
	if (pDSMeta && pDSMeta->meta_type == NVDS_META_FRAME_INFO) {
		pFrameMeta = (NvDsFrameMeta *)pDSMeta->meta_data;
		
		// get meta data
	}
}

But moving the image buffer pointer to another class seems to terminate abnormally.

I had similar symptoms with v1.5 and it seemed to be fine if I implemented it in the same file.

I don’t know the exact cause.


And what I said “event type index” is the line number of the event string pointed to by the label file (starting from 0).

In the example, it seemed to distinguish the events by the row index of the string in the label file,

but the model file created by someone could not distinguish the events because the value was negative.

Thank you. ChrisDing.
Thanks to this I succeeded in importing gpu image data.
However, if you stop the pipeline and recreate it, the following message appears:

Cuda error in file src/implicit_gemm.cu at line 648: invalid device pointer
cuda/customWinogradConvActLayer.cpp:240:
virtual void nvinfer1::rt::cuda::WinogradConvActLayer::allocateResources(const nvinfer1::rt::CommonContext&):
Assertion `convolutions.back().get()’ failed.

This is roughly the routine with the pipeline.

GstElement* pPipe = gst_parse_launc(szPipe, &pError);

nProbeID = gst_pad_add_probe(sinkpad_nvosd, …

gst_element_set_state(pPipe, GST_STATE_PLAYING);

while (bCheck) {
pMsg = gst_bus_pop(pBus);
// Message Handling
}
gst_object_unref(pBus);
gst_element_set_state(pPipe, GST_STATE_NULL);

gst_pad_remove_probe(sinkpad_nvosd, nProbeID);
GST_OBJECT_RELEASE(sinkpad_nvosd);

Pipeline release is normal, but an error occurs on the second generation.
Am I not using it well?

Can you check if this repo can help you?
https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps/tree/master/runtime_source_add_delete