Speed up stereo camera rectification and cropping (custom gstreamer plugin/ VPI)

Hi there.

I’m currently working on a custom gstreamer plugin in which I rectify a stereo image pair. I’m working on a Jetson Xavier AGX.

For the rectification process I use the remap algorithm from VPI, which works fine so far. After I’ve done the rectification I wan’t to crop the rectified images so that for further processing only the ROI is used.
I’ve accomplished this by using NvBufSurfTransform() but by doing so my processing time increases drastically.

Has anyone done anything similar or has anyone an idea how to accomplish rectification and cropping in a fast way.
Note: For now I’ve avoided using opencv.

Is it possible to do the cropping directly by using remap from vpi?

Moving it into Jetson AGX Xavier forum.

Please execute the steps and try again:
Nvvideoconvert issue, nvvideoconvert in DS4 is better than Ds5? - #3 by DaneLLL

These steps enable hardware converter at max clock and NvBufferTransform() runs in max throughput. Please give it a try. Besides, if you have simultaneous NvBufferTransform() calls, please create NvBufferSession for each task.

This really speeds up (fact. 3).

How do I use NvBufferSession?
I read in the API documentation that _NvBufferTransformParams should have a session (NvBufferSession) datafield but it seems like it doesn’t exist.

There is sample code in 07 sample. And here are topics for reference:
Possible multimedia api regression with decode interlace source - #28 by DaneLLL
A segfault occurs when creating NvVideoDecoder inside a child process - #9 by DaneLLL

There was a confusion because I used NvBufSurfTransform instead of NvBufferTransform. Now I’m able to access the data fields as expected,
but now I ran into another problem by using ExtractFdFromNvBuffer(). I read alot in the forum about this topic but nothing seems to help in my case.
I’m trying to copy the buffer within a probe callback function in which I try to use ExtractFdFromNvBuffer() but the dmabuf_fd is 0.
The probe callback is called right after a format conversion from NV12 to RGBA, which I do by using nvvideoconvert and a capsfilter.
When I change the nvvideoconvert plugin to nvvidconv then I get a dmabuf_fd which makes sense but unfortunately I’m using DS pluigins for which I read nvvideoconvert must be used.

Here is a sample from my callback function:

GstPadProbeReturn PipeThd::probe_cb_osdmeta(GstPad *pad, GstPadProbeInfo *info, gpointer user_data) {

	PipeThd *p_this = static_cast<PipeThd*>(user_data);
	GstBuffer *buf = (GstBuffer *) info->data;
	NvDsObjectMeta *obj_meta = NULL;
	NvDsMetaList *l_frame = NULL;
	NvDsMetaList *l_obj = NULL;
	NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta (buf);
	GstMapInfo in_map_info;
	memset(&in_map_info, 0, sizeof(in_map_info));
	if (!gst_buffer_map(buf, &in_map_info, GST_MAP_READ))
		LOG(DEBUG) << "Failed to map gst buffer";
		return GST_PAD_PROBE_OK;
        int dmabuf_fd;
	if(ExtractFdFromNvBuffer((void *)in_map_info.data, &dmabuf_fd))
	 	LOG(DEBUG) << "Failed to extract fd from buffer";
	printf("dmabuf_fd - %d\n", dmabuf_fd);

       //...do somethig with fd (copying)

     NvReleaseFd (dmabuf_fd);

Do i misunderstood something or is it the wrong approach to extract the dmabuf_fd?

Sorry I didn’t notice you use NvBufSurface APIs. The two Interfaces are not compatible. For DeepStream use-case, please use NvBufSurface APIs and run the hardware converter at maximum clock.

Thank you for your response.
Now I have want to know what processing time I’m ought to expect.
Copying a batch with 2 frames (2448 x 2058) varies from 4ms up to 10ms.
Is this about right or should I expect a faster processing time?

And I also would like to know if there is a way to crop the image directly within remap API?

Is remap the right way to rectify a stereo image pair?

I was able to crop the image directly within remap API by using opencv to generate the map.

cv::stereoRectify(_M1, _D1, _M2, _D2, image_size, _R, _T, _R1, _R2, _P1, _P2, _Q,
					  cv::CALIB_ZERO_DISPARITY, 0, image_size, &_validRoi[0], &_validRoi[1]);

alpha=0 means that the rectified
images are zoomed and shifted so that only valid pixels are visible (no black areas after rectification) - opencv2/calib3d.hpp

But I’m still interested to know if my processing time is right.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.