**• Hardware Platform Jetson ** - Jetson Xavier AGX and NX
• DeepStream Version - Deepstream 6.2
• JetPack Version (valid for Jetson only) - L4T R35.3.1
• TensorRT Version N/A
• Issue Type - questions
So I’m having my own gstreamer plugin to process frames from multiple sensors, batch it into one package and push it downwards the gstreamer pipeline to nvinfer. This plugin is based on gstnvarguscamerasrc plugin. The algorithm used in my plugin is similar to the one in gstnvarguscamerasrc, except for the batching part:
- 4 “Sensor’s threads” to retrieve the image and copy it (copy #1) into dmabuf:
// Get the IImageNativeBuffer extension interface and create the fd.
NV::IImageNativeBuffer *iNativeBuffer =
interface_cast<NV::IImageNativeBuffer>(iFrame->getImage());
if (!iNativeBuffer)
ORIGINATE_ERROR("IImageNativeBuffer not supported by Image.");
if (src->frameInfo->fd < 0)
{
// Create dmabuf descriptor for one-batched surface
src->frameInfo->fd = iNativeBuffer->createNvBuffer(streamSize,
NVBUF_COLOR_FORMAT_YUV420,
NVBUF_LAYOUT_BLOCK_LINEAR);
if (!src->silent)
CONSUMER_PRINT("Acquired Frame. %d\n", src->frameInfo->fd);
}
// Copy #1
else if (iNativeBuffer->copyToNvBuffer(src->frameInfo->fd) != STATUS_OK)
{
ORIGINATE_ERROR("IImageNativeBuffer not supported by Image.");
}
Main thread to process frame and push it to the pipeline. Copy dmabuf retrieved from every sensor into 4-batched surface:
// Create 4 batched surface and fill it in with framebufs from 4 sensors (copy #2):
ret = gst_buffer_pool_acquire_buffer (src->pool, &buffer, NULL); // batch is 4 frames here
.....
// dmabuf descriptors are for 4-batched surface
NvBufSurface* surf = (NvBufSurface *)outmap.data;
NvDsBatchMeta* batch_meta = createBatchMetaData(src);
NvBufferTransformParams transform_params = {};
for (int i = 0; i < src->sensors.size(); i++) {
// Copy #2
int retn = NvBufferTransform(consumerFrameInfo[i]->fd, (gint) surf->surfaceList[i].bufferDesc, &transform_params);
.... // Add some metadata to the batch
}
.... // Release used buffer to allow sensor thread grab new image
}
surf->numFilled = src->sensors.size();
surf->batchSize = src->sensors.size();
So this algorithm works fine. I’m getting what I need. But I have double copy in my algorithm and I started to notice that on JP5 based devices my FPS performance is significantly worse than on JP4 based devices (L4T R32.4.4, Deepstream 5.0).
Some quick profiling showed that combined frame copying(including both copy) is taking almost all frame time (target is 30FPS). That is, JP5 devices are processing frames on average for 35ms, while JP4 devices still remain within 33ms under same setup.
So my main idea to reduce resource consumption would be to avoid double copy while processing the frame. With my new approach I’m doing the following:
In main thread:
- Acquire gstbuffer from the pool (this would be 4 batched NvBufSurface)
- Pass the dmabuf fd from this surface(BuffDesc) to sensor thread (1x1 mapping between surface batch list and sensor’s threads).
- Don’t copy the dmabuf anymore, expect the image will be copied directly into surf->surfaceList[sensor_id].bufferDesc (for every batch)
In sensor thread (we have 4 such threads) we do now the following:
src->frameInfo->fd = dmabuf[sensor_idx]; // dmabuf[sensor_idx] is basically an array of descriptors, collected from **surf->surfaceList** array (bufferDesc field)
iNativeBuffer->copyToNvBuffer(src->frameInfo->fd); // Note that we don't create dmabuf anymore. Use preallocated fd from gst_buffer.
So the idea would be that we copy the image straight into the dmabuf from NvBufSurface that is mapped directly to our gst buffer. And this algorithm works fine for JP4 devices. But not for JP5. What I’m getting on the pipeline output for JP5 device is 3 empty buffers (green screen) for last 3 batches. The screen for first batch is showing me interchangeably frames from all 4 sensors. So it feels like iNativeBuffer->copyToNvBuffer(src->frameInfo->fd); is copying all frames straight into batch #1 (and just whatever thread was the last to execute will get its buffer stored and displayed).
Our further debugging showed that iNativeBuffer->copyToNvBuffer(src->frameInfo->fd) internally is using NvBufSurfaceFromFd. And NvBufSurfaceFromFd will always create same NvBufSurface for all 4 descriptors provided from allocated gst buffer. And for some reason the copying functionality ignores provided dmabuf fd, it is just copying image to first surfaceList from that surface. Please note that we also see that on JP4 copyToNvBuffer doesn’t use NvBufSurfaceFromFd and thus I’m getting exactly what I need, all 4 images are shown correctly.
Looking at examples from nvidia-l4t-jetson-multimedia-api_35.1.0-20220825113828_arm64.deb, I noticed that surf->surfaceList is always referenced either as pointer to surface or as surfaceList[0]. Meaning in all facilities in this package only first batch is being taken into account, any other batches in the surfaceList are ignored.
So with this story I have few questions:
- By creating a batch of frames in our custom cameraargussrc plugin from multiple sensors, are we taking the right approach? Maybe there is other ways to create batched frame(surface) for nvinfer processing
- If this is the right way to create batched frame, how can I avoid double copying in my plugin? Given that I retrieve the image from EGL stream and package it into 4-batched surface. I looked at different APIs, for NativeBuffer, Image inerface, NvBuffer, NvBufSurface. But none of those seem to help me avoid double copy.
- Can you confirm I’m using NvBufSurface correctly in this scenario? Also is it expected that providing 4-batched surface to iNativeBuffer->copyToNvBuffer will always write to 0 frame and never to other batches, no matter what dmabuf descriptor I provide?
Any help or suggestions would be very much appriciated.
