Parse NV12_ER Format Deepstream 5.0

• Hardware Platform Jetson Tx2
• DeepStream Version: 5.0
• JetPack Version 4.5.2
• TensorRT Version: 7.1.3

Hello, I am trying to parse a deepstream buffer where the surface is in NV12_ER color space and I am seeing the image attached from rawpixel.net when I use the code and pipeline below. The output says the color-space is NV12_ER.

Something interesting though, when I use the code and pipeline below with a convert to NVMM RGBA before the infer and an NVMM NV12 conversion after the infer, the buffer is in NV12_709 space and I am able to parse it correctly with the code below. Struggling to see how NV12_ER is laid out in memory

My gst pipeline is:
rtspsrc name=rtsp1 location=rtsp://127.0.0.1:8554/test ! nvv4l2decoder ! video/x-raw(memory:NVMM), format=NV12 ! m.sink_0
rtspsrc name=rtsp2 location=rtsp://127.0.0.1:8554/test ! nvv4l2decoder ! video/x-raw(memory:NVMM), format=NV12 ! m.sink_1
rtspsrc name=rtsp3 location=rtsp://127.0.0.1:8554/test ! nvv4l2decoder ! video/x-raw(memory:NVMM), format=NV12 ! m.sink_2
rtspsrc name=rtsp4 location=rtsp://127.0.0.1:8554/test ! nvv4l2decoder ! video/x-raw(memory:NVMM), format=NV12 ! m.sink_3
nvstreammux name=m batch-size=4 width=1920 height=1080 ! nvinfer config-file-path=config_infer_primary_yoloV5.txt batch-size=4 interval=6 unique-id=1 ! mycustom batch-size=4 ! fakesink

and the code in the custom plugin to parse and save the image is(used from this post: DeepStream SDK FAQ - #17 by mchi
):

printf(" batchSize = %d\n", surface->batchSize);
printf(" numFilled = %d\n", surface->numFilled);
printf(" memType = %d\n", surface->memType);
for (int bs = 0; bs < surface->batchSize; bs++) {
NvBufSurfaceParams *sur = &(surface->surfaceList[bs]);
if (sur == NULL)
break;
printf(" surfaceList[%d]:\n", bs);
printf(" width = %d\n", sur->width);
printf(" height = %d\n", sur->height);
printf(" pitch = %d\n", sur->pitch);
printf(" layout = %d\n", sur->layout);
printf(" colorFormat = %d\n", sur->colorFormat);
printf(" bufferDesc = %lu\n", sur->bufferDesc);
printf(" dataSize = %d\n", sur->dataSize);
printf(" dataPtr = %p\n", sur->dataPtr);
// NVIDIA DeepStream SDK API Reference: Main PageCPPv423NvBufSurfaceColorFormat
if (sur->colorFormat == NVBUF_COLOR_FORMAT_NV12)
printf(“it’s NVBUF_COLOR_FORMAT_NV12!\n”);
else
printf(“it’s not NVBUF_COLOR_FORMAT_NV12(%d)\n”, NVBUF_COLOR_FORMAT_NV12);
NvBufSurfacePlaneParams *planeParams = &(sur->planeParams);
printf(" NvBufSurfaceParams->planeParams:\n");
printf(" num_planes = %d\n", planeParams->num_planes);
for (p = 0; p < planeParams->num_planes; p++) {
printf(" width[%d] = %u\n", p, planeParams->width[p]);
printf(" height[%d] = %u\n", p, planeParams->height[p]);
printf(" pitch[%d] = %u\n", p, planeParams->pitch[p]);
printf(" offset[%d] = %u\n", p, planeParams->offset[p]);
printf(" psize[%d] = %u\n", p, planeParams->psize[p]);
printf(" bytesPerPix[%d] = %u\n", p, planeParams->bytesPerPix[p]);
}
NvBufSurfaceMappedAddr mAddr = &(sur->mappedAddr);
printf(" NvBufSurfaceParams->mappedAddr:\n");
for (p = 0; p < NVBUF_MAX_PLANES; p++) {
if (!mAddr->addr) {
printf(" mappedAddr->addr[%d] is NULL\n", p);
break;
}
printf(" mappedAddr->addr[%d] = %p\n", p, mAddr->addr[p]);
}
if (mAddr->eglImage == NULL)
printf(" mappedAddr->eglImage is NULL\n");
else
printf(" mappedAddr->eglImage is %p\n", mAddr->eglImage);
// write_video_frame
{
/
Map the buffer so that it can be accessed by CPU */
// int NvBufSurfaceMap(NvBufSurface surf, int index, int plane, NvBufSurfaceMemMapFlagstype)
for (p = 0; p < planeParams->num_planes; p++) {
if (NvBufSurfaceMap (surface, bs, p, NVBUF_MAP_READ) != 0)
printf(“NvBufSurfaceMap failed ===============\n”);
// sync to CPU for the two planes
if(surface->memType == NVBUF_MEM_SURFACE_ARRAY) {
/
Cache the mapped data for CPU access */
NvBufSurfaceSyncForCpu (surface, bs, p);
}
}
NvBufSurfaceMappedAddr *mAddr = &(sur->mappedAddr);
for (p = 0; p < NVBUF_MAX_PLANES; p++) {
if (!mAddr->addr) {
printf(" mappedAddr->addr[%d] is NULL\n", p);
break;
}
printf(" mappedAddr->addr[%d] = %p\n", p, mAddr->addr[p]);
}
char filename[64];
sprintf(filename, "output
%dx%d_bs-%d.nv12",
sur->width, sur->height, bs);
std::ofstream outputfile(filename);
for (p = 0; p < planeParams->num_planes; p++)
{
size_t bytes_to_write =
planeParams->bytesPerPix[p] * planeParams->width[p];
char *data = (char *) mAddr->addr[p];
for (int j = 0; j < planeParams->height[p]; j++)
{
outputfile.write(data, bytes_to_write);
if (!outputfile.good())
printf(“BAD OUTPUT”);
data += planeParams->pitch[p];
}
}
for (p = 0; p < planeParams->num_planes; p++)
NvBufSurfaceUnMap (surface, bs, p);
}
}

Can you dump the video from your rtsp stream and provide the video to us for analysis?

I think any video will work. if I put a tiling element and display or RTSP the tiled stream out, the output looks fine. It’s just the saved off raw image buffers that look like that.