NvMediaSurface does not suppor RGBA planar?

Autonomous Vehicles DRIVE AGX Xavier DRIVE AGX Xavier General
1

Please provide the following info:
Hardware Platform: [DRIVE AGX Xavier™ Developer Kit]
Software Version: [Example: DRIVE Software 10]
Host Machine Version: [native Ubuntu 18.04]

I want to create a NvMediaImage with RGBA, Planar, PL format so its memory buffer would be something like: RRR…RRGGG…GGBBB…BB, which also corresponds to NCHW layout in TensorRT. However, the default memory has been hard-coded to NVM_SURF_ATTR_MEMORY_PACKED (line 299 in nvmedia_surface.h).
If I manually set the memory type to NVM_SURF_ATTR_MEMORY_PLANAR instead of using NVM_SURF_FMT_SET_ATTR_RGBA(), then I would get this error in runtime:

[NvMImageParseSurfAllocAttr:367] Surface allocation unsupported for the surface type 99999
[NvMediaImageCreateNew:844] NvMImageParseSurfAllocAttr failed

I wonder is RGBA planar not supported in DRIVE OS or I did anything wrong? If it’s not supported in current version, is there any plan to support it in the future?
The reason I want a RGBA planar surface is NCHW is a very common memory layout in deep learning, and I want to feed the surface buffer of NvMediaImage into TensorRT for inference. If there’s any other way to get around this, please also suggest.

Thank you!

3

Dear @James22,
Could you share some code snippet/sample code which us failing.

I want to feed the surface buffer of NvMediaImage into TensorRT for inference. If there’s any other way to get around this, please also suggest.

You need to convert your NVMedia Image to CUDA buffer to feed into TensorRT engine. EGLStreams need to be used to facilitate the process. But, DW has ImageStreamer which simplifies the development. DW APIs to read the camera data and feed the same TensorRT engine for inference. Did you check object detector tracker and camera capture samples?

4

Hi @SivaRamaKrishnaNV,
First, a NvMediaImage is created using the code below

// set NvMediaSurfAllocAttr…
NVM_SURF_FMT_DEFINE_ATTR(surfFormatAttrs);
NVM_SURF_FMT_SET_ATTR_RGBA(surfFormatAttrs, RGBA, UINT, 8, PL);
NvMediaSurfaceType surfaceType = NvMediaSurfaceFormatGetType(surfFormatAttrs, NVM_SURF_FMT_ATTR_MAX);
NvMediaImage *img = NvMediaImageCreateNew(device, surfaceType, surfAllocAttrs.data(), surfAllocAttrs.size(), 0);

After this, the img can be used normally, and I can use JPG decoder to fill it with an image. However, its memory layout will be interleaved instead of planar. Also, the data cannot be used as NHWC layout because its surface type is RGBA instead of RGB, which makes channel number = 4 rather than 3 that most neural networks are trained with.

If I try to manually set the memory layout to planar, replace NVM_SURF_FMT_SET_ATTR_RGBA() with its definition (line 287 of nvmedia_surface.h) and set NVM_SURF_ATTR_MEMORY to be planar, then the code becomes:

// set NvMediaSurfAllocAttr…
NVM_SURF_FMT_DEFINE_ATTR(surfFormatAttrs);
surfFormatAttrs[0].type = NVM_SURF_ATTR_SURF_TYPE;
surfFormatAttrs[0].value = NVM_SURF_ATTR_SURF_TYPE_RGBA;
surfFormatAttrs[1].type = NVM_SURF_ATTR_LAYOUT;
surfFormatAttrs[1].value = NVM_SURF_ATTR_LAYOUT_PL;
surfFormatAttrs[2].type = NVM_SURF_ATTR_DATA_TYPE;
surfFormatAttrs[2].value = NVM_SURF_ATTR_DATA_TYPE_UINT;
surfFormatAttrs[3].type = NVM_SURF_ATTR_MEMORY;
surfFormatAttrs[3].value = NVM_SURF_ATTR_MEMORY_PLANAR; // Changed to planar
surfFormatAttrs[4].type = NVM_SURF_ATTR_SUB_SAMPLING_TYPE;
surfFormatAttrs[4].value = 0;
surfFormatAttrs[5].type = NVM_SURF_ATTR_BITS_PER_COMPONENT;
surfFormatAttrs[5].value = NVM_SURF_ATTR_BITS_PER_COMPONENT_8;
surfFormatAttrs[6].type = NVM_SURF_ATTR_COMPONENT_ORDER;
surfFormatAttrs[6].value = NVM_SURF_ATTR_COMPONENT_ORDER_RGBA;
NvMediaSurfaceType surfaceType = NvMediaSurfaceFormatGetType(surfFormatAttrs, NVM_SURF_FMT_ATTR_MAX);
NvMediaImage *img = NvMediaImageCreateNew(device, surfaceType, surfAllocAttrs.data(), surfAllocAttrs.size(), 0);

The I would get the runtime error

[NvMImageParseSurfAllocAttr:367] Surface allocation unsupported for the surface type 99999
[NvMediaImageCreateNew:844] NvMImageParseSurfAllocAttr failed

I would like to avoid using DW APIs as we were told the safety certification of DW comes much later than DRIVE OS. What about NvMedia APIs like NvMediaTensor? I couldn’d find much documentations about that. Would NvMediaTensor be Nvidia’s envisioned interface for bridging NvMediaImage and TensorRT?

5

Dear @James22,
I checked with our engineering team. only NVM_SURF_ATTR_MEMORY_PACKED flag is supported.