How can I get RGBA during raw output

[N.Miya]

Hardware Platform: [DRIVE AGX Xavier™ Developer Kit]
Software Version: [DRIVE Software 10]
Host Machine Version: [Ubuntu 18.04]
SDK Manager Version: [Example: 1.0.1.5538]

Hi,
I want to get DW_CAMERA_OUTPUT_CUDA_RGBA_UINT8 and DW_CAMERA_OUTPUT_NATIVE_RAW from Camera.
I changed the code, set the output-format to raw and got DW_CAMERA_OUTPUT_CUDA_RGBA_UINT8 and got the error.

///code
CHECK_DW_ERROR(dwSensorCamera_getImageAsync(&frameNvMedia, DW_CAMERA_OUTPUT_CUDA_RGBA_UINT8, cameraFrame));

[21-09-2019 15:20:08] Driveworks exception thrown: DW_NOT_AVAILABLE: Camera: getImage, selected output type is not available with the current settings/platform/configuration

terminate called after throwing an instance of ‘std::runtime_error’
what(): [2019-09-21 15:20:08] DW Error DW_NOT_AVAILABLE executing DW function:
dwSensorCamera_getImageAsync(&frameNvMedia, DW_CAMERA_OUTPUT_CUDA_RGBA_UINT8, cameraFrame)

I think output mode raw supports RGBA , is it wrong?
How to get RGBA?

[SivaRamaKrishnaNV]

Dear @N.Miya,
Are you modifying any DW sample? You want to get RGBA buffers of input image for furthur processing? It would be clear, If you can share a simple code snippet to understand issue.

[jpvans]

Hi

Do you know if there any guide how to merge DriveNet and PathPrediction sample apps all together. I followed the example for PX2 platform but it seems it has some difference (mostly on the common folder)

[N.Miya]

Dear @SivaRamaKrishnaNV,

Thank you for your help.

Are you modifying any DW sample?

Yes, I modified camera_multiple_gmsl sample.

@void initializeCameras()

// create and initialize all cameras
    for (portID = 0; portID < MAX_PORTS_COUNT; ++portID) {
        if (m_activeCamerasPerPort[portID] > 0) {
            std::string parameterString;
            std::string cameraName = getArgument((std::string("type-") + groupIDNames[portID]).c_str());

            parameterString += std::string("output-format=raw+data,");
            parameterString += std::string("camera-group=") + groupIDNames[portID];
            parameterString += ",camera-type=" + cameraName;
            parameterString += ",camera-count=4"; // when using the mask, just ask for all cameras, mask will select properly

            if (selectorMask.size() >= portID*4) {
                parameterString += ",camera-mask="+ selectorMask.substr(portID*4, std::min(selectorMask.size() - portID*4, size_t{4}));
            }

@void onRender() override

            do {
                status = dwSensorCamera_readFrame(&cameraFrame, cameraSiblingID, timeout, m_camera[csiPort]);
            } while(status == DW_NOT_READY || status == DW_TIME_OUT);

            if (status != DW_SUCCESS) {
                throw std::runtime_error("Camera error");
            }

            CHECK_DW_ERROR(dwRenderEngine_setTile(m_tileVideo[cameraSiblingID + tileIndex], m_renderEngine));
            CHECK_DW_ERROR(dwRenderEngine_resetTile(m_renderEngine));
            // get an image with the desired output format

            dwImageHandle_t frameNvMedia;
            // see sample_camera_gmsl for how to directly grab a CUDA processed image from Camera. Because of a limitation
            // on the PX2 DGPU, streaming CUDA->GL for rendering has performance issues, so for this sample
            // we grab native processed (with properties of NvMedia YUV420 planar), convert it to RGBA and
            // stream it to GL.
            CHECK_DW_ERROR(dwSensorCamera_getImageAsync(&frameNvMedia, DW_CAMERA_OUTPUT_CUDA_RGBA_UINT8, cameraFrame));

            // convert native (yuv420 planar nvmedia) to rgba nvmedia
            // CHECK_DW_ERROR(dwImage_copyConvert(m_rgbaFrame[csiPort], frameNvMedia, m_sdk));

            // stream that image to the GL domain
            CHECK_DW_ERROR(dwImageStreamerGL_producerSend(frameNvMedia, m_streamerCUDAtoGL[csiPort]));

Thanks.

[shayNV]

Hi @N.Miya,
from looking at your snippet code, the changes you have made are:

1. adding the parameter output-format=raw+data to the camera sensor creation
   → this is not the right way. if you are want to use RAW+DATA output instead of yuv (using the HW ISP) you should define the software ISP and use it. for output format raw please refer to the sample code of sample_camera_gmsl_raw.
2. changing dwSensorCamera_getImage to dwSensorCamera_getImageAsyncis not enough. when using async APIs it means the processing is done over a cuda stream, therefore it has to be defined and attached to the sensor and to the image streamer in order for them to be aligned.

please consider the attached corrected sample code using async command.main.cpp (22.7 KB)

[N.Miya]

Hi, @shayNV,
Thank you for checking my snippet code.

1.adding the parameter output-format=raw+data to the camera sensor creation
→ this is not the right way. if you are want to use RAW+DATA output instead of yuv (using the HW ISP) you should define the software ISP and use it. for output format raw please refer to the sample code of sample_camera_gmsl_raw .

I want to get RAW and RGBA by using HW ISP. Not use software ISP.
Can I get RAW and RGBA(using HW ISP)?
What does Output DW_IMAGE_FORMAT_RGBA_UINT8 mean ?

[SivaRamaKrishnaNV]

Dear @N.Miya,
What does Output DW_IMAGE_FORMAT_RGBA_UINT8 mean ?

It indicates COLORSPACE is RGBA and PIXELTYPE is UINT8. Please check Modules-> Drive core → Image processing → Image for more details about Image module.

The error Driveworks exception thrown: DW_NOT_AVAILABLE: Camera: getImage, selected output type is not available with the current settings/platform/configuration possibly because of using default isp-mode parameter in the sample. can you remove it and check with output-format=raw again? Let me know the progress

[SivaRamaKrishnaNV]

Dear @N.Miya,
Could you provide any update if the suggested solution helped to progress?

[N.Miya]

Dear @SivaRamaKrishnaNV

Sorry, I can’t found settings/platform/configuration.
Could you please tell me where the file is?

I found a similar thread.
Is this content still correct?
I’ll try using IPP.

Conversion and difference between DW_CAMERA_RAW_IMAGE and DW_CAMERA_PROCESSED_IMAGE

Hi, I’m writing a code where I have to get dwImageNvMedia from camera in raw (DW_CAMERA_RAW_IMAGE) format, and convert it to dwImageNvMedia RGBA. I found an example in driveworks where they are using DW_CAMERA_PROCESSED_IMAGE from camera and converting that dwImageNvMedia directly to RGBA. I tried to change the code to get dwImageNvMedia camera image in DW_CAMERA_RAW_IMAGE format, but if I try to convert that dwImageNvMedia to RGBA, following the example from camera_multiple_gmsl, I get NvMedi…

[SivaRamaKrishnaNV]

Dear @N.Miya,

Sorry, I can’t found settings/platform/configuration.
Could you please tell me where the file is?

You don’t need to update these files. As you are modifying the camera_multiple_gmsl sample code, I have asked to remove isp-mode parameter and check using with output-format=raw. You can see default parameters in main method in the sample. Please let us know if it helped.