dwImageCUDA to dwImageNvMedia

How to convert dwImageCUDA to dwImageNvMedia formate of an image?

Dear yangquanhui,
You can use Imagestreamer to read data from one format to another. Please check the discusion on the thread https://devtalk.nvidia.com/default/topic/1044946/driveworks/dwimagenvmedia-dwimagestreamer-dwimagecuda/ to see if it helps

But API dwImageFormatConverter_copyConvertNvMedia is deprecated. How to convert dwImageCUDA to dwImageNvMedia?

Dear yangquanhui,
The format conversion APIs have changed. Please check Image Format Conversion section in DW documentation.
ImageStreamer facilitates transfer of Images from one format to another( For example CPU/CUDA/GL/NVMEDIA <-> CPU/CUDA/GL/NVMEDIA). Please check ImageStramer Section in DW documentation for more details. Please check DW samples to know usage of ImageStreamer(For ex. sample_camera_gsml).

When I convert CUDA to NVMEDIA, below error occurs!

[4-7-2019 16:5:0] Driveworks exception thrown: DW_NOT_AVAILABLE: The requested streamer type is not available

terminate called after throwing an instance of 'std::runtime_error'
  what():  [2019-07-04 16:05:00] DW Error DW_NOT_AVAILABLE executing DW function:
 dwImageStreamer_initialize(&m_streamerCUDAtoNV[portID * 4 + cameraSiblingID], &yuvImageProperties, DW_IMAGE_NVMEDIA, m_sdk)
 at /home/nvidia/smart_recorder/src/camera_recorder.cpp:195

My code is :

// initializes software ISP for processing RAW RCCB images
dwSoftISPParams softISPParams;
CHECK_DW_ERROR(dwSoftISP_initParamsFromCamera(&softISPParams, &m_cameraProperties[portID]));
CHECK_DW_ERROR(dwSoftISP_initialize(&m_isp[portID * 4 + cameraSiblingID], &softISPParams, m_sdk));

// getting the properties directly from the ISP
CHECK_DW_ERROR(dwSoftISP_getDemosaicImageProperties(&rcbProperties, m_isp[portID * 4 + cameraSiblingID]));
CHECK_DW_ERROR(dwImage_create(&rcbImage, rcbProperties, m_sdk));
CHECK_DW_ERROR(dwImage_getCUDA(&rcbCUDAImage, rcbImage));
// bind the image as the output for demosaic process to the ISP, will be filled at the call of
// dwSoftISP_processDeviceAsync
CHECK_DW_ERROR(dwSoftISP_bindOutputDemosaic(rcbCUDAImage, m_isp[portID * 4 + cameraSiblingID]));

dwImageProperties yuvImageProperties{};
yuvImageProperties.format = DW_IMAGE_FORMAT_YUV420_UINT8_PLANAR;
yuvImageProperties.type = DW_IMAGE_NVMEDIA;

yuvImageProperties.width = rcbProperties.width;
yuvImageProperties.height = rcbProperties.height;

CHECK_DW_ERROR(dwImageStreamer_initialize(&m_streamerCUDAtoNV[portID * 4 + cameraSiblingID], &yuvImageProperties, DW_IMAGE_NVMEDIA, m_sdk));

However, I find this ppt.

http://on-demand.gputechconf.com/gtc-cn/2018/pdf/CH8712.pdf

In page 17, NVMEDIA–>CUDA,but CUDA cannnot --> NVMEDIA, must I convert CUDA to NVMEDIA using CUDA–>CPU–>NVMEDIA?

Dear yangquanhui,
As per the DW 1.2 documentation, It is supported.

The second parameter of dwImageStreamer_initialize should be producer. It means it should be CUDA Image. But yuvImageProperties is declared as NVMediaImage.
The general use case would be NVMedia -> CUDA(As camera captures image in NVMedia and we need CUDA Image for processing). May I know why you are looking at CUDA -> NVMedia?

But DW 1.2.400 no Image list.

My process is camera–>resize–>Serializer to h264. But the video is a bit black in the underground garage. Now I want to optimize image quality using camera–>isp–>resize–>Serializer to h264.

first, I want to process image using dwSoftISP_bindOutputDemosaic. But after isp, I get a CUDA image.

then, I want to resize image using NvMedia2DBlitEx, but input image must be NVMEDIA image.

So, I want to convert CUDA to NVMEDIA !

PX2 DW 1.2.400 support it ?

Yes. Please make CUDA as producer

Dear SivaRamaKrishna,

In camera–>isp—>CUDAtoNVMEDIA(streamer), My code is below. But error exits like this.

ISP and Streamer init:

m_isp_output = DW_SOFTISP_PROCESS_TYPE_DEMOSAIC;

    // initializes software ISP for processing RAW RCCB images
    dwSoftISPParams softISPParams;
    CHECK_DW_ERROR(dwSoftISP_initParamsFromCamera(&softISPParams, &m_cameraProperties[portID]));
    CHECK_DW_ERROR(dwSoftISP_initialize(&m_isp[portID * 4 + cameraSiblingID], &softISPParams, m_sdk));

    if ((m_isp_output & DW_SOFTISP_PROCESS_TYPE_DEMOSAIC) &&
        m_args["interpolation_demosaic"].get<int>() > 0) {
        dwSoftISP_setDemosaicMethod(DW_SOFTISP_DEMOSAIC_METHOD_INTERPOLATION, m_isp[portID * 4 + cameraSiblingID]);
    }

    // we need to allocate memory for a demosaic image and bind it to the ISP
    dwImageHandle_t rcbImage = DW_NULL_HANDLE;
    dwImageCUDA* rcbCUDAImage;
    dwImageProperties rcbProperties{};
    if (m_isp_output & DW_SOFTISP_PROCESS_TYPE_DEMOSAIC) {
        // getting the properties directly from the ISP
        CHECK_DW_ERROR(dwSoftISP_getDemosaicImageProperties(&rcbProperties, m_isp[portID * 4 + cameraSiblingID]));
        CHECK_DW_ERROR(dwImage_create(&rcbImage, rcbProperties, m_sdk));
        CHECK_DW_ERROR(dwImage_getCUDA(&rcbCUDAImage, rcbImage));
        // bind the image as the output for demosaic process to the ISP, will be filled at the call of
        // dwSoftISP_processDeviceAsync
        CHECK_DW_ERROR(dwSoftISP_bindOutputDemosaic(rcbCUDAImage, m_isp[portID * 4 + cameraSiblingID]));

        LOG(INFO) << "rcbProperties format: " << rcbProperties.format;
        LOG(INFO) << "cpuProperties format: " << DW_IMAGE_FORMAT_YUV420_UINT8_PLANAR;

        dwImageProperties nvImageProperties{};
        nvImageProperties.format = DW_IMAGE_FORMAT_YUV420_UINT8_PLANAR;
        nvImageProperties.type = DW_IMAGE_CUDA;

        if (m_isp_output & DW_SOFTISP_PROCESS_TYPE_DEMOSAIC) {
            nvImageProperties.width = rcbProperties.width;
            nvImageProperties.height = rcbProperties.height;
            LOG(INFO) << "w: " << rcbProperties.width;
            LOG(INFO) << "h: " << rcbProperties.height;
        }
        CHECK_DW_ERROR(dwImageStreamer_initialize(&m_streamerCUDAtoNV[portID * 4 + cameraSiblingID], &nvImageProperties, DW_IMAGE_NVMEDIA, m_sdk));
    }

Read frame:

dwStatus status = DW_NOT_READY;

        do {
            status = dwSensorCamera_readFrame(&frame, cameraSiblingID, timeout, m_camera[portID]);
            LOG(INFO) << "read frame status: " << status << " " << DW_NOT_READY;
        } while (status == DW_NOT_READY || status == DW_TIME_OUT);

        if (status != DW_SUCCESS) {
            LOG(ERROR) << "dwSensorCamera_readFrame failed!";
            throw std::runtime_error("Camera error");
        }

get cuda image and isp process:

dwImageHandle_t frameCUDA;
        CHECK_DW_ERROR(dwSensorCamera_getImage(&frameCUDA, DW_CAMERA_OUTPUT_CUDA_YUV_UINT8_PLANAR, frame));
        // status = dwSensorCamera_getImage(&frameCUDA, DW_CAMERA_OUTPUT_CUDA_YUV_UINT8_PLANAR, frame);
        LOG(INFO) << "get CUDA image handle done!";

        // raw images need to be processed through the softISP
        dwImageCUDA* rawImageCUDA;
        CHECK_DW_ERROR(dwImage_getCUDA(&rawImageCUDA, frameCUDA));
        LOG(INFO) << "get CUDA image done!";

        CHECK_DW_ERROR(dwSoftISP_bindInputRaw(rawImageCUDA, m_isp[portID]));
        // request the softISP to perform a demosaic and a tonemap. This is for edmonstration purposes, the demosaic
        // output will not be used in this sample, only the tonemap output
        CHECK_DW_ERROR(dwSoftISP_setProcessType(m_isp_output, m_isp[portID]));
        CHECK_DW_ERROR(dwSoftISP_processDeviceAsync(m_isp[portID]));
        LOG(INFO) << "isp async done!";

streamer convert CUDA to NVMEDIA:

CHECK_DW_ERROR(dwImageStreamer_producerSend(rcbImage, m_streamerCUDAtoNV[portID * 4 + cameraSiblingID]));

dwImageHandle_t frameNV;
        CHECK_DW_ERROR(dwImageStreamer_consumerReceive(&frameNV, timeout, m_streamerCUDAtoNV[portID * 4 + cameraSiblingID]));

Error info of driverworks:

[1-7-2019 16:11:38] Driveworks exception thrown: DW_INVALID_ARGUMENT: ImageStreamer: input image properties don't match streamer's. Expected/Given: , Format=3000/2004, width=1920/1920, height=1208/1208

terminate called after throwing an instance of 'std::runtime_error'
  what():  [2019-07-01 16:11:38] DW Error DW_INVALID_ARGUMENT executing DW function:
 dwImageStreamer_producerSend(rcbImage, m_streamerCUDAtoNV[portID * 4 + cameraSiblingID])
 at /home/nvidia/smart_recorder/src/camera_recorder.cpp:293

ISP output image Format(DW_IMAGE_FORMAT_RCB_FLOAT16_PLANAR-2004) is different from streamer init Format(DW_IMAGE_FORMAT_YUV420_UINT8_PLANAR-3000), Why?

if I modify streamer init Format to DW_IMAGE_FORMAT_RCB_FLOAT16_PLANAR, such error also occurs:

[1-7-2019 16:42:53] Driveworks exception thrown: DW_FAILURE: Image: pixelFormatToNvMediaSurfaceType, Invalid image pixel format.

terminate called after throwing an instance of 'std::runtime_error'
  what():  [2019-07-01 16:42:53] DW Error DW_FAILURE executing DW function:
 dwImageStreamer_initialize(&m_streamerCUDAtoNV[portID * 4 + cameraSiblingID], &nvImageProperties, DW_IMAGE_NVMEDIA, m_sdk)
 at /home/nvidia/smart_recorder/src/camera_recorder.cpp:208

How to eliminate these errors?

Thanks

Hi yangquanhui,

Sorry for late response.
Is this still a blocking issue on your project or you have resolved the problem?

Thanks

Hi yangquanhui,

The error says that the format of the image passed to the streamer is different from what the streamer expects. In fact

nvImageProperties.format = DW_IMAGE_FORMAT_YUV420_UINT8_PLANAR;
CHECK_DW_ERROR(dwImageStreamer_initialize(&m_streamerCUDAtoNV[portID * 4 + cameraSiblingID], &nvImageProperties, DW_IMAGE_NVMEDIA, m_sdk));

this is the format used to create the streamer

while this

CHECK_DW_ERROR(dwSoftISP_getDemosaicImageProperties(&rcbProperties, m_isp[portID * 4 + cameraSiblingID]));
CHECK_DW_ERROR(dwImage_create(&rcbImage, rcbProperties, m_sdk));

has the format of the image being streamed (2004 corresponds to DW_IMAGE_FORMAT_RCB_FLOAT16_PLANAR)

I had solved my question.

1.init

dwImageProperties tonemapProperties{};
    tonemapProperties.width = demosaicProperties.width;
    tonemapProperties.height = demosaicProperties.height;
    tonemapProperties.type = demosaicProperties.type;
    tonemapProperties.format = DW_IMAGE_FORMAT_RGBA_UINT8;
    tonemapProperties.memoryLayout = DW_IMAGE_MEMORY_TYPE_PITCH;
    dwImageCUDA* rgbaCUDAImage = nullptr;
    CHECK_DW_ERROR(dwImage_create(&m_rgba_image_handler[portID * 4 + cameraSiblingID], tonemapProperties, m_sdk));
    CHECK_DW_ERROR(dwImage_getCUDA(&rgbaCUDAImage, m_rgba_image_handler[portID * 4 + cameraSiblingID]));
    // bind the image as the output for demosaic process to the ISP, will be filled at the call of
    // dwSoftISP_processDeviceAsync
    CHECK_DW_ERROR(dwSoftISP_bindOutputTonemap(rgbaCUDAImage, m_isp[portID * 4 + cameraSiblingID]));
    CHECK_DW_ERROR(dwImageStreamer_initialize(&m_streamerCUDAtoNV[portID * 4 + cameraSiblingID], &tonemapProperties, DW_IMAGE_NVMEDIA, m_sdk));

2.get image to soft-isp process

dwImageHandle_t raw_img_handle;
    CHECK_DW_ERROR(dwSensorCamera_getImage(&raw_img_handle, DW_CAMERA_OUTPUT_CUDA_RAW_UINT16, frame));
    //CHECK_DW_ERROR(dwSensorCamera_getImage(&raw_img_handle, DW_CAMERA_OUTPUT_CUDA_YUV_UINT8_PLANAR, frame));-----------here must be cuda raw image
    LOG(INFO) << "get RAW CUDA image handle done!";
    // raw images need to be processed through the softISP
    dwImageCUDA* rawImageCUDA = nullptr;
    CHECK_DW_ERROR(dwImage_getCUDA(&rawImageCUDA, raw_img_handle));
    LOG(INFO) << "get CUDA image done!";
    
    CHECK_DW_ERROR(dwSoftISP_bindInputRaw(rawImageCUDA, m_isp[portID * 4 + cameraSiblingID]));
    // request the softISP to perform a demosaic and a tonemap. This is for edmonstration purposes, the demosaic
    // output will not be used in this sample, only the tonemap output
    CHECK_DW_ERROR(dwSoftISP_setProcessType(m_isp_output, m_isp[portID * 4 + cameraSiblingID]));
    CHECK_DW_ERROR(dwSoftISP_processDeviceAsync(m_isp[portID * 4 + cameraSiblingID]));

3.cuda to nvmedia

// cuda to nvmedia
        CHECK_DW_ERROR(dwImageStreamer_producerSend(m_rgba_image_handler[portID * 4 + cameraSiblingID],
                                                    m_streamerCUDAtoNV[portID * 4 + cameraSiblingID]));
        CHECK_DW_ERROR(dwImageStreamer_consumerReceive(&nv_frame_handle, timeout, m_streamerCUDAtoNV[portID * 4 + cameraSiblingID]));

CHECK_DW_ERROR(dwImageStreamer_consumerReturn(&nv_frame_handle, m_streamerCUDAtoNV[portID * 4 + cameraSiblingID]));
        CHECK_DW_ERROR(dwImageStreamer_producerReturn(nullptr, timeout, m_streamerCUDAtoNV[portID * 4 + cameraSiblingID]));

It’s good to know you resolved the problem, thanks for the feedback!

I met the same converting CUDA to NVMedia image problem and get the message:

NvMediaEglValidateLocalTVMROutputSurface: Pitch must be 3840
NvMediaEglFillLocalVideoSurface: NvMediaEglValidateLocalTVMROutputSurface failed
NvMediaEglValidateLocalTVMROutputSurface: Pitch must be 3840
NvMediaEglFillLocalImageSurface: NvMediaEglValidateLocalTVMROutputSurface failed
[14-8-2019 10:41:8] ImageStreamer: cannot receive, EGLstream busy
[14-8-2019 10:41:8] Driveworks exception thrown: DW_INVALID_ARGUMENT: Cannot cast to C handle, given instance is a nullptr

terminate called after throwing an instance of ‘std::runtime_error’
what(): [2019-08-14 10:41:08] DW Error DW_INVALID_ARGUMENT executing DW function:
dwImageStreamer_consumerReceive(&m_pendingReturn, m_timeout, m_streamer)
at /home/jin/driveworks_sample/src/framework/…/framework/SimpleStreamer.hpp:105
Aborted (core dumped)

And solved after setting CUDA image’s pitch=3840.