Creating Tonemap CUDA Image from a CPU Image using the SoftISP Module

Autonomous Vehicles DRIVE PX 2 DriveWorks
1

Hello,

My colleagues and I are attempting to implement a version of the image processing workflow demonstrated in the DriveWorks 1.2 sample_camera_replay program. In the program shown below, we populate a dwImageCPU object with arbitrary raw image data, stream the image to a dwImageCUDA object, and then pass the image to the SoftISP module to create a tonemap image output. This tonemap image is then streamed to a dwImageGL object and rendered to a GLFW window. However, the tonemap image renders to the window as an empty black frame, so it appears that the image data is being lost/dropped somewhere in the program.

Are there any debugging tools available, or any ways to directly access the image data in dwImageCUDA objects, so that we can find where our image data is being lost?

//driveworks includes
include “/usr/local/driveworks-1.2/include/dw/Driveworks.h”
include “/usr/local/driveworks-1.2/include/dw/core/Context.h”
include “/usr/local/driveworks-1.2/include/dw/isp/SoftISP.h”
include “/usr/local/driveworks-1.2/include/dw/image/Image.h”
include “/usr/local/driveworks-1.2/include/dw/sensors/Sensors.h”
include “/usr/local/driveworks-1.2/include/dw/sensors/camera/Camera.h”
include “/usr/local/driveworks-1.2/include/dw/core/Version.h”
include “/usr/local/driveworks-1.2/include/dw/core/VersionCurrent.h”
include “/usr/local/driveworks-1.2/include/dw/renderer/RenderEngine.h”
include “/usr/local/driveworks-1.2/include/dw/renderer/Renderer.h”
include “/usr/local/driveworks-1.2/include/dw/image/ImageStreamer.h”
include “/usr/local/driveworks-1.2/include/dw/gl/GL.h”

//other driveworks includes
include “/usr/local/driveworks-1.2/samples/src/image/image_common/utils.hpp”
include “/usr/local/driveworks-1.2/samples/src/framework/DriveWorksSample.hpp”
include “/usr/local/driveworks-1.2/samples/src/framework/Log.hpp”
include “/usr/local/driveworks-1.2/samples/src/framework/Window.hpp”
include “/usr/local/driveworks-1.2/samples/src/framework/Window.cpp”
include “/usr/local/driveworks-1.2/samples/src/framework/WindowGLFW.hpp”
include “/usr/local/driveworks-1.2/samples/src/framework/WindowGLFW.cpp”

int main()
{

// ===============
// INITIALIZE GLFW
// ===============

glfwInit();

GLFWwindow* window = glfwCreateWindow(1920, 1236, "GL Window", NULL, NULL);

if (window == NULL)
{
    std::cout << "Failed to create GLFW window" << std::endl;
    glfwTerminate();
    return -1;
}

glfwMakeContextCurrent(window);

glViewport(0, 0, 1920, 1236);


// =====================
// INITIALIZE DRIVEWORKS
// =====================

//create driveworks context handle
dwContextHandle_t m_context = DW_NULL_HANDLE;

//create sdk parameters array
dwContextParameters sdkParams = {};

//get dw verson
dwVersion sdkVersion;
dwGetVersion(&sdkVersion);

//print driveworks version to terminal
std::cout << std::endl << "DriveWorks Version: " << sdkVersion.major << "." << sdkVersion.minor << "." << sdkVersion.patch << std::endl << std::endl;
    
//initialize driveworks
dwStatus status = dwInitialize(&m_context, sdkVersion, &sdkParams);

if(status == DW_SUCCESS) std::cout << "DriveWorks initialized successfully!" << std::endl << std::endl;
else std::cout << "Driveworks initialization failure. Code: " << status << std::endl << std::endl;



// ===================================================
// INITIALIZE RENDER ENGINE MODULE AND RENDERER MODULE
// ===================================================

//create a Render Engine handle
dwRenderEngineHandle_t m_renderEngine = DW_NULL_HANDLE;

//create a Render Engine parameters array
dwRenderEngineParams RenderEngineParams{};

//create a Renderer handle	
dwRendererHandle_t m_renderer = DW_NULL_HANDLE;	

//create a rectangle variable for the Renderer region of interest
dwRect rect;
rect.width  = 1920;
rect.height = 1236;
rect.x      = 0;
rect.y      = 0;
	
//populate the Render Engine parameters with default values
status = dwRenderEngine_initDefaultParams(&RenderEngineParams, 1000, 1000);

if(status == DW_SUCCESS) std::cout << "Render Engine default parameters initialized successfully!" << std::endl;
else std::cout << "Render Engine parameter initialization failure. Code: " << status << std::endl;

//initialie the Render Engine module
status = dwRenderEngine_initialize(&m_renderEngine, &RenderEngineParams, m_context);

if(status == DW_SUCCESS) std::cout << "Render Engine initialized successfully!" << std::endl;
else std::cout << "Render Engine initialization failure. Code: " << status << std::endl;

//initialize the Renderer module
status = dwRenderer_initialize(&m_renderer, m_context);

if(status == DW_SUCCESS) std::cout << "Renderer initialized successfully!" << std::endl;
else std::cout << "Renderer initialization failure. Code: " << status << std::endl;

//set the size of the Renderer region of interest	     
status = dwRenderer_setRect(rect, m_renderer);

if(status == DW_SUCCESS) std::cout << "Renderer screen area set successfully!" << std::endl << std::endl;
else std::cout << "Renderer screen area set failure. Code: " << status << std::endl << std::endl;



// =========================
// CREATE AN INPUT CPU IMAGE
// =========================

//initialize cpu image properties
dwImageProperties inputImageProperties{};
inputImageProperties.type = DW_IMAGE_CPU;
inputImageProperties.format = DW_IMAGE_FORMAT_RAW_UINT16;
inputImageProperties.memoryLayout = DW_IMAGE_MEMORY_TYPE_PITCH;
inputImageProperties.width = 1920;
inputImageProperties.height = 1236;

//create cpu image handle
dwImageHandle_t imageCPU_Handle;

status = dwImage_create(&imageCPU_Handle, inputImageProperties, m_context);

if(status == DW_SUCCESS) std::cout << "Input image handle created successfully!" << std::endl;
else std::cout << "Input image handle creation failure. Code: " << status << std::endl;

//create driveworks cpu image
dwImageCPU *imageCPU;

status = dwImage_getCPU(&imageCPU, imageCPU_Handle);

if(status == DW_SUCCESS) std::cout << "Input image object created successfully!" << std::endl;
else std::cout << "Input image object creation failure. Code: " << status << std::endl;

//get the plane count of the input cpu image
size_t planeCount;
dwImage_getPlaneCount(&planeCount, DW_IMAGE_FORMAT_RAW_UINT16);

//populate the cpu image with arbitrary data
for (uint32_t plane = 0; plane < planeCount; plane++)
{
    for (uint32_t i = 0; i < inputImageProperties.height; i++) 
    {
	for (uint32_t j = 0; j < imageCPU->pitch[plane]; j++) 
	{
	    imageCPU->data[plane][i * imageCPU->pitch[plane] + j] = 213; //insert arbitrary data into the cpu image		
	}
    }
}



// ====================================
// INITIALIZE A CPU2CUDA IMAGE STREAMER
// ====================================

//create streamer handle
dwImageStreamerHandle_t m_streamerCPU2CUDA = DW_NULL_HANDLE;

//initialize the streamer
status = dwImageStreamer_initialize(&m_streamerCPU2CUDA, &inputImageProperties, DW_IMAGE_CUDA, m_context);
 
//check streamer status
if(status == DW_SUCCESS) std::cout << "Image Streamer CPU2CUDA initialized successfully!" << std::endl;
else std::cout << "Image Streamer CPU2CUDA initialization failure. Code: " << status << std::endl;



// ==============================================
// SEND THE INPUT CPU IMAGE TO THE IMAGE STREAMER
// ==============================================

//send cpu image to streamer module
status = dwImageStreamer_producerSend(imageCPU_Handle, m_streamerCPU2CUDA);

//check if image stream was successful
if(status == DW_SUCCESS) std::cout << "CPU Image Streamed to CUDA Image successfully!" << std::endl;
else std::cout << "CPU2CUDA Image Streaming failure. Code: " << status << std::endl;



// ====================================
// FETCH CUDA IMAGE FROM IMAGE STREAMER
// ====================================

//create image handle for cuda image
dwImageHandle_t imageCUDA_Handle;

//fetch cuda image from streamer module
status = dwImageStreamer_consumerReceive(&imageCUDA_Handle, 1000, m_streamerCPU2CUDA);

//check if image fetch was successful
if(status == DW_SUCCESS) std::cout << "CPU Image Received as CUDA Image successfully!" << std::endl;
else std::cout << "CPU2CUDA Image Streaming Recieve failure. Code: " << status << std::endl;

//create empty cuda image
dwImageCUDA* imageCUDA;

//link cuda image to cuda img handle
status = dwImage_getCUDA(&imageCUDA, imageCUDA_Handle);

if(status == DW_SUCCESS) std::cout << "Streamed CUDA image linked to CUDA image handle successfully!" << std::endl;
else std::cout << "Streamed CUDA image handle linking failure. Code: " << status << std::endl;



// =================================
// CLEAN UP CPU2CUDA IMAGE STREAMER
// =================================

//return image to producer
status = dwImageStreamer_consumerReturn(&imageCUDA_Handle, m_streamerCPU2CUDA);

//check if consumer returned image successfully
if(status == DW_SUCCESS) std::cout << "Streamed CUDA image returned by consumer successfully!" << std::endl;
else std::cout << "Streamed CUDA image consumer return failure. Code: " << status << std::endl;

//Return streamed image to original image object
status = dwImageStreamer_producerReturn(&imageCPU_Handle, 1000, m_streamerCPU2CUDA);

//check if producer returned successfully
if(status == DW_SUCCESS) std::cout << "Streamed image returned by producer successfully!" << std::endl;
else std::cout << "Streamed image producer return failure. Code: " << status << std::endl;

//Release the Image Streamer
status = dwImageStreamer_release(&m_streamerCPU2CUDA);

//check if streamer was released successfully
if(status == DW_SUCCESS) std::cout << "Image Streamer CPU2CUDA released successfully!" << std::endl << std::endl;
else std::cout << "CPU2CUDA Image Streamer release failure. Code: " << status << std::endl << std::endl;




// =========================
// INITIALIZE SOFTISP MODULE
// =========================

//create handle for softisp module
dwSoftISPHandle_t m_softISP = DW_NULL_HANDLE;

//initialize softisp parameters
dwSoftISPParams softISPParams = {};
softISPParams.cameraRevision = 0;
softISPParams.height = 1236;
softISPParams.method = DW_TONEMAP_METHOD_AGTM;			
softISPParams.rawFormat = DW_CAMERA_RAW_FORMAT_RCCB;
softISPParams.width = 1920;

//initialize softisp module
status = dwSoftISP_initialize(&m_softISP, &softISPParams, m_context);

if(status == DW_SUCCESS) std::cout << "softISP module initialized successfully!" << std::endl;
else std::cout << "softISP module initialization failure. Code: " << status << std::endl;



// =======================================================
// CONFIGURE SOFTISP FOR DEMOSAIC AND TONEMAP IMAGE OUTPUT
// =======================================================

//set the softISP demosaic method
status = dwSoftISP_setDemosaicMethod(DW_SOFTISP_DEMOSAIC_METHOD_INTERPOLATION, m_softISP);

//check if demosaic method was set successfully
if(status == DW_SUCCESS) std::cout << "softISP demosaicing method set successfully!" << std::endl;
else std::cout << "softISP demosaicing method set failure. Code: " << status << std::endl;

//define the demosaicing region of interest
dwRect cropRegion = {};
cropRegion.height = 1236;
cropRegion.width = 1920;
cropRegion.x = 0;
cropRegion.y = 0;

//set the demosaicing region of interest
status = dwSoftISP_setDemosaicROI(cropRegion, m_softISP);

if(status == DW_SUCCESS) std::cout << "demosaicing ROI set successfully!" << std::endl;
else std::cout << "demosaicing ROI set failure. Code: " << status << std::endl;		

//set the softisp process type
uint32_t processType = DW_SOFTISP_PROCESS_TYPE_DEMOSAIC;
processType |= DW_SOFTISP_PROCESS_TYPE_TONEMAP;

status = dwSoftISP_setProcessType(processType, m_softISP);

if(status == DW_SUCCESS) std::cout << "softisp process type set successfully!" << std::endl;
else std::cout << "softisp process type set failure. Code: " << status << std::endl;

//set the softISP tonemap method
status = dwSoftISP_setTonemapType(DW_TONEMAP_METHOD_AGTM, m_softISP);

if(status == DW_SUCCESS) std::cout << "Tonemap type set successfully!" << std::endl;
else std::cout << "Tonemap type set failure. Code: " << status << std::endl;


// ========================================================================
// CREATE A DESTINATION IMAGE FOR THE DEMOSAIC OUTPUT OF THE SOFTISP MODULE
// ========================================================================

//create and populate an image properties array using the softISP properties that have already been defined
dwImageProperties demosaicImageProperties = {};

//set demosaic image properties
status = dwSoftISP_getDemosaicImageProperties(&demosaicImageProperties, m_softISP);

//check if demosaic image properties were populated successfully
if(status == DW_SUCCESS) std::cout << "demosaic image properties retrieved from softISP properties successfully!" << std::endl;
else std::cout << "demosaic image properties retrieval failure. Code: " << status << std::endl;

//create a handle for the demosaic image
dwImageHandle_t demosaicImage_Handle = DW_NULL_HANDLE;

status = dwImage_create(&demosaicImage_Handle, demosaicImageProperties, m_context);

if(status == DW_SUCCESS) std::cout << "empty demosaic image object created successfully!" << std::endl;
else std::cout << "empty demosaic image object creation failure. Code: " << status << std::endl;

//create an empty CUDA image object for the demosaic image output
dwImageCUDA* demosaicImageCUDA;

status = dwImage_getCUDA(&demosaicImageCUDA, demosaicImage_Handle);

if(status == DW_SUCCESS) std::cout << "empty demosaic CUDA image object created successfully!" << std::endl;
else std::cout << "empty demosaic CUDA image object creation failure. Code: " << status << std::endl;

//bind the empty CUDA image object to the softISP module
status = dwSoftISP_bindOutputDemosaic(demosaicImageCUDA, m_softISP);

//check if image bind was successful
if(status == DW_SUCCESS) std::cout << "demosaicing image output bound successfully!" << std::endl << std::endl;
else std::cout << "demosaicing image output binding failure. Code: " << status << std::endl << std::endl;




// =======================================================================
// CREATE A DESTINATION IMAGE FOR THE TONEMAP OUTPUT OF THE SOFTISP MODULE
// =======================================================================

//create and populate a tonemap image properties array	
dwImageProperties tonemapImageProperties;

tonemapImageProperties = demosaicImageProperties;
tonemapImageProperties.height = 1236;
tonemapImageProperties.width = 1920;
tonemapImageProperties.memoryLayout = DW_IMAGE_MEMORY_TYPE_PITCH; 
    tonemapImageProperties.format = DW_IMAGE_FORMAT_RGBA_UINT8;
tonemapImageProperties.type = DW_IMAGE_CUDA;

//create a handle for the tonemap image
dwImageHandle_t tonemapImage_Handle = DW_NULL_HANDLE;

status = dwImage_create(&tonemapImage_Handle, tonemapImageProperties, m_context);

if(status == DW_SUCCESS) std::cout << "empty tonemap image object created successfully!" << std::endl;
else std::cout << "empty tonemap image object creation failure, code: " << status << std::endl;

//create an empty CUDA image object for the tonemap image output
dwImageCUDA* tonemapImageCUDA;

status = dwImage_getCUDA(&tonemapImageCUDA, tonemapImage_Handle);

if(status == DW_SUCCESS) std::cout << "empty tonemap CUDA image object created successfully!" << std::endl;
else std::cout << "empty tonemap CUDA image object creation failure, code: " << status << std::endl;

//bind the empty CUDA image object to the softISP module
status = dwSoftISP_bindOutputTonemap(tonemapImageCUDA, m_softISP);

if(status == DW_SUCCESS) std::cout << "tonemap image output bound successfully!" << std::endl << std::endl;
else std::cout << "tonemap image output binding failure, code: " << status << std::endl << std::endl;



// ==========================================================================
// BIND THE RAW IMAGE INPUT, AND CREATE THE SELECTED IMAGE OUTPUTS
// ===========================================================================

//bind the input CUDA image to the softISP module
status = dwSoftISP_bindInputRaw(imageCUDA, m_softISP);

if(status == DW_SUCCESS) std::cout << "image input bound to the softISP successfully!" << std::endl;
else std::cout << "softISP image input binding failure. Code: " << status << std::endl;

//process the input CUDA image and put the demosaic and tonemap output into the empty CUDA image objects that were previously bound to the softISP module
status = dwSoftISP_processDeviceAsync(m_softISP);

//check if image processing was successful
if(status == DW_SUCCESS) std::cout << "image processing completed successfully!" << std::endl << std::endl;
else std::cout << "image processing failure. Code: " << status << std::endl << std::endl;



// =================================
// INITIALIZE CUDA2GL IMAGE STREAMER
// =================================

//create null streamer handle
dwImageStreamerHandle_t m_streamerCUDA2GL = DW_NULL_HANDLE;

//initialize image streamer
status = dwImageStreamer_initialize(&m_streamerCUDA2GL, &tonemapImageProperties, DW_IMAGE_GL, m_context);

if(status == DW_SUCCESS) std::cout << "Image Streamer CUDA2GL initialized successfully!" << std::endl;
else std::cout << "Image Streamer CUDA2GL initialization failure. Code: " << status << std::endl;


// ============================================
// STREAM TONEMAP CUDA IMAGE TO STREAMER MODULE
// ============================================

//send cpu image to streamer module
status = dwImageStreamer_producerSend(tonemapImage_Handle, m_streamerCUDA2GL);

//check if image stream was successful
if(status == DW_SUCCESS) std::cout << "Tonemap CUDA Image Streamed to GL Image successfully!" << std::endl;
else std::cout << "CUDA2GL Image Streaming failure. Code: " << status << std::endl;



// ====================================
// FETCH GL IMAGE FROM IMAGE STREAMER
// ====================================

//create image handle for the GL image
dwImageHandle_t imageGL_Handle;

//fetch GL image from the streamer module
status = dwImageStreamer_consumerReceive(&imageGL_Handle, 1000, m_streamerCUDA2GL);

if(status == DW_SUCCESS) std::cout << "Tonemap CUDA Image Received as GL Image successfully!" << std::endl;
else std::cout << "CUDA2GL Image Streaming Recieve failure. Code: " << status << std::endl;

//create the GL image
dwImageGL* imageGL;

status = dwImage_getGL(&imageGL, imageGL_Handle);

if(status == DW_SUCCESS) std::cout << "Tonemap GL image linked to GL image handle successfully!" << std::endl;
else std::cout << "Tonemap GL image handle linking failure. Code: " << status << std::endl;



// =================================
// CLEAN UP CUDA2GL IMAGE STREAMER
// =================================

//return image to producer
status = dwImageStreamer_consumerReturn(&imageGL_Handle, m_streamerCUDA2GL);

//check if consumer returned image successfully
if(status == DW_SUCCESS) std::cout << "Streamed GL image returned by consumer successfully!" << std::endl;
else std::cout << "Streamed GL image consumer return failure. Code: " << status << std::endl;

//Return streamed image to original image object
status = dwImageStreamer_producerReturn(&tonemapImage_Handle, 1000, m_streamerCUDA2GL);

//check if producer returned successfully
if(status == DW_SUCCESS) std::cout << "Tonemap CUDA image returned by producer successfully!" << std::endl;
else std::cout << "CUDA2GL producer return failure. Code: " << status << std::endl;

//release the image streamer
status = dwImageStreamer_release(&m_streamerCUDA2GL);

//check if streamer was released successfully
if(status == DW_SUCCESS) std::cout << "Image Streamer CUDA2GL released successfully!" << std::endl << std::endl;
else std::cout << "CUDA2GL Image Streamer release failure. Code: " << status << std::endl << std::endl;


// =================================
// RENDER THE GL TEXTURE TO A WINDOW
// =================================

glClearColor(0.2f, 0.3f, 0.3f, 1.0f);
glClear(GL_COLOR_BUFFER_BIT);

status = dwRenderer_renderTexture(imageGL->tex, imageGL->target, m_renderer);

if(status == DW_SUCCESS) std::cout << "GL image texture rendered successfully!" << std::endl << std::endl;
else std::cout << "GL image texture render failure. Code: " << status << std::endl << std::endl;


while(!glfwWindowShouldClose(window))
{
	glfwSwapBuffers(window);
	glfwPollEvents();    
}

return 0;

}

2

Dear @mcfalcon,
Can you double check dwImageCUDA image data before sending to GL? You can copy the dwImageCUDA dptr array and verify if the expected data is sent?

3

Hello @SivaRamaKrishnaNV, thank you for your reply.

We have attempted to access the image data within the dwImageCuda dptr data field. However, the data that we are seeing is not the same as the data loaded into the CPU image initially. The code we are using to read some of the data can be seen below. Are we accessing the data correctly?

for(int i = 0; i < 1000; i++) std::cout << (unsigned int) ( (uint8_t*)imageCUDA->dptr )[i] << " ";
std::cout << std::endl;

When we populated the CPU image with data, we set every byte to the value “213”, but when we print out some data from dptr, it looks like undefined memory.

4

Hi There,

I’m interesting if the issue had been worked out with any solutions? Does there any update regarding to this, it highly appreciate if any information? Thanks a lot!