dwDataConditioner not working like expected

Autonomous Vehicles DRIVE PX 2 DriveWorks
1

Hi,

I’m working on tensorRT inference. For my model it is necessary to subtract the mean image.

In the DriveWorks API documentation is mentioned:

/// Mean image to be subtracted. Default is nullptr.
/// Mean image is expected to be float16 or float32. The pixel format is required to be R or RGBA with
/// interleaved channels. The dimensions of the mean image must meet the dimensions of network input.
const dwImageCUDA *meanImage;

The definition of dwImageCUDA:

/// Defines a CUDA image.
typedef struct {
    /// Defines the properties of the image.
    dwImageProperties prop;
    /// Defines the memory layout of the image.
    dwImageCUDAMemoryType layout;
    /// Defines the pitch of each plane in bytes.
    size_t pitch[DW_MAX_IMAGE_PLANES]; // pitch in bytes
    /// Holds the pointer to the image planes.
    void *dptr[DW_MAX_IMAGE_PLANES];
    /// Holds the CUDA image plane data.
    cudaArray_t array[DW_MAX_IMAGE_PLANES];
    /// Specifies the time, in microseconds, when the image was acquired.
    dwTime_t timestamp_us;
} dwImageCUDA;

Right now I have the two files: meanImage.png and meanImage.binaryproto.
The question is: how can the image file be loaded to the required dwImageCUDA *meanImage?

2

Hello johannHaselberger,

Could you please refer to the DriveWorks Development Guide?
You can find the file file:///usr/local/driveworks-0.3/doc/nvdwx_html/NVIDIA_DriveWorks_DevGuide.pdf on DrivePX2?
Page52 - Inference - Object Detection part. Thanks.

3

I refered the guide already. I am now a few steps further.

What I did so far:

  1. Loaded the mean.png file with LodePNG
  2. Created a dwImageCPU (trivial data type = DW_TYPE_UINT8 as this is the only supported format for dwImageCPU)
  3. Used a dwImageStreamer to get a dwImageCUDA

This is working (I also used a streamer to get the gl image and render it for debugging).

But if assign the mean dwImageCUDA to the dwDataConditioner it complains about, that only float16 or float32 pixel values are allowed.

So my final question for this thread is: how get a dwImageCUDA with trivial data type = DW_TYPE_FLOAT32 from a dwImageCUDA with trivial data type = DW_TYPE_UINT8?

I was not able to find something similar in the documentation. The dwImageConverter and the dwImageStreamer could not be used for this task or?

4

I discarded my previous approach and loading the mean image directly to dwImageCUDA.
But sadly, the dwDataConditioner is not working like expected.
For the pitch value I assumed 4 * 4 * 200. The first 4 because of the four channels (rgba from lodepng), the next 4 because of the float32_t format and the 200 because of the image width.
Regarding the float32_t 4 pitch value I’m not quite sure if this is the correct value.

For testing I created two mean images: a totally black (everything zero) and totally white (all values 255) one. These images have the same shapes as my network input → 200x66x3 (Width x Height x Channels).

I’m also rendering the current model input.

My expectations:

  • no mean image: model input equals roi of camera input
  • black mean image: model input equals roi of camera input -> subtraction with 0
  • white mean image: black model input, because after subtraction each pixel value equals 0

For no mean image specified, the model input is displayed correctly.
The same applies for the black mean image.

But for the white mean image the model input looks quite weird (see attached file).

The code to load the mean image:

// image as rgbargbargba....
        std::vector<unsigned char> meanImage;
	unsigned meanWidth, meanHeight;

	// get the mean image
	lodepng::decode(meanImage, meanWidth, meanHeight,
			meanFilePath, LCT_RGBA);
	

	std::cout << "Get float32_t array from loaded png" << std::endl;
	float32_t interleavedPixelArray[4 * 200 * 66];
	for (int i = 0; i < (int) meanImage.size(); i++) {
		interleavedPixelArray[i] = (float32_t) meanImage[i];
	}

	std::cout << "Creating dwImageCUDA" << std::endl;
	dwImageProperties _meanImageProperties;
	_meanImageProperties.height = 66;
	_meanImageProperties.width = 200;
	_meanImageProperties.planeCount = 1;
	_meanImageProperties.pxlFormat = DW_IMAGE_RGBA;
	_meanImageProperties.pxlType = DW_TYPE_FLOAT32;
	_meanImageProperties.type = DW_IMAGE_CUDA;

	meanImageCUDA_NEW.layout = DW_IMAGE_CUDA_PITCH;
	meanImageCUDA_NEW.pitch[0] = 4 * 4 * 200;
	meanImageCUDA_NEW.prop = _meanImageProperties;
	meanImageCUDA_NEW.timestamp_us = 0;

	// alloc memory 
	cudaMalloc((void **) &meanImageCUDA_NEW.dptr[0],
			dwSizeOf(DW_TYPE_FLOAT32) * 4 * 200 * 66);

	// copy values
	cudaMemcpy(meanImageCUDA_NEW.dptr[0], &interleavedPixelArray,
			sizeof(float32_t) * 4 * 200 * 66, cudaMemcpyHostToDevice);

Is there something wrong in my code (maybe the pitch value) or a bug concerning the dwDataConditioner?

wrongModelInput.png
wrongModelInput.png793×258 187 KB