Rotate image with DriveWorks APIs

[johannes.gehring]

DRIVE OS Version: 6.0.6

Issue Description:

Dear Nvidia Support Team,

I’m trying to implement a “rotate image” function using DriveOS 6.0.6 and the Driveworks API, but I don’t quite get the expected results. Currently, a 90° rotation is sufficient, therefore I would the function to flip a FullHD image from 1920x1080px to 1080x1920px. As the rotate function of any image viewer would.

My current approach is to use the rectification pipeline for this, as it provides an interface for rotation via a homography matrix. As the output Camera Model, I just use the input model. This does rotate the image, but the resulting image is not as expected. The (FullHD) image is flipped, but the visible image appears as a square, i.e. it does not keep the original aspect ratio. If I try to assign the expected dimensions to teh imgProperties within the rotateImage function, the image is stretched vertically.

dwCameraModelHandle_t cameraModelIn = DW_NULL_HANDLE;
dwRigHandle_t rigConfig = DW_NULL_HANDLE;
dwRig_initializeFromFile(&rigConfig, sdk, argv\[2\]);
uint32_t cameraId = 0;
char cameraName[ ] = “camera:unpositioned:0”;
CHECK_DW_ERROR(dwRig_findSensorByName(&cameraId, cameraName, rigConfig));
CHECK_DW_ERROR(dwCameraModel_initialize(&cameraModelIn, cameraId, rigConfig));
CHECK_DW_ERROR(
dwRectifier_initialize(&rectifier, cameraModelIn, cameraModelIn, sdk));

dwImageHandle_t rotateImage(dwImageHandle_t inputImage, dwRectifierHandle_t rectifier, dwContextHandle_t sdk) {
      dwImageProperties imgProperties{};
      dwImageHandle_t rectifiedImage;
      dwImage_getProperties(&imgProperties, inputImage);
      dwRectifier_appendAllocationAttributes(&imgProperties, rectifier);
      dwImage_create(&rectifiedImage, imgProperties, sdk);
      dwImageCUDA *outputImageCuda;
      dwImageCUDA *inputImageCuda;
      CHECK_DW_ERROR(dwImage_getCUDA(&inputImageCuda, inputImage));
      CHECK_DW_ERROR(dwImage_getCUDA(&outputImageCuda, rectifiedImage));
      CHECK_DW_ERROR(
      dwRectifier_setHomographyFromRotation(90.0, 0.0, 0.0, rectifier));
      CHECK_DW_ERROR(
      dwRectifier_warp(outputImageCuda, inputImageCuda, true, rectifier));
      return rectifiedImage;
}

Original image:

Expected image:

current result without setting the dimensions:

current result when setting the dimensions:

You can ignore the different appearances of the images. They where recorded at different times of day.

In the function description of “dwRectifier_warp” you write:

NOTE: the coordinates of input/output images are implicitly adapted to match the input/output camera if the sizes of the images don’t match the cameras. This will result in stretching of the image and possible loss of aspect ratio.

What would be the correct way to rotate an image or what am I missing to correctly rotate an image using the rectifier pipeline? Is there any way to mitigate this loss of aspect ratio? Am I missing an API which does this easily?

Thanks in advance,
Johannes

[SivaRamaKrishnaNV]

Dear @johannes.gehring ,
May I know the details of use case like, you want to read live camera data and rotate 90 degrees and feed to modules like DNN to perform some inference?

[johannes.gehring]

Thanks for your quick response. My use case is related to this topic from my colleague:

Encoding processed images with DriveWorks 6.0.6

DRIVE OS Version: 6.0.6.0-32441545 Issue Description: We’re trying to encode processed images with H265 or H264. Hello Nvidia Forum Support, we are developing an application with an Nvidia Drive AGX Orin device to read out multiple cameras, do some processing like scaling and rectifying. We then want to send the processed images to another device (Nvidia Drive AGX Thor) which should display the multiple processed images as videos. Currently, we are struggling with the encoding part. We want …

We are trying to fetch images from GMSL cameras using a Drive AGX Orin, do some image processing like rotation and undistortion on them, encode them to a H264/265 video bitstream and send them to another device via RTP. As the monitor displaying the resulting video stream shall be oriented vertically, we have to rotate the image by 90°.

[SivaRamaKrishnaNV]

@johannes.gehring

dwRectifier_setHomographyFromRotation(90.0, 0.0, 0.0, rectifier));

matrix = {0,1,H-1,
1, 0,0,
0,0,1}

Could you try usingdwRectifier_setHomography(matrix, handler) API instead of dwRectifier_setHomographyFromRotation and share your observation

[johannes.gehring]

Thank you for your response.

I’ve tried several matrices:
{0,1,<current imgProperties.height, i.e. 1920> -1 ,1, 0,0,0,0,1}
{0,1,<current imgProperties.height, i.e. 1920>,1, 0,0,0,0,1}
{0,1,<current imgProperties.height, i.e. 1920>,-1, 0,0,0,0,1} (openCVs approach)
{0,1,<expected new width, i.e. 1080>,1, 0,0,0,0,1}
{0, -1, 0, 1, 0, 0, 0, 0, 1} CCW rotation
{0, 1, 0, -1, 0, 0, 0, 0, 1} CW rotation

The last two matrices result in the same images (CW/CCW rotated) as with “setHomographyFromRotation”, the others result in very weird looking results. If I try to adjust the imgProperties dimensions to the expected resulting dimensions, the same vertically stretched images result.

I assumed you mean “image height” with your “H” within the matrix. Could you clarify on that?

[SivaRamaKrishnaNV]

Dear @johannes.gehring ,
Can CUDA kernel can be used to achieve this? You can check generateImage() function as reference to modify the buffer using CUDA kernel from /usr/local/driveworks/samples/src/image/image_common/utils.cu

[johannes.gehring]

Thank you for your response, I will look into it. In the mean time: Can you reproduce this behavior, and what would your approach to rotate an image (strongly preferring Driveworks APIs) be? If there’s no solution using DW, would it make sense trying to integrate CV-CUDA or openCV-Cuda and bypass DW for image processing?

[SivaRamaKrishnaNV]

@johannes.gehring

Can you reproduce this behavior

Could you please share the complete sample code to repro so that I can make changes easily for a solution

[johannes.gehring]

#include <dw/core/context/Context.h>
#include <unistd.h>

#include <cctype>
#include <cstdint>
#include <cstdio>
#include <filesystem>
#include <framework/Log.hpp>
#include <iostream>

#include "dw/calibration/cameramodel/CameraModel.h"
#include "dw/core/base/Status.h"
#include "dw/rig/Rig.h"
#include "dw/sensors/Sensors.h"
#include "framework/Checks.hpp"
#include "image_processing.hpp"
#include "utils.hpp"

int main(int argc, char const* argv[]) {
  dwContextHandle_t sdk = DW_NULL_HANDLE;
  dwContextParameters sdkParams = {};
  dwVersion sdkVersion;
  dwSALHandle_t sal = DW_NULL_HANDLE;
  dwRectifierHandle_t rectifier = DW_NULL_HANDLE;
  dwCameraModelHandle_t cameraModelIn = DW_NULL_HANDLE;
  dwRigHandle_t rigConfig = DW_NULL_HANDLE;
  dwSensorHandle_t camera = DW_NULL_HANDLE;
  dwSensorParams params{};
  const char* sensorName = nullptr;
  const char* sensorProtocol = nullptr;
  const char* sensorParameter = nullptr;
  uint32_t cameraId = 0;
  dwGetVersion(&sdkVersion);
  dwInitialize(&sdk, sdkVersion, &sdkParams);
  dwSAL_initialize(&sal, sdk);

  std::cout << argc << argv << std::endl;
  std::cout << "Context of Driveworks SDK successfully initialized."
            << std::endl;
  std::cout << "Version: " << sdkVersion.major << "." << sdkVersion.minor << "."
            << sdkVersion.patch << std::endl;

  CHECK_DW_ERROR(dwRig_initializeFromFile(&rigConfig, sdk, argv[1]));
  CHECK_DW_ERROR(dwRig_getSensorName(&sensorName, cameraId, rigConfig));
  CHECK_DW_ERROR(dwRig_getSensorProtocol(&sensorProtocol, cameraId, rigConfig));
  CHECK_DW_ERROR(
      dwRig_getSensorParameter(&sensorParameter, cameraId, rigConfig));
  params.parameters = sensorParameter;
  params.protocol = sensorProtocol;
  CHECK_DW_ERROR(dwSAL_createSensor(&camera, params, sal));
  CHECK_DW_ERROR(dwSensor_start(camera));
  CHECK_DW_ERROR(dwRig_findSensorByName(&cameraId, sensorName, rigConfig));
  CHECK_DW_ERROR(dwCameraModel_initialize(&cameraModelIn, cameraId, rigConfig));
  CHECK_DW_ERROR(
      dwRectifier_initialize(&rectifier, cameraModelIn, cameraModelIn, sdk));

  dwImageHandle_t image = DW_NULL_HANDLE;
  std::filesystem::path img_file_path;
  dwStatus status = DW_INTERNAL_ERROR;
  dwCameraFrameHandle_t frame = DW_NULL_HANDLE;
  do {
    status = dwSensorCamera_readFrame(&frame, 50000, camera);
    if (status == DW_TIME_OUT) {
      usleep(5000);
    }
  } while (status != DW_SUCCESS);
  CHECK_DW_ERROR(dwSensorCamera_getImage(
      &image, dwCameraOutputType::DW_CAMERA_OUTPUT_CUDA_RGBA_UINT8, frame));
  image = rectifyImage(image, 1920, 1080, rectifier, sdk);
  savePng("test.png", image);
  CHECK_DW_ERROR(dwImage_destroy(image));
  std::cout << "Image destroyed" << std::endl;
  std::cout << "cm cleaned" << std::endl;
  CHECK_DW_ERROR(dwSAL_release(sal));
  std::cout << "dwSAL_release() successful" << std::endl;
  CHECK_DW_ERROR(dwRectifier_release(rectifier));
  std::cout << "dwRectifier_release() successful" << std::endl;
  CHECK_DW_ERROR(dwRig_release(rigConfig));
  std::cout << "dwRig_release() successful" << std::endl;
  CHECK_DW_ERROR(dwRelease(sdk));
  std::cout << "dwRelease() successful" << std::endl;
  return 0;
}

dwImageHandle_t rectifyImage(dwImageHandle_t inputImage, uint16_t width,
                             uint16_t height, dwRectifierHandle_t rectifier,
                             dwContextHandle_t sdk) {
  dwImageProperties imgProperties{};
  dwImageHandle_t rectifiedImage;
  std::cout << height << width << std::endl;
  dwImage_getProperties(&imgProperties, inputImage);
  dwRectifier_appendAllocationAttributes(&imgProperties, rectifier);
  dwImage_create(&rectifiedImage, imgProperties, sdk);

  switch (imgProperties.type) {
    case DW_IMAGE_CUDA:
      dwImageCUDA *outputImageCuda;
      dwImageCUDA *inputImageCuda;
      CHECK_DW_ERROR(dwImage_getCUDA(&inputImageCuda, inputImage));
      CHECK_DW_ERROR(dwImage_getCUDA(&outputImageCuda, rectifiedImage));
      CHECK_DW_ERROR(
          dwRectifier_setHomographyFromRotation(90.0, 0.0, 0.0, rectifier));
      CHECK_DW_ERROR(
          dwRectifier_warp(outputImageCuda, inputImageCuda, true, rectifier));
      break;
    default:
      throw std::runtime_error(
          "Image type currently not supported for rectification!\n");
  }
  return rectifiedImage;
}

This should be an minimal sample to procuce an image with the applied rotation. It is saved as a PNG using your tutorial an exporting a PNG with lodepng

[johannes.gehring]

Hi @SivaRamaKrishnaNV, any update on this topic? Thank you in advance

[johannes.gehring]

ping.

[SivaRamaKrishnaNV]

Dear @johannes.gehring ,
Thanks for sharing the sample snippet. Could you provide the used rig file and input video for this sample ?

[johannes.gehring]

Hi @SivaRamaKrishnaNV, here is the rig file we’re currently using. The images are taken directly from the camera, so I can’t provide you with an video file. But any FullHD image will do…

rig.json.txt (877 Bytes)

[johannes.gehring]

@SivaRamaKrishnaNV Any update on this matter?

[SivaRamaKrishnaNV]

@johannes.gehring

savePng("test.png", image);

Could you share the code snippet for savePng.

@johannes.gehring

dwRectifier_setHomographyFromRotation(90.0, 0.0, 0.0, rectifier));

what is the output image observed for this case?

matrix = [0, -1, H-1, 1, 0, 0, 0, 0, 1]
dwRectifier_setHomography(matrix, handler)

[johannes.gehring]

Hi @SivaRamaKrishnaNV,

The code for savePng is taken directly from the DriveWorks Tutorials:

void savePng(std::filesystem::path destPath, dwImageHandle_t image) {
  size_t elemSize = 0;
  size_t planeCount = 0;
  uint32_t planeChannels[DW_MAX_IMAGE_PLANES];
  dwVector2ui planeSize[DW_MAX_IMAGE_PLANES];
  dwImageProperties properties{};
  CHECK_DW_ERROR(dwImage_getProperties(&properties, image));

  CHECK_DW_ERROR(dwImage_getDataLayout(&elemSize, &planeCount, planeChannels,
                                       planeSize, &properties));
  size_t size = properties.width * properties.height * elemSize *
                planeChannels[0];  // use only 0th plane for planar image
  size_t dstWidth = planeChannels[0] * elemSize * properties.width;
  std::unique_ptr<uint8_t[]> bytes(new uint8_t[size]);
  if (properties.type == DW_IMAGE_CUDA) {
    dwImageCUDA* imgCuda = DW_NULL_HANDLE;
    CHECK_DW_ERROR(dwImage_getCUDA(&imgCuda, image));
    cudaError_t error =
        cudaMemcpy2D(bytes.get(), dstWidth, imgCuda->dptr[0], imgCuda->pitch[0],
                     dstWidth, properties.height, cudaMemcpyDeviceToHost);
    if (error != cudaSuccess) {
      logError("Error while copying memory from CUDA\n");
    }
  }
  uint32_t res =
      lodepng_encode_file(destPath.c_str(), bytes.get(), properties.width,
                          properties.height, LCT_RGBA, elemSize * 8);
  if (res != 0) {
    std::cerr << " saveImage: lodepng_encode_file returned error= "
              << lodepng_error_text(res) << "(" << res << ")\n";
  }
}

As mentioned before, I don’t understand what you mean by “H-1”, so here’s the code for the following images:

dwImageHandle_t rectifyImage(dwImageHandle_t inputImage, uint16_t width,
                             uint16_t height, dwRectifierHandle_t rectifier,
                             dwContextHandle_t sdk) {
  dwImageProperties imgProperties{};
  dwImageHandle_t rectifiedImage;
  std::cout << height << width << std::endl;
  dwImage_getProperties(&imgProperties, inputImage);
  dwRectifier_appendAllocationAttributes(&imgProperties, rectifier);
  dwImage_create(&rectifiedImage, imgProperties, sdk);
  dwMatrix3f homography = {
      0, -1, static_cast<float32_t>(imgProperties.height - 1), 1, 0, 0, 0,
      0, 1};
  switch (imgProperties.type) {
    case DW_IMAGE_CUDA:
      dwImageCUDA *outputImageCuda;
      dwImageCUDA *inputImageCuda;
      CHECK_DW_ERROR(dwImage_getCUDA(&inputImageCuda, inputImage));
      CHECK_DW_ERROR(dwImage_getCUDA(&outputImageCuda, rectifiedImage));
      CHECK_DW_ERROR(dwRectifier_setHomography(&homography, rectifier));
      CHECK_DW_ERROR(
          dwRectifier_warp(outputImageCuda, inputImageCuda, true, rectifier));
      break;
    default:
      throw std::runtime_error(
          "Image type currently not supported for rectification!\n");
  }
  return rectifiedImage;
}

Original image from our lab:

Resulting image after rectification (gibberish):

[johannes.gehring]

@SivaRamaKrishnaNV any news?

[SivaRamaKrishnaNV]

Dear @johannes.gehring ,
my apologies for delay. I will work on a code sample for this and update you by next week.

[SivaRamaKrishnaNV]

Dear @johannes.gehring ,
Could you check my shared observations in your code snippet via private message

[carolyuu]

Could you please provide any update for this topic?