Timestamping LibArgus images with Cuda consumer

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
,
1

Hi!

For a project, I was hoping to get accurate (kernel) timestamps of images taken by the camera. Taking inspiration from the argus samples in /usr/src/jetson_multimedia_api/argus/samples, a teammate of mine wrote some LibArgus code with a Cuda consumer in order to quickly convert the images to a ROS acceptable format (BGR8 in this case). This worked wonders for getting the image in the correct format, but we could not work out a way to get the kernel timestamp from the LibArgus library with the Cuda consumer. None of the documentation I could find indicates that any Cuda datatype can be converted to the ICaptureMetadata datatype so that getSensorTimestamp() could be called.

Thus, I have two questions, and I can resolve this issue if either one has an answer:
1). Can any Cuda datatype in a Cuda consumer be converted to the ICaptureMetadata datatype so that getSensorTimestamp() can be called on it?
2). Using an IFrameConsumer, what is the format of the image that is outputted, and can that image be converted to a Cuda datatype later so that I can continue to use the Cuda image conversion to BGR8 which has served us in the past?

Any advice is appreciated. Below is the relevant Cuda consumer code. It should look similar to those argus samples, so hopefully it is not too difficult to read.

class CudaFrameAcquire {
  public:
    CudaFrameAcquire(CUeglStreamConnection& connection);
    ~CudaFrameAcquire();

    bool publish(bool leftFrame);
  
  private:
    CUeglStreamConnection& m_connection;
    CUgraphicsResource m_resource;
    CUeglFrame m_frame;
    CUstream m_stream;
};

CudaFrameAcquire::CudaFrameAcquire(CUeglStreamConnection& connection)
                                   : m_connection(connection)
                                   , m_stream(NULL), m_resource(0) {
  CUresult result = cuEGLStreamConsumerAcquireFrame(&m_connection, &m_resource, &m_stream, -1);
  if (result == CUDA_SUCCESS) {
    cuGraphicsResourceGetMappedEglFrame(&m_frame, m_resource, 0, 0);
  }
}

CudaFrameAcquire::~CudaFrameAcquire() {
  if (m_resource) {
    cuEGLStreamConsumerReleaseFrame(&m_connection, m_resource, &m_stream);
  }
}

bool CudaFrameAcquire::publish(bool leftFrame) {
  CUDA_RESOURCE_DESC cudaResourceDesc;
  memset(&cudaResourceDesc, 0, sizeof(cudaResourceDesc));
  cudaResourceDesc.resType = CU_RESOURCE_TYPE_ARRAY;

  cudaResourceDesc.res.array.hArray = m_frame.frame.pArray[0];
  CUsurfObject cudaSurfObj1 = 0;
  CUresult cuResult = cuSurfObjectCreate(&cudaSurfObj1, &cudaResourceDesc);
  if (cuResult != CUDA_SUCCESS) {
    ORIGINATE_ERROR("Unable to create surface object 1 (%s)", getCudaErrorString(cuResult));
  }
  
  cudaResourceDesc.res.array.hArray = m_frame.frame.pArray[1];
  CUsurfObject cudaSurfObj2 = 0;
  cuResult = cuSurfObjectCreate(&cudaSurfObj2, &cudaResourceDesc);
  if (cuResult != CUDA_SUCCESS) {
    ORIGINATE_ERROR("Unable to create surface object 2 (%s)", getCudaErrorString(cuResult));
  }

  //This calls our conversion code to BGR8
  float delta = convert(cudaSurfObj1, cudaSurfObj2, m_frame.width, m_frame.height, oBuffer);
  cuSurfObjectDestroy(cudaSurfObj1);
  cuSurfObjectDestroy(cudaSurfObj2);

  sensor_msgs::Image output;

  //This is the inaccurate timestamp we hope to replace with the getSensorTimestamp() timestamp
  output.header.stamp = ros::Time::now();
  sensor_msgs::fillImage(output, sensor_msgs::image_encodings::BGR8, m_frame.height, m_frame.width, 3 * m_frame.width, (void*) oBuffer);

  //We are using two cameras, so we publish to the respective ROS topic 
  if (leftFrame) {
    left_image_pub.publish(output);
  } else {
    right_image_pub.publish(output);
  }
  return true;
}
3

Could you check if the multimedia API sample cudaHistogram if help to get clue.

4

Just did. The sample doesn’t include any timestamping of images that I could see. It timestamps the process of converting the image into the histogram, but this doesn’t timestamp the frame itself. Please let me know if I am interpreting this example incorrectly.

5

Can’t use ICaptureMetadata getSensorTimestamp() and pass it to cuda?

6

Do you mean that I use an IFrame consumer, get the metadata, then convert the frame to cuda?

Assuming that you don’t, I don’t see how I could get the ICaptureMetadata using a cuda consumer given that I get the frame directly from a cuGraphicsResourceGetMappedEglFrame() call. I don’t see any way to get an ICaptureMetadata out of a CUGraphicsResource or a CUeglFrame. If I’m wrong about this, please let me know.

Assuming that you do, is there any information or a sample I could read on how to convert an IFrame into a cuda datatype?

7

Suppose you already get the timestamp from the argus API but don’t know how to pass it to cuda is that right?

8

Yeah, that sounds right. How do I convert an IFrame into cuda? And how efficient is it?