How to synchronize camera with lidar?

Hello,

I want to use lidar and camera at the same time.

How can i use lidar and camera on the one main code, and synchronize lidar with camera.

Hello Minjun,

please refer to the samples code as they are meant to be a starting point for developers to create their program:
lidar_replay_sample - https://docs.nvidia.com/driveworks/dwx_lidar_replay_sample.html
https://docs.nvidia.com/driveworks/lidar_usecase1.html
camera_gmsl_sample - https://docs.nvidia.com/driveworks/dwx_camera_gmsl_sample.html
https://docs.nvidia.com/driveworks/camera_usecase3.html

you should use the timestamp to synchronize the lidar with a camera:

  • for the LiDAR the dwSensorLidar_readPacket api returns a dwLidarDecodedPacket structure that contains the timestamp.
  • for the camera, once you’ve acquired an dwImageObject through dwSensorCamera_getImage/dwSensorCamera_getImageAsync, you can use dwImage_getTimestamp api to get the timestamp of the image.

Thank you.

I have another questions.

First,

I have found the dwLidarDecodedPacket has the parameter “hostTimestamp”.

I think this have the host pc DW’s time. Is it right?

but I can’t find how can i get host pc DW’s time when i receive the camera data.

I have found ‘dwImage_getTimestamp’ function and this has two parameters
output parameter is “timestamp” a pointer to the timestamp
input parameter is “image” a handle to the image.

When i put a parameter which type is dwImageHandle_t which is a input of the fucntion in the ‘dwImage_getTimestamp’, does this function return a host pc DW’s time?

Second,

I want to know there is a sample code or tutorial NVIDIA service to logging multi sensor which is synchronized time.

or I have to make this on my own?

Can you told me the sample code to record a camera and lidar at sametime?

I want to record these sensors at same time with time sync

Dear Minjun,
We do not have DW sample to show using camera and Lidar at a time. If you are looking to record sensor data, you can check using our recorder tool. Please see https://docs.nvidia.com/driveworks/dwx_recorder_tool.html

Dear Minjun,

Yes, this is the host pc DW’s time

yes, this function returns a host pc DW’s time.

Thank you.

Your answer was very helpful.

Lastly, I would like to do an experiment like this, and I would like to ask your opinion.

My goal is to synchronize the camera with Rida as mentioned above.

I refer to the two sample codes you recommended and the doc you provided.

Lidar Calibration Sample
https://docs.nvidia.com/driveworks/dwx_lidar_calibration_sample.html

Camera Calibration Sample
https://docs.nvidia.com/driveworks/dwx_camera_calibration_sample.html

In the example for Camera Calibration, I accepted that the “dwEgotion_update” function is a function that reproduces the vehicle’s Egomotion to match the timestamp of the image received through Camera.

Thank you. Your answer was very helpful to me.

Lastly, I would like to do an experiment like this, and I would like to ask your opinion.

My goal is to synchronize the camera with Rida as mentioned above.

I refer to the two sample codes you recommended and the doc you provided.

Lidar Calibration Sample
https://docs.nvidia.com/driveworks/dwx_lidar_calibration_sample.html

Camera Calibration Sample
https://docs.nvidia.com/driveworks/dwx_camera_calibration_sample.html

In the example for Camera Calibration, I accepted that the “dwEgotion_update” function is a function that reproduces the vehicle’s Egomotion to match the timestamp of the image received through Camera.

I would like to know if there is a function that updates the data of the caemra or Lidar to match timestamp, such as the “dwEgomotion_update” function.

Hi Minjun,

Can you elaborate on what you are trying to achieve, and explain why you want to change the data coming from the camera and lidar?

thanks

I’m sorry, my explanation was very bad and not friendly.

Ultimately, What i want to do is estimating 3D position of objects to autonomous driving.

My development environment is as follows.

I have cars equipped with lidars and a cameras and use Drive AGX Xavier to handle the data from sensors.

I can estimate 3D position of objects from lidars data and i can estimated 3D position of objects from camera data using Computer Vision Processing.

You know Computer Vision Processing is powerful to object classification and get lateral distance but it is very weak in estimating longitudinal distances.

But lidar is powerful to estimate longitudinal distances.

So, I think if i can fuse lidar with camera I can know the lidar distance data of the camera pixel.

To do this, I have to time synchronize camera data and lidar data and calibrate camera and lidar.

That is what i try to achieve and why i ask a question about changing the data coming from the camera and lidar.

hello Minjun,

if running on the then process with the same context of DW, then dwLidarDecodedPacket that has the parameter “hostTimestamp” has the same time domain of the function ‘dwImage_getTimestamp’ output.

why would you want to change the data coming from the sensors? after changing them there is no guaranty that the data will be referring to the same time.

are you using different DW contexts or different processes for the data acquiring?