Synchronizing Camera Shutter Timing Across Multiple Orin Devices

Autonomous Vehicles DRIVE AGX Orin DRIVE AGX Orin General
1

DRIVE OS Version: 6.0.10

The following document explains how to synchronize shutter timing across multiple cameras connected to a single Orin:

In addition to that scenario, we would like to synchronize camera shutter timing across multiple Orin devices in a multi-Orin system.

Questions

1. Shutter Timing Control Across Multiple Orin Devices

Is it possible to control camera shutter timing, that is, exposure timing, directly through the DriveWorks API?

If there is no direct way to control shutter timing, would the following approach be feasible?

  • Synchronize the system clocks of multiple Orin devices with an NTP server.
  • Call dwSAL_start() on each Orin based on the same system time.

2. Obtaining the Shutter Timing of Each Camera Frame

Is there a way to obtain the shutter timing, or exposure start time, of each frame directly from the DriveWorks API?

If not, would the following approximation be reasonable?

exposureStart ~= sofTimestampUs - exposureDurationUs[0]

In this assumption, exposureDurationUs[0] corresponds to the long-exposure value in an HDR exposure configuration.

6

Does that mean, you want to synchronize camera timestamps across cameras connected to different orin?

yes.

7

Dear @SivaRamaKrishnaNV
Yes. Specifically, we are testing the following configuration:
Orin 1: three IMX728 cameras
Orin 2: three IMX728 cameras
Orin 3: three IMX728 cameras and four IMX623 cameras
Our goal is to synchronize the shutter timing across all of these cameras.

9

You can synchronize the two orin using an external GM or use one orin as master and other as slave. Once synchrozition is achieved , the camera timestamps will be sync across the orin.

10

Dear @SivaRamaKrishnaNV
Thank you for your response.
My objective is to align camera frames (shutter timing) across three NVIDIA Drive AGX Orin devices.
If all Orin devices are synchronized using an external Grandmaster (PTP),
can the camera frame timestamps (SOF/EOF) obtained via the DriveWorks Camera API be considered automatically aligned across the devices?

11

Correct. As timestamps are PTP synced, the camera timestamps across orins should be in sync

12

Dear @SivaRamaKrishnaNV

I understand. Thank you.
I’ll try capturing the timestamp logs again with PTP synchronized.

14

Dear @SivaRamaKrishnaNV

Additional Questions About Synchronizing Camera Shutter Timing Across Multiple Orin Devices

I conducted the following experiment and would like to ask a few follow-up questions.

Experimental Environment

Hardware

  • Orin #1
  • Orin #2
  • External Grandmaster (GM)

Camera Configuration

Orin #1

  • [csi-ab][link-0] Smartlead BF8S105A-100-02
  • [csi-ab][link-1] Smartlead BF8S103G-100-00

Orin #2

  • [csi-ab][link-0] Smartlead BF8S105A-100-00
  • [csi-ab][link-1] Smartlead BF8S103G-100-00

Experimental Procedure

  1. Stop the existing synchronization

    • Orin #1 and Orin #2 are normally synchronized with the external GM via NTP, so I stopped NTP for this verification.

  2. Synchronize PTP between the external GM and each Orin

    GM (Master)

    sudo ptp4l -i mgbe0 -m

    Orin (Slave)

    sudo ptp4l -i mgbe0_0 -s -H -m

  3. Synchronize the PHCs and system clock on Orin #1 and Orin #2

    • Using mgbe0_0 (/dev/ptp0) as the reference, I synchronized ptp1 through ptp4 and the system clock.
    phc2sys -s /dev/ptp0 -c /dev/ptp1 -O 0 -m
phc2sys -s /dev/ptp0 -c /dev/ptp2 -O 0 -m
phc2sys -s /dev/ptp0 -c /dev/ptp3 -O 0 -m
phc2sys -s /dev/ptp0 -c /dev/ptp4 -O 0 -m
phc2sys -s /dev/ptp0 -c CLOCK_REALTIME -O 0 -m

  4. Verify PTP and system clock synchronization

    • On both Orin devices, I periodically collected the following values:
      • clock_gettime(CLOCK_REALTIME)
      • /dev/ptp0 through /dev/ptp4
    • I confirmed that the system clock and PHCs were synchronized.
    • I created a tool for this synchronization check.

image
image1920×910 461 KB

  1. Rig file configuration

    • In the rig file specified for sample_camera, I added the following sensor definition:
    {
  "name": "time:nvpps:rec:00",
  "parameter": "reference-type=NONE,nvpps-device=/dev/nvpps0",
  "protocol": "time.nvpps"
}

  2. Start sample_camera on both Orin devices at the same time

    • Based on /usr/local/driveworks/samples/src/sensors/camera/camera, I added code to obtain the following timestamps.
    • Immediately after dwSensorCamera_readFrame():
      • system clock
      • dwContext_getCurrentTime()
    • For each acquired frame:
      • dwImage_getTimestamp()
      • dwSensorCamera_getImageTimestamps()

    Source code example:

    status = dwSensorCamera_readFrame(&frame[i], 333333, m_camera[i]);
sys_time[i] = (double)get_clock_time_us()/1000000;
dwContext_getCurrentTime(&current_time[i], m_context);

~

for (uint32_t i = 0; i < m_totalCameras; ++i){
   dwTime_t timeStamp;
   CHECK_DW_ERROR(dwImage_getTimestamp(&timeStamp, img[i]));

   dwImageTimestamps imageTimestamps{};
   CHECK_DW_ERROR(dwSensorCamera_getImageTimestamps(&imageTimestamps, frame[i]));

   printf("■[Cam:%d][Run][sys:%lf][GetCurrent:%lf][getTimestamp:%lf][sof:%lf][eof:%lf][tscEof:%lf]\n",
          i, sys_time[i], (double)current_time[i]/1000000, (double)timeStamp/1000000,
          (double)imageTimestamps.sofTimestampUs/1000000, (double)imageTimestamps.eofTimestampUs/1000000, (double)imageTimestamps.tscEofTimestampUs/1000000);
}

I tested sample_camera in two cases: without adding permissions to /dev/nvpps0, and after adding permissions.

Result 1 (/dev/nvpps0 permissions: crw-------)

Below are excerpts from the logs for camera index [0].

The log values are shown in the following order:

  • sys: system clock
  • GetCurrent: dwContext_getCurrentTime()
  • getTimestamp: dwImage_getTimestamp()
  • sof: sofTimestampUs from dwSensorCamera_getImageTimestamps()
  • eof: eofTimestampUs from dwSensorCamera_getImageTimestamps()
  • tscEof: tscEofTimestampUs from dwSensorCamera_getImageTimestamps()

Orin #1 (orin1_1.txt)

orin1_1.txt(514): ■[Cam:0][sys:1777352850.655282][GetCurrent:1777352850.655287][getTimestamp:2435.024546][sof:0.000000][eof:1777352850.655224][tscEof:2435.024546]
orin1_1.txt(517): ■[Cam:0][sys:1777352850.655818][GetCurrent:1777352850.655820][getTimestamp:2435.057879][sof:0.000000][eof:1777352850.655802][tscEof:2435.057879]
orin1_1.txt(520): ■[Cam:0][sys:1777352850.655984][GetCurrent:1777352850.655986][getTimestamp:2435.091213][sof:0.000000][eof:1777352850.655972][tscEof:2435.091213]
orin1_1.txt(523): ■[Cam:0][sys:1777352850.656130][GetCurrent:1777352850.656132][getTimestamp:2435.124546][sof:0.000000][eof:1777352850.656117][tscEof:2435.124546]
orin1_1.txt(572): ■[Cam:0][sys:1777352850.665905][GetCurrent:1777352850.665969][getTimestamp:2435.157879][sof:0.000000][eof:1777352850.665890][tscEof:2435.157879]
orin1_1.txt(637): ■[Cam:0][sys:1777352850.699307][GetCurrent:1777352850.699310][getTimestamp:2435.191212][sof:0.000000][eof:1777352850.699278][tscEof:2435.191212]
orin1_1.txt(705): ■[Cam:0][sys:1777352850.732690][GetCurrent:1777352850.732692][getTimestamp:2435.224545][sof:0.000000][eof:1777352850.732659][tscEof:2435.224545]
orin1_1.txt(772): ■[Cam:0][sys:1777352850.772785][GetCurrent:1777352850.772787][getTimestamp:2435.257878][sof:0.000000][eof:1777352850.772758][tscEof:2435.257878]
orin1_1.txt(837): ■[Cam:0][sys:1777352850.799316][GetCurrent:1777352850.799319][getTimestamp:2435.291211][sof:0.000000][eof:1777352850.799287][tscEof:2435.291211]
orin1_1.txt(905): ■[Cam:0][sys:1777352850.832641][GetCurrent:1777352850.832643][getTimestamp:2435.324544][sof:0.000000][eof:1777352850.832613][tscEof:2435.324544]

Orin #2 (orin2_1.txt)

orin2_1.txt(514): ■[Cam:0][sys:1777352850.770882][GetCurrent:1777352850.770886][getTimestamp:2470.692089][sof:0.000000][eof:1777352850.770827][tscEof:2470.692089]
orin2_1.txt(517): ■[Cam:0][sys:1777352850.771376][GetCurrent:1777352850.771378][getTimestamp:2470.725422][sof:0.000000][eof:1777352850.771358][tscEof:2470.725422]
orin2_1.txt(520): ■[Cam:0][sys:1777352850.771564][GetCurrent:1777352850.771566][getTimestamp:2470.758755][sof:0.000000][eof:1777352850.771552][tscEof:2470.758755]
orin2_1.txt(523): ■[Cam:0][sys:1777352850.771727][GetCurrent:1777352850.771729][getTimestamp:2470.792088][sof:0.000000][eof:1777352850.771715][tscEof:2470.792088]
orin2_1.txt(572): ■[Cam:0][sys:1777352850.792749][GetCurrent:1777352850.792815][getTimestamp:2470.825421][sof:0.000000][eof:1777352850.792733][tscEof:2470.825421]
orin2_1.txt(637): ■[Cam:0][sys:1777352850.826116][GetCurrent:1777352850.826118][getTimestamp:2470.858753][sof:0.000000][eof:1777352850.826094][tscEof:2470.858753]
orin2_1.txt(705): ■[Cam:0][sys:1777352850.859458][GetCurrent:1777352850.859460][getTimestamp:2470.892087][sof:0.000000][eof:1777352850.859438][tscEof:2470.892087]
orin2_1.txt(772): ■[Cam:0][sys:1777352850.899061][GetCurrent:1777352850.899063][getTimestamp:2470.925420][sof:0.000000][eof:1777352850.899041][tscEof:2470.925420]
orin2_1.txt(837): ■[Cam:0][sys:1777352850.926096][GetCurrent:1777352850.926098][getTimestamp:2470.958753][sof:0.000000][eof:1777352850.926076][tscEof:2470.958753]
orin2_1.txt(905): ■[Cam:0][sys:1777352850.959454][GetCurrent:1777352850.959456][getTimestamp:2470.992086][sof:0.000000][eof:1777352850.959431][tscEof:2470.992086]
  • dwImage_getTimestamp() and tscEofTimestampUs return TSC values.
  • When I extract entries where eofTimestampUs is close between Orin #1 and Orin #2, there is still a 7 ms difference as shown below, so it is difficult to conclude that the shutters were triggered simultaneously.
orin1_1.txt(837): ■[Cam:0][sys:1777352850.799316][GetCurrent:1777352850.799319][getTimestamp:2435.291211][sof:0.000000][eof:1777352850.799287][tscEof:2435.291211]
orin2_1.txt(572): ■[Cam:0][sys:1777352850.792749][GetCurrent:1777352850.792815][getTimestamp:2470.825421][sof:0.000000][eof:1777352850.792733][tscEof:2470.825421]

I have already confirmed with dwImage_getMetaData() that the camera exposure time is the same on Orin #1 and Orin #2.

[0]exposureTime                               :0.000[ms]
[0]sensorStatistics.exposureDurationUs[0]     :10.000[ms]
[0]sensorStatistics.exposureDurationUs[1]     :10.000[ms]
[0]sensorStatistics.exposureDurationUs[2]     :0.055[ms]

Result 2 (/dev/nvpps0 permissions: crwxrwxrwx)

Orin #1 (orin1_2.txt)

orin1_2.txt(517): ■[Cam:0][sys:1777353045.054953][GetCurrent:1777353045.054958][getTimestamp:1777353048.793556][sof:1777353048.762998][eof:1777353048.793556][tscEof:2629.424427]
orin1_2.txt(520): ■[Cam:0][sys:1777353045.055417][GetCurrent:1777353045.055419][getTimestamp:1777353048.826889][sof:1777353048.796331][eof:1777353048.826889][tscEof:2629.457760]
orin1_2.txt(523): ■[Cam:0][sys:1777353045.055584][GetCurrent:1777353045.055586][getTimestamp:1777353048.860223][sof:1777353048.829664][eof:1777353048.860223][tscEof:2629.491094]
orin1_2.txt(526): ■[Cam:0][sys:1777353045.055732][GetCurrent:1777353045.055734][getTimestamp:1777353048.893556][sof:1777353048.862997][eof:1777353048.893556][tscEof:2629.524427]
orin1_2.txt(575): ■[Cam:0][sys:1777353045.065770][GetCurrent:1777353045.065839][getTimestamp:1777353048.926889][sof:1777353048.896330][eof:1777353048.926889][tscEof:2629.557760]
orin1_2.txt(640): ■[Cam:0][sys:1777353045.099192][GetCurrent:1777353045.099195][getTimestamp:1777353048.960222][sof:1777353048.929663][eof:1777353048.960222][tscEof:2629.591093]
orin1_2.txt(708): ■[Cam:0][sys:1777353045.132540][GetCurrent:1777353045.132543][getTimestamp:1777353048.993555][sof:1777353048.962997][eof:1777353048.993555][tscEof:2629.624426]
orin1_2.txt(775): ■[Cam:0][sys:1777353045.169627][GetCurrent:1777353045.169630][getTimestamp:1777353049.026888][sof:1777353048.996330][eof:1777353049.026888][tscEof:2629.657759]
orin1_2.txt(840): ■[Cam:0][sys:1777353045.199163][GetCurrent:1777353045.199166][getTimestamp:1777353049.060221][sof:1777353049.029663][eof:1777353049.060221][tscEof:2629.691092]

Orin #2 (orin2_2.txt)

orin2_2.txt(516): ■[Cam:0][sys:1777353045.180952][GetCurrent:1777353045.180957][getTimestamp:1777353048.878090][sof:1777353048.847532][eof:1777353048.878090][tscEof:2665.091958]
orin2_2.txt(519): ■[Cam:0][sys:1777353045.181488][GetCurrent:1777353045.181490][getTimestamp:1777353048.911423][sof:1777353048.880865][eof:1777353048.911423][tscEof:2665.125291]
orin2_2.txt(522): ■[Cam:0][sys:1777353045.181654][GetCurrent:1777353045.181656][getTimestamp:1777353048.944756][sof:1777353048.914198][eof:1777353048.944756][tscEof:2665.158624]
orin2_2.txt(525): ■[Cam:0][sys:1777353045.181797][GetCurrent:1777353045.181798][getTimestamp:1777353048.978089][sof:1777353048.947531][eof:1777353048.978089][tscEof:2665.191957]
orin2_2.txt(574): ■[Cam:0][sys:1777353045.192646][GetCurrent:1777353045.192718][getTimestamp:1777353049.011422][sof:1777353048.980864][eof:1777353049.011422][tscEof:2665.225290]
orin2_2.txt(639): ■[Cam:0][sys:1777353045.226057][GetCurrent:1777353045.226060][getTimestamp:1777353049.044755][sof:1777353049.014197][eof:1777353049.044755][tscEof:2665.258623]
orin2_2.txt(707): ■[Cam:0][sys:1777353045.259411][GetCurrent:1777353045.259413][getTimestamp:1777353049.078088][sof:1777353049.047530][eof:1777353049.078088][tscEof:2665.291956]
orin2_2.txt(774): ■[Cam:0][sys:1777353045.298528][GetCurrent:1777353045.298531][getTimestamp:1777353049.111421][sof:1777353049.080863][eof:1777353049.111421][tscEof:2665.325289]
  • I assume that the change in dwImage_getTimestamp() to Unix time, and the fact that sofTimestampUs was populated with Unix time, were caused by the /dev/nvpps0 setting.
  • When I extract entries where eofTimestampUs is close between Orin #1 and Orin #2, there is still a 15 ms difference as shown below, so it is difficult to conclude that the shutters were triggered simultaneously.
orin1_2.txt(775): ■[Cam:0][sys:1777353045.169627][GetCurrent:1777353045.169630][getTimestamp:1777353049.026888][sof:1777353048.996330][eof:1777353049.026888][tscEof:2629.657759]
orin2_2.txt(574): ■[Cam:0][sys:1777353045.192646][GetCurrent:1777353045.192718][getTimestamp:1777353049.011422][sof:1777353048.980864][eof:1777353049.011422][tscEof:2665.225290]

Questions

  • Is it correct to determine whether the shutters are synchronized by comparing eofTimestampUs?
  • Are there any shortcomings or mistakes in this experimental setup or procedure when evaluating shutter synchronization under PTP synchronization?

For example:

  • whether using reference-type=NONE is appropriate
  • whether I should use an indicator other than eofTimestampUs