Recorder timestamps sync issue

Autonomous Vehicles DRIVE AGX Orin DRIVE AGX Orin General
1

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.10.0
DRIVE OS 6.0.8.1
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)
DRIVE AGX Orin Developer Kit (940-63710-0010-200)
DRIVE AGX Orin Developer Kit (940-63710-0010-100)
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
2.1.0
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

Hi,

Following my previous topic: Timestamps Issue with DriveWorks Sensor Manager (GMSL Cameras and GPS) - #12 by al.es
I extended my testing and my rig setup sensors and I’m still tackling this timestamps issue.
My setup consists of 6 cameras, CAN and GPS. Here’s the rig:
setup_rig.txt (6.7 KB)

GPS
I wrote a custom gps driver using your examples. Within my driver I made sure to assign timestamp to the grabbed data using the library function dwContext_getCurrentTime. The driver works well and the timestamps make sense.
2. CAN
I’m using the H1 harness to retrieve CAN data using can.socket.
3. Cameras
using sdk supported Entron F008A120RM0AV2 cameras as camera.gmsl

Recording tool
I’m observing the following behavior when using your recorder tool:
Cameras timestamps are synced. Even though initial up time of each camera is different they are synced. I verified this offline by filming a clock with all cameras and displayed the frames by timestamps and indeed they are aligned.
Other sensors timestamps (CAN,GPS) are lagging by about 4.5 seconds (may defer slightly) behind the cameras timestamps. I verified this offline as well by examining the sensors data vs the images.
example of the above: 1st set of matching camera sensors frames arrive at 1734436857034866 (with micro seconds difference between the 6 sensors). the matching data that is aligned to it from CAN and GPS is at 1734436852126543, 1734436852066543 respectively.

Real time usage
When running in real time I see the same behavior. The sensor manager returns the latest data from each sensor, thus when camera frames are grabbed it returns 6 frames, however when CAN and GPS data frames are grabbed it keeps printing these warning (example):

SensorManager::watchOneSensor() WARNING: Sensor gps[0] is lagging by 4575 ms
SensorManager::watchOneSensor() WARNING: Sensor can[0] is lagging by 4595 ms

the data within the frames is indeed the latest however the timestamps are not synced vs the cameras sensors timestamps.
In my previous post you’ve explained that this is due to the initialization time of the camera sensors and each sensor has a different start time. But the 6 camera sensors are synced even though they have different init times so my gap is why this is not supported for the other sensors as well.

My questions:
How can I ensure that camera sensors timestamps are synced with other sensors of my rig such that all recorded data with your recorder tool is synced?
How can I ensure that camera sensors timestamps are synced with other sensors of my rig such that live stream data will have synced timestamps? (I imagine that solving 1 would solve this as well)
Is it possible to schedule a short online meeting to discuss this issue?

This is currently a major blocker for us to use the recorder tool of NVIDIA Orin DRIVE.

Thanks,
Alon

2

Dear @al.es,
Could you share the full recorder command, terminal log and timestamps files of cameras/ CAN/GPS.

Restart the target(dont run any other commands), remove the time sensor from rig file and share the recorder logs and sensor timestamp files for observation.

3

Hi @SivaRamaKrishnaNV ,

I performed the test for 6 cameras + CAN.

nvidia_recorder_timestamps_logs_files.zip (1.6 MB)

attached in the zip (timestamps files + terminal log for each):

  1. run with time sensor.
  2. run without time sensor after restarting the target.

Didn’t notice much difference between them. the diff in the timestamps is yet there when aligning the data: all 6 camera sensors data is aligned even though in the timestamps files they have different starting times while the CAN timestamp is lagging behind by about 4.5sec relative to the cameras.

both times I run according to your instructions ( DriveWorks SDK Reference: Basic Recording Tool ):
sudo -s
cd /usr/local/driveworks/tools/capture
./recorder <rig_file>
s <recording_path>
s /dev/null
q

Thanks,
Alon

5

@SivaRamaKrishnaNV any update?
this is still a gap

6

Dear @al.es,
One issue I see here is dwContext_getCurrentTime() is based on CLOCK_MONOTONIC and camera timestamps are TSC->PTP converted. Could you check the return value of dwContext_isTimePTPSynchronized() when using dwContext_getCurrentTime()?