Timestamps Issue with DriveWorks Sensor Manager (GMSL Cameras and GPS)

al.es · September 16, 2024, 3:23pm

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.10.0
DRIVE OS 6.0.8.1
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)
DRIVE AGX Orin Developer Kit (940-63710-0010-200)
DRIVE AGX Orin Developer Kit (940-63710-0010-100)
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
2.1.0
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

Issue Description

Hi,
I am currently using the DriveWorks Sensor Manager to capture data from 6 GMSL camera streams and a GPS device on an NVIDIA Orin DRIVE AGX system. I’m encountering issues with timestamp accuracy across both the camera and GPS sensors. I’m running my compiled application with sudo, I noticed that when not running with sudo the timestamps don’t make any sense as unix.

Issue with Camera Timestamps:

I use the dwImage_getTimestamp function to obtain timestamps from the camera streams.
During today’s testing (September 16th), the timestamps from all six cameras consistently translate to September 8th instead of the current date.
All camera timestamps are synchronized, but they share this incorrect time gap of 8 days.

Issue with GPS Timestamps:

For the GPS device, I examined both timestamp_us and utcTimeUs fields from the dwGPSFrame.
Observations:
- timestamp_us exhibits the same issue as the camera sensors, showing September 8th instead of September 16th.
- timestamp_us is approximately 4.5 seconds behind the camera sensor timestamps.
- utcTimeUs appears correct, increasing at a 1Hz rate.

Example Debug Prints:

[16-09-2024 13:36:37] SensorManager::watchOneSensor() WARNING: Sensor gps[0] is lagging by 4633 ms
Camera 0 Image: timestamp=1725824679664678, dims=3848x2168
Camera 1 Image: timestamp=1725824679664667, dims=1924x1084
Camera 2 Image: timestamp=1725824679664729, dims=1924x1084
Camera 3 Image: timestamp=1725824679664631, dims=1924x1084
Camera 4 Image: timestamp=1725824679664637, dims=1924x1084
Camera 5 Image: timestamp=1725824679664692, dims=1924x1084
GPS Data: timestamp=1725824675130143, utc_timestamp=1726493797010000

System Time Configuration:
I have verified the system’s date and time configuration, which appears correct:

nvidia@tegra-ubuntu:~$ date
Mon Sep 16 12:52:55 UTC 2024
nvidia@tegra-ubuntu:~$ timedatectl status
Local time: Mon 2024-09-16 12:53:04 UTC
Universal time: Mon 2024-09-16 12:53:04 UTC
RTC time: n/a
Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: no
NTP service: inactive
RTC in local TZ: no

Questions:

Does DriveWorks use a different source clock, such as an internal RTC, for generating timestamps?
How can I correct the time discrepancy between the camera and GPS sensors?
What steps can I take to ensure that all sensor timestamps are synchronized to the global system clock?
Can you shed some light on the gps.uart parameter output-timestamp={host,raw,synced,smoothed}? I tried all of them and not using it at all and it didn’t make a difference.

my rig configuration file for reference:
rig_file.txt (8.3 KB)

Thanks for your assistance!

Best regards,
Alon

SivaRamaKrishnaNV · September 16, 2024, 5:01pm

Means… it does not report sep 8th timings and some random time? Does it report any 1970 times?

Is the target connected to internet?

al.es · September 17, 2024, 7:42am

It reports times that date to 2003 (example: 1041826914380) when not running with sudo.
the target is connected to the internet.

SivaRamaKrishnaNV · September 17, 2024, 8:25am

Dear @al.es ,

If I use 1041826.914380 in https://www.epochconverter.com/ , it shows timestamp from 1970 which indicates it is a TSC timestamp. TSC timestamp gets reset in each reboot. You can verify it. So, TSC-> PTP sync is not happening for camera timestamps. Could you check the timestamp with recorder tool with the same rig file.

al.es · September 17, 2024, 9:22am

When I run the same rig file with the recorder tool I get valid timestamps, however not synced.
For our use case I need the cameras to produce valid and synced timestamps in online mode.

SivaRamaKrishnaNV · September 17, 2024, 9:45am

Dear @al.es ,
With recorder tool do you see today’s timestamps or a week back timestamps?

al.es · September 17, 2024, 9:53am

with the recorder tool I see today’s timestamps

al.es · September 17, 2024, 2:52pm

@SivaRamaKrishnaNV
update:
running now with this config file and after a reboot:
rig_file.txt (6.7 KB)
the timestamps received by the camera sensors are today’s in unix.
However, I still face the issue of the 4.5sec time gap between the gps sensor and the camera sensors.
example:

GPS Data: timestamp=1726584339573549, utc_timestamp=1726584375010000, 
Camera 0 Image: timestamp=1726584344115226, dims=3848x2168
Camera 1 Image: timestamp=1726584344115150, dims=1924x1084
Camera 2 Image: timestamp=1726584344115138, dims=1924x1084
Camera 3 Image: timestamp=1726584344115150, dims=1924x1084
Camera 4 Image: timestamp=1726584344115205, dims=1924x1084
Camera 5 Image: timestamp=1726584344115146, dims=1924x1084
[17-09-2024 14:46:15] SensorManager::watchOneSensor() WARNING: Sensor gps[0] is lagging by 4575 ms

How can I solve this?

SivaRamaKrishnaNV · September 17, 2024, 5:24pm

Hi @al.es,
Could you try with one camera and GPS sensor? Do you see any framedrop or data packet loss messages in recorder tool?

SivaRamaKrishnaNV · September 18, 2024, 8:26am

Dear @al.es,
I see camera sensor timestamps have delay. This is expected as cameras initialization and starting involves many steps. How about chopping(DriveWorks SDK Reference: Recording Chopping Tool) the intial 4.5 sec GPS data ?

al.es · September 18, 2024, 8:40am

yes below are the results:

I did tests using this rig_file:
1_camera_1_gps_rig.txt (2.7 KB)
once with gps device set to gps.ublox and once gps.uart:

gps.ublox

recorder tool output
I manage to record using the recorder tool but I don’t see any timestamps within the file.
This is the output when using the recording tool, starting a recording and stopping it with s /dev/null and then q:

[18-09-2024 08:17:54] Platform: Detected Drive Orin P3710
[18-09-2024 08:17:54] Platform: currently selected GPU device integrated ID 0
[18-09-2024 08:17:54] Context::mountResourceCandidateDataPath resource FAILED to mount from ‘./resources’: VirtualFileSystem: Failed to mount ‘./resources/resources.pak’
[18-09-2024 08:17:54] Context::mountResourceCandidateDataPath resource FAILED to mount from ‘/usr/local/driveworks-5.14/tools/capture/data’: VirtualFileSystem: Failed to mount ‘/usr/local/driveworks-5.14/tools/capture/data/resources.pak’
[18-09-2024 08:17:54] Context::mountResourceCandidateDataPath resource FAILED to mount from ‘/usr/local/driveworks-5.14/tools/capture/…/…/data’: VirtualFileSystem: Failed to mount ‘/usr/local/driveworks-5.14/tools/capture/…/…/data/resources.pak’
[18-09-2024 08:17:54] Context::mountResourceCandidateDataPath resource FAILED to mount from ‘/usr/local/driveworks-5.14/data’: VirtualFileSystem: Failed to mount ‘/usr/local/driveworks-5.14/data/resources.pak’
[18-09-2024 08:17:54] Context::findResourcesPackageInPathWalk: Could not find ./resources/resources.pak in upto 7 parent directories from /usr/local/driveworks-5.14/tools/capture/…/…/targets/aarch64-Linux/lib/libdw_base.so.5.14
[18-09-2024 08:17:54] Context::findResourcesPackageInPathWalk: Could not find ./resources/resources.pak in upto 7 parent directories from /usr/local/driveworks-5.14/targets/aarch64-Linux/lib/libdw_base.so.5.14
[18-09-2024 08:17:54] SDK: No resources(.pak) mounted, some modules will not function properly
[18-09-2024 08:17:54] egl::Display: found 1 EGL devices
[18-09-2024 08:17:54] egl::Display: use drm device: /dev/dri/card0
[18-09-2024 08:17:54] [18-09-2024 08:17:54] Initialize DriveWorks SDK v5.14.74
[18-09-2024 08:17:54] [18-09-2024 08:17:54] Release build with GNU 9.3.0 from buildbrain-branch-0-g30b7ba5bb0c against Drive PDK v6.0.8.0
[18-09-2024 08:17:54] Rig overlay directory does not exist: /storage/driveworks/data/vehicle_cfg
[18-09-2024 08:17:54] Platform: currently selected GPU device integrated ID 0
[18-09-2024 08:17:54] rig::DatabaseOverlayer: Failed to open a file ‘/tmp/car_vin’ - file likely does not exist - skipping overlay
[18-09-2024 08:17:54] No valid data file found for camera:front:center in parameter string: camera-name=F008A120RM0AV2,interface=csi-ab,CPHY-mode=1,link=0,format=h264,output-format=processed,async-record=0,frame-mod=2,file-buffer-size=16777216,skip-eeprom=0 (using configuration folder /home/nvidia/ar0820_camera_simple_office_testing/)
[18-09-2024 08:17:54] No valid data file found for gps in parameter string: device=/dev/ttyACM0,baud=115200,fifo-size=1024,output-timestamp=synced (using configuration folder /home/nvidia/ar0820_camera_simple_office_testing/)
[18-09-2024 08:17:54] No valid data file found for time:nvpps:rec:00 in parameter string: reference-type=NONE,nvpps-device=/dev/nvpps0 (using configuration folder /home/nvidia/ar0820_camera_simple_office_testing/)
[18-09-2024 08:17:54] rig::DatabaseOverlayer: Failed to open a file ‘/tmp/car_vin’ - file likely does not exist - skipping overlay
[18-09-2024 08:17:54] SensorFactory::createSensor() → time.nvpps, nvpps-device=/dev/nvpps0
[18-09-2024 08:17:54] Found driver: time.nvpps
[18-09-2024 08:17:54] TimeSensor: failed to set NVPPS parameters. Error: Invalid argument
[18-09-2024 08:17:54] This platform does not support NVPPS GPIO mode. Fallback to timer mode
[18-09-2024 08:17:54] TimeSensor: initialized with no UTC time reference.
[18-09-2024 08:17:54] SensorFactory: Override default timeSensor with the user passed one
[18-09-2024 08:17:54] EndpointNVPPS: started on /dev/nvpps0
[18-09-2024 08:17:54] SensorFactory::createSensor() → camera.gmsl, camera-name=F008A120RM0AV2,interface=csi-ab,CPHY-mode=1,link=0,format=h264,output-format=processed,async-record=0,frame-mod=2,file-buffer-size=16777216,skip-eeprom=0
[18-09-2024 08:17:54] Found driver: camera.gmsl
[18-09-2024 08:17:54] SensorFactory::createSensor() → camera.gmsl.master,
[18-09-2024 08:17:54] Found driver: camera.gmsl.master
[18-09-2024 08:17:55] CameraMaster::parseDevBlock Getting device info list.
[18-09-2024 08:17:55] devBlock: 0 Slave = 0 Interface = csi-ab Camera_name = F008A120RM0AV2 Link = 0 GroupInit = 0
[18-09-2024 08:17:55] Camera Match Name: F008A120RM0AV2 Description: Entron F008A120RM0A module - 120-deg FOV, 24-bit capture, MIPI-AR0820, MAX9295 linkIndex: 4294967295 serInfo.Name: MAX9295
[18-09-2024 08:17:55] Authentication is disabled for camera F008A120RM0AV2
[18-09-2024 08:17:55] Client, Setting up information for camera ID 0
[18-09-2024 08:17:55] Client, successfully found info for camera ID 0 bound to id 0
[18-09-2024 08:17:55] CameraClient: using NITO found at /usr/share/camera/F008A120RM0AV2.nito
[18-09-2024 08:17:55] SensorFactory::createSensor() → gps.ublox, device=/dev/ttyACM0,baud=115200,fifo-size=1024,output-timestamp=synced
[18-09-2024 08:17:55] Found driver: gps.ublox
[18-09-2024 08:17:55] SensorFactory::createSensor() → time.nvpps, reference-type=NONE,nvpps-device=/dev/nvpps0
[18-09-2024 08:17:55] Found driver: time.nvpps
[18-09-2024 08:17:55] TimeSensor: initialized with no UTC time reference.
[18-09-2024 08:17:55] SensorFactory: Override default timeSensor with the user passed one
[18-09-2024 08:17:56] EndpointNVPPS: stopped on /dev/nvpps0
[18-09-2024 08:17:56] EndpointNVPPS: started on /dev/nvpps0
[18-09-2024 08:17:56] SIPLMaster::SIPLMaster: Setting up master camera
[18-09-2024 08:17:56] CameraMaster available outputs: 1
[18-09-2024 08:17:56] CameraClient: setting pipeline config for camera ID 0
[18-09-2024 08:17:56] CameraClient, SIPLNotifier created
[18-09-2024 08:17:56] CameraMaster: master initiation
MAX96712: Revision 5 detected
MAX96712 Link 0: PHY optimization was enabled
MAX9295: Revision 8 detected!
Sensor AR0820 GRBG Rev 2.1 detected!
[18-09-2024 08:17:57] CameraMaster: Bootstrap NvSIPL and ClientPipelines complete
[18-09-2024 08:17:57] CameraClient::readEEPROMInfo:
[18-09-2024 08:17:57] mmIntrinsicModelID: 27
[18-09-2024 08:17:57] mmIntrinsicCoeff[0]: -0.000000
[18-09-2024 08:17:57] mmIntrinsicCoeff[1]: 0.000000
[18-09-2024 08:17:57] mmIntrinsicCoeff[2]: 0.000000
[18-09-2024 08:17:57] mmIntrinsicCoeff[3]: 0.000000
[18-09-2024 08:17:57] mmIntrinsicCoeff[4]: 0.000000
[18-09-2024 08:17:57] mmIntrinsicCoeff[5]: 0.000000
[18-09-2024 08:17:57] mmIntrinsicCoeff[6]: 0.000000
[18-09-2024 08:17:57] mmIntrinsicCoeff[7]: 0.000000
[18-09-2024 08:17:57] mmIntrinsicCoeff[8]: 0.000000
[18-09-2024 08:17:57] mmIntrinsicCoeff[9]: 1.892143e+11
[18-09-2024 08:17:57] mmIntrinsicCoeff[10]: 0.000000
[18-09-2024 08:17:57] mmIntrinsicCoeff[11]: 0.000000
[18-09-2024 08:17:57] mmIntrinsicCoeff[12]: 0.000000
[18-09-2024 08:17:57] mmIntrinsicCoeff[13]: 0.000000
[18-09-2024 08:17:57] mmIntrinsicCoeff[14]: 0.000170
[18-09-2024 08:17:57] mmIntrinsicCoeff[15]: 8.000000
[18-09-2024 08:17:57] mmIntrinsicMeasuredFlagID: 237
[18-09-2024 08:17:57] fwVersionMajor: 0
[18-09-2024 08:17:57] fwVersionMinor: 0
[18-09-2024 08:17:57] fwVersionPatch: 0
[18-09-2024 08:17:57] [18-09-2024 08:17:57] SAL: bootstrap complete
[18-09-2024 08:17:57] CameraClient: serializer bitrate not specified. Using 8000000.
[18-09-2024 08:17:57] CameraClient: h264 and h265 video format encoding is deprecated. Please switch to a different format.
[18-09-2024 08:17:57] EncoderNvMedia: Setting encode on instance 0
[18-09-2024 08:17:57] EncoderNvMedia: Header Information not available for h264/h265 , Default Header Data
Rig: ar0820_1_camera.json NewSink: /dev/null
[18-09-2024 08:17:57] CameraClient: allocating image pool for ICP
[18-09-2024 08:17:57] Allocating NvSci group
[18-09-2024 08:17:57] CameraClient: allocate image pool for 0
[18-09-2024 08:17:57] Allocating NvSci ISP0 images
[18-09-2024 08:17:57] EndpointTTY: started on /dev/ttyACM0
[18-09-2024 08:17:57] SensorManager::readSensor() Failed reading sensor #1: DW_NOT_READY
[18-09-2024 08:17:57] SensorManager::start() started
Press s to start, just to see progress, q to quit.
[18-09-2024 08:17:57] SIPLClient: Acquisition started
[18-09-2024 08:17:57] [18-09-2024 08:17:58] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:17:59] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:00] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:01] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:02] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:03] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:04] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:05] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:06] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:07] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:08] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:09] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:10] SIPLDeviceBlockNotificationHandler: Queue timeout
s[18-09-2024 08:18:11] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:12] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:13] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:14] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:15] SIPLDeviceBlockNotificationHandler: Queue timeout
/home/nvidia/ar0820_camera_simple_office_testing/gps_timestamps_testing[18-09-2024 08:18:16] SIPLDeviceBlockNotificationHandler: Queue timeout

[18-09-2024 08:18:16] EncoderNvMedia: dropping PFrame before IFrame
[18-09-2024 08:18:16] EncoderNvMedia: dropping PFrame before IFrame
[18-09-2024 08:18:16] EncoderNvMedia: dropping PFrame before IFrame
[18-09-2024 08:18:17] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:18] SIPLDeviceBlockNotificationHandler: Queue timeout
Rig: ar0820_1_camera.json NewSink: /home/nvidia/ar0820_camera_simple_office_testing/gps_timestamps_testing
Rig: ar0820_1_camera.json Sensor: camera:front:center Bytes: 1187334
Rig: ar0820_1_camera.json Sensor: gps Bytes: 19962
Rig: ar0820_1_camera.json Sensor: time:nvpps:rec:00 Bytes: 72
Rig: ar0820_1_camera.json EndOfSensorsInfo
[18-09-2024 08:18:19] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:20] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:21] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:22] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:23] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:24] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:25] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:26] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:27] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:28] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:29] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:30] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:31] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:32] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:33] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:34] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:35] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:36] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:37] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:38] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:39] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:40] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:41] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:42] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:43] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:44] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:45] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:46] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:47] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:48] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:49] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:50] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:51] SIPLDeviceBlockNotificationHandler: Queue timeout
s [18-09-2024 08:18:52] SIPLDeviceBlockNotificationHandler: Queue timeout
/dev/[18-09-2024 08:18:53] SIPLDeviceBlockNotificationHandler: Queue timeout
null
[18-09-2024 08:18:54] IndexTable: saving to file “/home/nvidia/ar0820_camera_simple_office_testing/gps_timestamps_testing/gps.bin.seek”
[18-09-2024 08:18:54] SensorSerializer: could not save index table
[18-09-2024 08:18:54] IndexTable: saving to file “/home/nvidia/ar0820_camera_simple_office_testing/gps_timestamps_testing/time_nvpps_rec_00.bin.seek”
Rig: ar0820_1_camera.json NewSink: /dev/null
Rig: ar0820_1_camera.json Sensor: camera:front:center Bytes: 188358
Rig: ar0820_1_camera.json Sensor: gps Bytes: 2183
Rig: ar0820_1_camera.json Sensor: time:nvpps:rec:00 Bytes: 72
Rig: ar0820_1_camera.json EndOfSensorsInfo
[18-09-2024 08:18:54] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:55] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:56] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:57] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:58] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:18:59] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:19:00] SIPLDeviceBlockNotificationHandler: Queue timeout
q
[18-09-2024 08:19:01] SIPLNotifier: Notification received from pipeline index:0 of type: NOTIF_WARN_ICP_FRAME_DROP
[18-09-2024 08:19:01] SIPLNotifier: Notification received from pipeline index:0 of type: NOTIF_WARN_ICP_FRAME_DROP
[18-09-2024 08:19:01] SIPLNotifier: Notification received from pipeline index:0 of type: NOTIF_WARN_ICP_FRAME_DROP
[18-09-2024 08:19:01] SIPLNotifier: Notification received from pipeline index:0 of type: NOTIF_WARN_ICP_FRAME_DROP
[18-09-2024 08:19:01] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:19:01] EndpointTTY: stopped on /dev/ttyACM0
[18-09-2024 08:19:02] SIPLClient: ImageQueue timeout
[18-09-2024 08:19:02] SIPLNotifier: Event notification queue timeout
[18-09-2024 08:19:02] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:19:03] SIPLClient: ImageQueue timeout
[18-09-2024 08:19:03] SIPLNotifier: Event notification queue timeout
[18-09-2024 08:19:03] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:19:03] EndpointNVPPS: stopped on /dev/nvpps0
[18-09-2024 08:19:03] SensorManager::stop() stopped
[18-09-2024 08:19:03] Buffers(Processed) should be released before calling DeInit() on sipl camera
[18-09-2024 08:19:04] SIPLClient: ImageQueue timeout
[18-09-2024 08:19:04] Deinit master camera
[18-09-2024 08:19:04] SIPLNotifier: Event notification queue shutdown
[18-09-2024 08:19:04] CameraClient: Stopping client
[18-09-2024 08:19:04] Releasing camera master
[18-09-2024 08:19:04] SIPLDeviceBlockNotificationHandler: Queue timeout
[18-09-2024 08:19:04] [18-09-2024 08:19:04] Releasing Driveworks SDK Context

online app usage
When I use the same rig file for online, I don’t get any messages since the sensor manager doesn’t recognize the timestamps of the sensor (it’s 0) and I get this warning:

[18-09-2024 07:51:25] SensorManager::watchOneSensor() WARNING: Sensor gps[0] is lagging by 1726645854809 ms

gps.uart

recorder tool output
I get this seg fault:

root@tegra-ubuntu:/usr/local/driveworks/tools/capture# ./recorder /home/nvidia/ar0820_camera_simple_office_testing/ar0820_1_camera.json
[18-09-2024 07:47:17] Platform: Detected Drive Orin P3710
[18-09-2024 07:47:17] Platform: currently selected GPU device integrated ID 0
[18-09-2024 07:47:17] Context::mountResourceCandidateDataPath resource FAILED to mount from ‘./resources’: VirtualFileSystem: Failed to mount ‘./resources/resources.pak’
[18-09-2024 07:47:17] Context::mountResourceCandidateDataPath resource FAILED to mount from ‘/usr/local/driveworks-5.14/tools/capture/data’: VirtualFileSystem: Failed to mount ‘/usr/local/driveworks-5.14/tools/capture/data/resources.pak’
[18-09-2024 07:47:17] Context::mountResourceCandidateDataPath resource FAILED to mount from ‘/usr/local/driveworks-5.14/tools/capture/…/…/data’: VirtualFileSystem: Failed to mount ‘/usr/local/driveworks-5.14/tools/capture/…/…/data/resources.pak’
[18-09-2024 07:47:17] Context::mountResourceCandidateDataPath resource FAILED to mount from ‘/usr/local/driveworks-5.14/data’: VirtualFileSystem: Failed to mount ‘/usr/local/driveworks-5.14/data/resources.pak’
[18-09-2024 07:47:17] Context::findResourcesPackageInPathWalk: Could not find ./resources/resources.pak in upto 7 parent directories from /usr/local/driveworks-5.14/tools/capture/…/…/targets/aarch64-Linux/lib/libdw_base.so.5.14
[18-09-2024 07:47:17] Context::findResourcesPackageInPathWalk: Could not find ./resources/resources.pak in upto 7 parent directories from /usr/local/driveworks-5.14/targets/aarch64-Linux/lib/libdw_base.so.5.14
[18-09-2024 07:47:17] SDK: No resources(.pak) mounted, some modules will not function properly
[18-09-2024 07:47:17] egl::Display: found 1 EGL devices
[18-09-2024 07:47:17] egl::Display: use drm device: /dev/dri/card0
[18-09-2024 07:47:17] [18-09-2024 07:47:17] Initialize DriveWorks SDK v5.14.74
[18-09-2024 07:47:17] [18-09-2024 07:47:17] Release build with GNU 9.3.0 from buildbrain-branch-0-g30b7ba5bb0c against Drive PDK v6.0.8.0
[18-09-2024 07:47:17] Rig overlay directory does not exist: /storage/driveworks/data/vehicle_cfg
[18-09-2024 07:47:17] Platform: currently selected GPU device integrated ID 0
[18-09-2024 07:47:17] rig::DatabaseOverlayer: Failed to open a file ‘/tmp/car_vin’ - file likely does not exist - skipping overlay
[18-09-2024 07:47:17] No valid data file found for camera:front:center in parameter string: camera-name=F008A120RM0AV2,interface=csi-ab,CPHY-mode=1,link=0,format=h264,output-format=processed,async-record=0,frame-mod=2,file-buffer-size=16777216,skip-eeprom=0 (using configuration folder /home/nvidia/ar0820_camera_simple_office_testing/)
[18-09-2024 07:47:17] No valid data file found for gps in parameter string: device=/dev/ttyACM0,baud=115200,fifo-size=1024,output-timestamp=synced (using configuration folder /home/nvidia/ar0820_camera_simple_office_testing/)
[18-09-2024 07:47:17] No valid data file found for time:nvpps:rec:00 in parameter string: reference-type=NONE,nvpps-device=/dev/nvpps0 (using configuration folder /home/nvidia/ar0820_camera_simple_office_testing/)
[18-09-2024 07:47:17] rig::DatabaseOverlayer: Failed to open a file ‘/tmp/car_vin’ - file likely does not exist - skipping overlay
[18-09-2024 07:47:17] SensorFactory::createSensor() → time.nvpps, nvpps-device=/dev/nvpps0
[18-09-2024 07:47:17] Found driver: time.nvpps
[18-09-2024 07:47:17] TimeSensor: failed to set NVPPS parameters. Error: Invalid argument
[18-09-2024 07:47:17] This platform does not support NVPPS GPIO mode. Fallback to timer mode
[18-09-2024 07:47:17] TimeSensor: initialized with no UTC time reference.
[18-09-2024 07:47:17] SensorFactory: Override default timeSensor with the user passed one
[18-09-2024 07:47:17] EndpointNVPPS: started on /dev/nvpps0
[18-09-2024 07:47:17] SensorFactory::createSensor() → camera.gmsl, camera-name=F008A120RM0AV2,interface=csi-ab,CPHY-mode=1,link=0,format=h264,output-format=processed,async-record=0,frame-mod=2,file-buffer-size=16777216,skip-eeprom=0
[18-09-2024 07:47:17] Found driver: camera.gmsl
[18-09-2024 07:47:17] SensorFactory::createSensor() → camera.gmsl.master,
[18-09-2024 07:47:17] Found driver: camera.gmsl.master
[18-09-2024 07:47:18] CameraMaster::parseDevBlock Getting device info list.
[18-09-2024 07:47:18] devBlock: 0 Slave = 0 Interface = csi-ab Camera_name = F008A120RM0AV2 Link = 0 GroupInit = 0
[18-09-2024 07:47:18] Camera Match Name: F008A120RM0AV2 Description: Entron F008A120RM0A module - 120-deg FOV, 24-bit capture, MIPI-AR0820, MAX9295 linkIndex: 4294967295 serInfo.Name: MAX9295
[18-09-2024 07:47:18] Authentication is disabled for camera F008A120RM0AV2
[18-09-2024 07:47:18] Client, Setting up information for camera ID 0
[18-09-2024 07:47:18] Client, successfully found info for camera ID 0 bound to id 0
[18-09-2024 07:47:18] CameraClient: using NITO found at /usr/share/camera/F008A120RM0AV2.nito
[18-09-2024 07:47:18] SensorFactory::createSensor() → gps.uart, device=/dev/ttyACM0,baud=115200,fifo-size=1024,output-timestamp=synced
[18-09-2024 07:47:18] Found driver: gps.uart
[18-09-2024 07:47:18] SensorFactory::createSensor() → time.nvpps, reference-type=NONE,nvpps-device=/dev/nvpps0
[18-09-2024 07:47:18] Found driver: time.nvpps
[18-09-2024 07:47:18] TimeSensor: initialized with no UTC time reference.
[18-09-2024 07:47:18] SensorFactory: Override default timeSensor with the user passed one
[18-09-2024 07:47:19] EndpointNVPPS: stopped on /dev/nvpps0
[18-09-2024 07:47:19] EndpointNVPPS: started on /dev/nvpps0
[18-09-2024 07:47:19] SIPLMaster::SIPLMaster: Setting up master camera
[18-09-2024 07:47:19] CameraMaster available outputs: 1
[18-09-2024 07:47:19] CameraClient: setting pipeline config for camera ID 0
[18-09-2024 07:47:19] CameraClient, SIPLNotifier created
[18-09-2024 07:47:19] CameraMaster: master initiation
MAX96712: Revision 5 detected
MAX96712 Link 0: PHY optimization was enabled
MAX9295: Revision 8 detected!
Sensor AR0820 GRBG Rev 2.1 detected!
[18-09-2024 07:47:20] CameraMaster: Bootstrap NvSIPL and ClientPipelines complete
[18-09-2024 07:47:20] CameraClient::readEEPROMInfo:
[18-09-2024 07:47:20] mmIntrinsicModelID: 27
[18-09-2024 07:47:20] mmIntrinsicCoeff[0]: -0.000000
[18-09-2024 07:47:20] mmIntrinsicCoeff[1]: 0.000000
[18-09-2024 07:47:20] mmIntrinsicCoeff[2]: 0.000000
[18-09-2024 07:47:20] mmIntrinsicCoeff[3]: 0.000000
[18-09-2024 07:47:20] mmIntrinsicCoeff[4]: 0.000000
[18-09-2024 07:47:20] mmIntrinsicCoeff[5]: 0.000000
[18-09-2024 07:47:20] mmIntrinsicCoeff[6]: 0.000000
[18-09-2024 07:47:20] mmIntrinsicCoeff[7]: 0.000000
[18-09-2024 07:47:20] mmIntrinsicCoeff[8]: 0.000000
[18-09-2024 07:47:20] mmIntrinsicCoeff[9]: 1.892143e+11
[18-09-2024 07:47:20] mmIntrinsicCoeff[10]: 0.000000
[18-09-2024 07:47:20] mmIntrinsicCoeff[11]: 0.000000
[18-09-2024 07:47:20] mmIntrinsicCoeff[12]: 0.000000
[18-09-2024 07:47:20] mmIntrinsicCoeff[13]: 0.000000
[18-09-2024 07:47:20] mmIntrinsicCoeff[14]: 0.000170
[18-09-2024 07:47:20] mmIntrinsicCoeff[15]: 8.000000
[18-09-2024 07:47:20] mmIntrinsicMeasuredFlagID: 237
[18-09-2024 07:47:20] fwVersionMajor: 0
[18-09-2024 07:47:20] fwVersionMinor: 0
[18-09-2024 07:47:20] fwVersionPatch: 0
[18-09-2024 07:47:20] [18-09-2024 07:47:20] SAL: bootstrap complete
[18-09-2024 07:47:20] CameraClient: serializer bitrate not specified. Using 8000000.
[18-09-2024 07:47:20] CameraClient: h264 and h265 video format encoding is deprecated. Please switch to a different format.
[18-09-2024 07:47:20] EncoderNvMedia: Setting encode on instance 0
[18-09-2024 07:47:20] EncoderNvMedia: Header Information not available for h264/h265 , Default Header Data
Rig: ar0820_1_camera.json NewSink: /dev/null
[18-09-2024 07:47:20] CameraClient: allocating image pool for ICP
[18-09-2024 07:47:20] Allocating NvSci group
[18-09-2024 07:47:20] CameraClient: allocate image pool for 0
[18-09-2024 07:47:20] Allocating NvSci ISP0 images
[18-09-2024 07:47:20] EndpointTTY: started on /dev/ttyACM0
[18-09-2024 07:47:20] SensorManager::readSensor() Failed reading sensor #1: DW_NOT_READY
[18-09-2024 07:47:20] SensorManager::start() started
Press s to start, just to see progress, q to quit.
terminate called after throwing an instance of ‘dw::core::OutOfBoundsException’
what(): span: access out of bounds
terminate called recursively
Segmentation fault

online app output
I also get an out of bounds error message:

[18-09-2024 08:13:25] Driveworks exception thrown: span: access out of bounds

However, I handle it within the sensor manager loop and the app runs and receives the messages with the 4.5sec gap in timestamp I experience.

The gps I’m using is a ublox f9p.

Regarding your latest remark: “I see camera sensor timestamps have delay.”
Does it mean the data is queued and is 4.5sec older or that the sensor manager assigns old timestamps to new data? I tried introducing 5sec sleep before calling dwSensorManager_start but that’s no use since it only really starts the cameras after calling dwSensorManager_start …
The tool you’re suggesting may be helpful for offline post processing but I need a solution for online usage. What do you suggest other users? I imagine I’m not the first to encounter this issue from what you describe.

Thanks,
Alon

SivaRamaKrishnaNV · September 18, 2024, 9:21am

No. Each sensor has different intialization time. When recorder starts, each sensor recording starts at different time. So initial timestamps could be different.

How about discarding the first few gps frames based on the initial delay you noticed ?

al.es · September 19, 2024, 9:31am

How about discarding the first few gps frames based on the initial delay you noticed ?

I can’t count on that since this delay may vary.
Is there a way to make the sensor manager ignore the timestamp of a sensor and just stream the events it produces when they’re available regardless of their timestamp?

SivaRamaKrishnaNV · September 19, 2024, 10:09am

Dear @al.es,
Did you check the sensormanager workflow at DriveWorks SDK Reference: Sensor Manager Workflow
dwSensorManager_acquireNextEvent gets next available sensor event. Does it not serve the purpose?

You can ignore GPS events till camera frame is not noticed.

system · October 17, 2024, 12:08pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Driveworks Entron camera queue timeout DRIVE AGX Orin General driveworks	12	83	October 17, 2024
Driveworks sample camera emulation with AR0231 failed DRIVE AGX Xavier General driveworks-cameras	9	1041	October 5, 2023
Driverworks 5.10 on Orin 6.0.6, gives error of NOTIF_WARN_ICP_FRAME_DROP while recording raw videos using recording tool DRIVE AGX Orin General driveworks-cameras	10	1131	May 26, 2023
Unable to run recorder-qtgui on Drive AGX Orin DRIVE AGX Orin General driveworks-recording	38	96	October 3, 2024
Disable ISP(Not use .nito) in sample_camera demo DRIVE AGX Orin General driveworks	16	673	June 26, 2024
Failure in Dumping Registers from Sensor: AR0231 DRIVE AGX Xavier General driveos-nvmedia	20	1231	August 24, 2023
./sample_cameraserver_client running failed. Error is as follows: Driveworks exception thrown: DW_FAILURE: Camera getImage: failed for DW_CA DRIVE AGX Orin General driveworks-cameras	10	960	September 12, 2022
Driveworks sample_camera throws error DRIVE AGX Orin General driveworks-cameras	12	782	October 5, 2023
dwImage_getTimestamp return delay by 2 minutes timestamp DRIVE AGX Orin General driveworks	25	178	January 3, 2025
Driveworks sample_camera_usb failed DRIVE AGX Orin General driveworks-cameras	17	1009	June 22, 2023

Timestamps Issue with DriveWorks Sensor Manager (GMSL Cameras and GPS)

Related topics