Sample_lidar_replay continuously reports gap in sensor timestamps

Autonomous Vehicles DRIVE AGX Thor DRIVE AGX Thor General
1

DRIVE OS Version: 7.0.3

Issue Description: When processing a live lidar stream from a Velodyne VLS128 with sample_lidar_replay the tool continously reports a message indicating a gap in sensor timestamps.

Command used:

/usr/local/driveworks/bin/sample_lidar_replay --protocol=lidar.socket --params=device=VELO_VLS128,ip=192.168.14.218,port=2368,scan-frequency=20,time-smoothing=true --offscreen=2

The sensor is set to 1200 RPM (20 scan/s) and in Dual Return mode. The symptom also occurs in Single Return mode.

The tail of output:

... 

[09-07-2026 16:47:46] Received 10/469 packets
[09-07-2026 16:47:46] Received 8/469 packets
[09-07-2026 16:47:46] Received 9/469 packets
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 293 us, assuming 5 packets dropped
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903628577 (current) vs. 2903628752 (previous)
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903628518 (current) vs. 2903628694 (previous)
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 292 us, assuming 5 packets dropped
[09-07-2026 16:47:46] Received 9/469 packets
[09-07-2026 16:47:46] Received 9/469 packets
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 292 us, assuming 5 packets dropped
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903629629 (current) vs. 2903629804 (previous)
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903629571 (current) vs. 2903629746 (previous)
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 292 us, assuming 5 packets dropped
[09-07-2026 16:47:46] Received 9/469 packets
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 293 us, assuming 5 packets dropped
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903630155 (current) vs. 2903630331 (previous)
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903630096 (current) vs. 2903630273 (previous)
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 294 us, assuming 5 packets dropped
[09-07-2026 16:47:46] Received 1/469 packets
[09-07-2026 16:47:46] Received 9/469 packets
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 291 us, assuming 5 packets dropped
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903630741 (current) vs. 2903630915 (previous)
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903630682 (current) vs. 2903630857 (previous)
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 292 us, assuming 5 packets dropped
[09-07-2026 16:47:46] Received 9/469 packets
[09-07-2026 16:47:46] Received 9/469 packets
[09-07-2026 16:47:46] Received 9/469 packets
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 293 us, assuming 5 packets dropped
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903632319 (current) vs. 2903632495 (previous)
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903632261 (current) vs. 2903632437 (previous)
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 292 us, assuming 5 packets dropped
[09-07-2026 16:47:46] Received 9/469 packets
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 292 us, assuming 5 packets dropped
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903632845 (current) vs. 2903633020 (previous)
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903632787 (current) vs. 2903632962 (previous)
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 292 us, assuming 5 packets dropped
[09-07-2026 16:47:46] Received 9/469 packets
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 293 us, assuming 5 packets dropped
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903633371 (current) vs. 2903633547 (previous)
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903633313 (current) vs. 2903633488 (previous)
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 292 us, assuming 5 packets dropped
[09-07-2026 16:47:46] Received 10/469 packets
[09-07-2026 16:47:46] Received 9/469 packets
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 293 us, assuming 5 packets dropped
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903634482 (current) vs. 2903634658 (previous)
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903634423 (current) vs. 2903634599 (previous)
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 294 us, assuming 5 packets dropped
[09-07-2026 16:47:46] Received 9/469 packets
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 293 us, assuming 5 packets dropped
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903635009 (current) vs. 2903635185 (previous)
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903634951 (current) vs. 2903635126 (previous)
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 292 us, assuming 5 packets dropped
[09-07-2026 16:47:46] Received 9/469 packets
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 292 us, assuming 5 packets dropped
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903635535 (current) vs. 2903635710 (previous)
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903635477 (current) vs. 2903635651 (previous)
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 292 us, assuming 5 packets dropped
[09-07-2026 16:47:46] Received 10/469 packets
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 293 us, assuming 5 packets dropped
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903636119 (current) vs. 2903636295 (previous)
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903636061 (current) vs. 2903636236 (previous)
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 292 us, assuming 5 packets dropped
[09-07-2026 16:47:46] Received 8/469 packets
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 291 us, assuming 5 packets dropped
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903636646 (current) vs. 2903636821 (previous)
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903636588 (current) vs. 2903636763 (previous)
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 292 us, assuming 5 packets dropped
[09-07-2026 16:47:46] Received 9/469 packets
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 294 us, assuming 5 packets dropped
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903637115 (current) vs. 2903637291 (previous)
[09-07-2026 16:47:46] LidarUdp: detected UDP reordering: 2903637056 (current) vs. 2903637232 (previous)
[09-07-2026 16:47:46] LidarUdp: detected a gap in sensor timestamps: 293 us, assuming 5 packets dropped
[DW][INFO] Process Successfully.
[DW][INFO] Timing results:
Thread main:
-onProcess                CPU:   nanus, std= 364       | GPU:   nanus, std= 507       | samples=23265
-onRender                 CPU:   nanus, std=  24       | GPU:   nanus, std=  23       | samples=23264

[09-07-2026 16:47:46] Driveworks VisualizationSDK released
[09-07-2026 16:47:46] SensorFactory::releaseSensor() - lidar.socket, device=VELO_VLS128,ip=192.168.14.218,port=2368,scan-frequency=20,time-smoothing=true
[09-07-2026 16:47:47] TimeSensor::addData() Buffer full, draining data
[09-07-2026 16:47:49] EndpointNVPPS: stopped on /dev/nvpps0
[09-07-2026 16:47:49] [09-07-2026 16:47:49] Releasing Driveworks SDK Context
[09-07-2026 16:47:49] DriveworksGL SDK released
nvidia@tegra-ubuntu:~$

Does this actually indicate some problem with the flow of network packets?

A network capture seems to suggest that all of the packets are arriving to the Drive AGX Thor in order and without any packet loss.

I note that the CPU utilization for sample_lidar_replay seems to consume about 90% of one core.

Is the a real problem with the sensor or network or a known issue with the lidar.socket reader?

5

Dear @david.cattley ,
Could you check with ethtool, tcpdump if any packets are dropped?

6

packets are not being dropped nor are they being re-ordered (there is no router in the path anyway).

Those messages

only show up in live play using sample_lidar_replay and do not show up in the output of recorder or in the output of sample_lidar_replay when it replays a recording.

7

I have a tool that monitors the lidar packet stream using libpcap on ingress to the DriveAGX Thor. This tools “sees” every lidar packet that comes in the ethernet interface at L2 and so it would detect any issues on the wire (e.g. if the sensor, switch, wires, mgbe driver, whatever caused a packet stream error).

The tool does full lidar packet validation and analysis of the timestamp and azimuth data and is capable of detecting missing, duplicated, or out-of-order packets.

It does not detect any such thing while sample_lidar is running and spewing out the repeated

[20-07-2026 14:53:09] Received 9/469 packets
[20-07-2026 14:53:09] LidarUdp: detected a gap in sensor timestamps: 291 us, assuming 5 packets dropped
[20-07-2026 14:53:09] LidarUdp: detected UDP reordering: 3226485382 (current) vs. 3226485556 (previous)
[20-07-2026 14:53:09] LidarUdp: detected UDP reordering: 3226485323 (current) vs. 3226485498 (previous)
[20-07-2026 14:53:09] LidarUdp: detected a gap in sensor timestamps: 292 us, assuming 5 packets dropped
[20-07-2026 14:53:09] Received 9/469 packets
[20-07-2026 14:53:09] LidarUdp: detected a gap in sensor timestamps: 292 us, assuming 5 packets dropped
[20-07-2026 14:53:09] LidarUdp: detected UDP reordering: 3226485907 (current) vs. 3226486082 (previous)
[20-07-2026 14:53:09] LidarUdp: detected UDP reordering: 3226485848 (current) vs. 3226486024 (previous)
[20-07-2026 14:53:09] LidarUdp: detected a gap in sensor timestamps: 293 us, assuming 5 packets dropped
[20-07-2026 14:53:09] Received 1/469 packets
[20-07-2026 14:53:09] Received 9/469 packets
[20-07-2026 14:53:09] LidarUdp: detected a gap in sensor timestamps: 292 us, assuming 5 packets dropped
[20-07-2026 14:53:09] LidarUdp: detected UDP reordering: 3226486491 (current) vs. 3226486666 (previous)
[20-07-2026 14:53:09] LidarUdp: detected UDP reordering: 3226486432 (current) vs. 3226486608 (previous)
[20-07-2026 14:53:09] LidarUdp: detected a gap in sensor timestamps: 293 us, assuming 5 packets dropped
[20-07-2026 14:53:09] Received 9/469 packets
[20-07-2026 14:53:09] LidarUdp: detected a gap in sensor timestamps: 293 us, assuming 5 packets dropped
[20-07-2026 14:53:09] LidarUdp: detected UDP reordering: 3226487017 (current) vs. 3226487193 (previous)
[20-07-2026 14:53:09] LidarUdp: detected UDP reordering: 3226486958 (current) vs. 3226487134 (previous)
[20-07-2026 14:53:09] LidarUdp: detected a gap in sensor timestamps: 293 us, assuming 5 packets dropped
[20-07-2026 14:53:09] Received 9/469 packets
[20-07-2026 14:53:09] LidarUdp: detected a gap in sensor timestamps: 293 us, assuming 5 packets dropped
[20-07-2026 14:53:09] LidarUdp: detected UDP reordering: 3226487544 (current) vs. 3226487720 (previous)
[DW][INFO] Process Successfully.
[DW][INFO] Timing results:
Thread main:
-onProcess                CPU: 48544us, std=1829       | GPU: 49983us, std=1926       | samples=682
-onRender                 CPU:    81us, std= 494       | GPU:     2us, std= 501       | samples=681

[20-07-2026 14:53:09] Driveworks VisualizationSDK released
[20-07-2026 14:53:09] SensorFactory::releaseSensor() - lidar.socket, device=VELO_VLS128,ip=192.168.14.218,port=2368,scan-frequency=20
[20-07-2026 14:53:11] EndpointNVPPS: stopped on /dev/nvpps0
[20-07-2026 14:53:11] [20-07-2026 14:53:11] Releasing Driveworks SDK Context
[20-07-2026 14:53:11] DriveworksGL SDK released
[DW][INFO] Release Successfully.

the monitor tool output:

[(192.168.14.218) 08379884:046010] 3593618373 59:53.618373 :INFO   : Status
  Source:          192.168.14.218
  Data:
    Packets:         8379884
    Timestamp:       3593618314 (59:53.618314)
    ΔUTC:           +37s -317µs  (n=17103 avg +37s -490µs, stddev 152µs)
    Sensor:          VLS-128
    Returns:         Dual
    Avg interval:    58 µs
    RPM:             1186.44  (avg 1197.52, stddev 12.923)
    Phaselock:       90.15  (avg 90.17, stddev 0.230)
    pkts/s:          18181.8
    points/s:        4654545.5
    bytes/s:         21927272.7
    TOH rollovers:   0
    Errors:          0
    Warnings:        0
  Telemetry:
    Packets:         46010
    Timestamp:       3593607646 (59:53.607646)
    ΔUTC:           +37s -805µs  (n=93 avg +37s -803µs, stddev 54µs)
    GPS PPS:         0 (Absent)
    Thermal:         0 (Ok)
    Temp top:        41°C
    Temp bottom:     40°C
    Shutdown temp:   201°C
    Power-up temp:   127°C
    NMEA:            $GPRMC,145916,V,4212.8231,N,07143.0716,W,,,200726,014.5,W,N*00
      Sentence:        GPRMC
      Checksum:        ok
      Status:          V (void)
    pkts/s:          93
    bytes/s:         47614.4
    TOH rollovers:   0
    Errors:          0
  Capture:
    recv:            8424499
    drop:            0
    ifdrop:          0
    truncated:       0

Note that the NMEA status of V (void) is irrelevant as the sensor is locked to gPTP timesync. The GPS sentence is simply “informational” as the GPS antenna is occluded from seeing the sky in this test rig.

The timestamps from the sensor are synchronized as the too reports the timestamp delta as being on the current leap-seconds of 37s and a small <500us delta (typical).

So there seemingly is something wrong with the data path in DriveWorks that is confusing, dropping, or re-ordering the packets. Or the log messages are just wrong.