Ouster OS2-128 issue

zyczkowski · March 16, 2021, 12:58pm

Hi, when I try to see data from ouster OS2-128 lidar with sample_lidar_replay Nvidia drive AGX has a problem with showing lidars data. Lidar works correctly when connected using the ouster studio or ROS application. Params we have checked with a lidars web interface and are correct.

I send a log from trying to run lidar. Thanks for any help :)

./sample_lidar_replay --protocol=lidar.socket --params=device=OUSTER_OS2_128,ip=192.168.1.12,port=45065,scan-frequency=10,hres=512,dip=192.168.1.10,force-set=true
Starting my sample application…
[05-03-2021 21:00:01] Platform: Detected DDPX - Tegra B
[05-03-2021 21:00:01] TimeSource: monotonic epoch time offset is 1614969515189446
[05-03-2021 21:00:01] TimeSource: Could not detect valid PTP time source at nvpps. Fallback to eth0
[05-03-2021 21:00:01] TimeSource Eth: Lost PTP time synchronizaton. Synchronized time will not be available from this timesource.
[05-03-2021 21:00:01] TimeSource: Could not detect valid PTP time source at ‘eth0’. Fallback to CLOCK_MONOTONIC.
[05-03-2021 21:00:01] Platform: number of GPU devices detected 1
[05-03-2021 21:00:01] Platform: currently selected GPU device integrated ID 0
[05-03-2021 21:00:01] Context::getDataPathFromSelfLocation DATA_ROOT found at: /usr/local/driveworks-3.5/data
[05-03-2021 21:00:01] SDK: No resources(.pak) mounted, some modules will not function properly
[05-03-2021 21:00:01] SDK: Create NvMediaDevice
[05-03-2021 21:00:01] SDK: Create NvMedia2D
[05-03-2021 21:00:01] SDK: use EGL display as provided
[05-03-2021 21:00:01] TimeSource: monotonic epoch time offset is 1614969515189446
[05-03-2021 21:00:01] TimeSource: Could not detect valid PTP time source at nvpps. Fallback to eth0
[05-03-2021 21:00:01] TimeSource Eth: Lost PTP time synchronizaton. Synchronized time will not be available from this timesource.
[05-03-2021 21:00:01] TimeSource: Could not detect valid PTP time source at ‘eth0’. Fallback to CLOCK_MONOTONIC.
[05-03-2021 21:00:01] Initialize DriveWorks SDK v3.5.75
[05-03-2021 21:00:01] Release build with GNU 7.3.1 from heads/buildbrain-branch-0-gc61a9a35bd0 against Drive PDK v5.2.0.0
[05-03-2021 21:00:01] SensorFactory::createSensor() → lidar.socket, device=OUSTER_OS2_128,ip=192.168.1.12,port=45065,scan-frequency=10,hres=512,dip=192.168.1.10,force-set=true
[05-03-2021 21:00:01] LidarSocket::getParameters, return-mode not specified. Using default value DUAL
[05-03-2021 21:00:01] hfov-start is empty, default value 0 will be used
[05-03-2021 21:00:01] hfov-end is empty, default value 360 will be used
[05-03-2021 21:00:01] Destination IP address, Horizontal Resolution is used only by OUSTER Lidar
[05-03-2021 21:00:01] Return Mode is used only by Velodyne Lidar
[05-03-2021 21:00:01] Horizontal FOV is used only by Ouster and Velodyne Lidars
[05-03-2021 21:00:01] , Device magic number = 3235826430
[05-03-2021 21:00:01] initializeOuster: Current get_config_param active udp_ip is "“192.168.1.200"”, requested setting is “192.168.1.10” !!!
[05-03-2021 21:00:01] initializeOuster: Current get_config_param active azimuth_window is “[0, 360000]”, requested setting is “[0, 36000]” !!!
[05-03-2021 21:00:01] initializeOuster: Current get_config_param active phase_lock_enable is “false”, requested setting is “true” !!!
[05-03-2021 21:00:01] initializeOuster: Current get_config_param active phase_lock_offset is “0”, requested setting is “270000” !!!
[05-03-2021 21:00:01] initializeOuster: command "reinitialize
" failed with "error: timestamp_mode TIME_FROM_INTERNAL_OSC incompatible with p
[05-03-2021 21:00:01] initializeOuster: command "write_config_txt
" failed with "hase_lock_enable. An external time sync source is required, see
[05-03-2021 21:00:01] Json parse error [json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid literal; last read: ‘th’. Retrying…
[05-03-2021 21:00:02] Json parse error [json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - invalid literal; last read: ‘w’. Retrying…
[05-03-2021 21:00:03] Sensor RUNNING
[05-03-2021 21:00:03] Serial number = 992029000682
Firmware version = ousteros-image-prod-aries-v2.0.0+20201124065024
[05-03-2021 21:00:03] std::exception thrown: [json.exception.type_error.302] type must be array, but is null
terminate called after throwing an instance of ‘std::runtime_error’
what(): [2021-03-05 21:00:03] DW Error DW_FAILURE executing DW function:
dwSAL_createSensor(&m_lidarSensor, params, m_sal)
at /dvs/git/dirty/gitlab-master_av/dw/sdk/samples/sensors/lidar/lidar_replay/main.cpp:124
Aborted (core dumped)

Software Version
DRIVE OS Linux 5.2.0 and DriveWorks 3.5

Target Operating System
Linux

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)

SDK Manager Version
1.4.1.7402

Host Machine Version
native Ubuntu 18.04

VickNV · March 16, 2021, 2:42pm

Hi @zyczkowski ,

Please check if any information in Ouster OS2-128 Logging Issue helps on your problem. Thanks.

zyczkowski · March 16, 2021, 3:53pm

Yes, We have already seen this post but it doesn’t help.
Thanks

VickNV · March 16, 2021, 8:47pm

Please refer to the related topic’s command and messages to try on Xavier A first and see if can resolve your abnormal messages.

Als, I saw you have the same firmware as the topic.

SivaRamaKrishnaNV · March 17, 2021, 5:44am

Dear @zyczkowski ,
Have you checked with ouster for ousteros-image-prod-aries-v1.14.0-beta.1+ci+git+nvidia@850828ff2c+dev_fw_PR-793-20200128044023-staging FW vesrion?

pallav.bhalla · March 17, 2021, 10:19am

We are also facing similar issues as Ouster Logging issue. Where can we find ousteros-image-prod-aries-v1.14.0-beta.1+ci+git+nvidia@850828ff2c+dev_fw_PR-793-20200128044023-staging LIDAR firmware?

SivaRamaKrishnaNV · March 17, 2021, 10:27am

Dear @pallav.bhalla ,
Could you check with OUSTER for ousteros-image-prod-aries-v1.14.0-beta.1+ci+git+nvidia@850828ff2c+dev_fw_PR-793-20200128044023-staging ?

SivaRamaKrishnaNV · March 18, 2021, 4:48pm

Dear @pallav.bhalla and @zyczkowski
Could you manually set the configuration of the sensor, drop force-set=true from command line params and update the FW version to ousteros-image-prod-aries-v1.14.0-beta.1+ci+git+nvidia@850828ff2c+dev_fw_PR-793-20200128044023-staging from OUSTER . Let us know the update?

pallav.bhalla · March 18, 2021, 6:59pm

@SivaRamaKrishnaNV,
I contacted OUSTER support for ousteros-image-prod-aries-v1.14.0-beta.1+ci+git+nvidia@850828ff2c+dev_fw_PR-793-20200128044023-staging firmware image. Upon flashing the firmware, I can visualize the LIDAR data. Although, the prompts of UDP packets dropping are still there.

I am attaching a screenshot for reference with the lidar_mode set as:
1024x10:

SivaRamaKrishnaNV · March 22, 2021, 5:33am

Dear @pallav.bhalla ,
So even after using the ousteros-image-prod-aries-v1.14.0-beta.1+ci+git+nvidia@850828ff2c+dev_fw_PR-793-20200128044023-staging you notice packet drop issue? Have you checked watch -n 1 'ethtool -S eth0 | grep fifo' ?

Could you share the complete used command and log to verify ?

daniel.vadranapu1 · March 22, 2021, 10:56am

watch -n 1 'ethtool -S eth0 | grep fifo' gives the following output:
mmc_rx_fifo_overflow: 0

Log:
./sample_lidar_replay --protocol=lidar.socket --params=device=OUSTER_OS2_128,ip=192.168.0.238,dip=192.168.0.225,port=7502,scan-frequency=10,hres=1024,force-set=true --output-dir=/home/nvidia/Desktop/lidar
Starting my sample application…
[22-03-2021 16:18:32] Platform: Detected DDPX - Tegra B
[22-03-2021 16:18:32] TimeSource: monotonic epoch time offset is 1616390059079386
[22-03-2021 16:18:32] TimeSource: Could not detect valid PTP time source at nvpps. Fallback to eth0
[22-03-2021 16:18:32] TimeSource Eth: Lost PTP time synchronizaton. Synchronized time will not be available from this timesource.
[22-03-2021 16:18:32] TimeSource: Could not detect valid PTP time source at ‘eth0’. Fallback to CLOCK_MONOTONIC.
[22-03-2021 16:18:32] Platform: number of GPU devices detected 2
[22-03-2021 16:18:32] Platform: currently selected GPU device discrete ID 0
[22-03-2021 16:18:32] Context::getDataPathFromSelfLocation DATA_ROOT found at: /usr/local/driveworks-3.5/data
[22-03-2021 16:18:32] SDK: No resources(.pak) mounted, some modules will not function properly
[22-03-2021 16:18:32] SDK: Create NvMediaDevice
[22-03-2021 16:18:32] SDK: Create NvMedia2D
[22-03-2021 16:18:32] SDK: use EGL display as provided
[22-03-2021 16:18:32] TimeSource: monotonic epoch time offset is 1616390059079386
[22-03-2021 16:18:32] TimeSource: Could not detect valid PTP time source at nvpps. Fallback to eth0
[22-03-2021 16:18:32] TimeSource Eth: Lost PTP time synchronizaton. Synchronized time will not be available from this timesource.
[22-03-2021 16:18:32] TimeSource: Could not detect valid PTP time source at ‘eth0’. Fallback to CLOCK_MONOTONIC.
[22-03-2021 16:18:32] Initialize DriveWorks SDK v3.5.75
[22-03-2021 16:18:32] Release build with GNU 7.3.1 from heads/buildbrain-branch-0-gc61a9a35bd0 against Drive PDK v5.2.0.0
[22-03-2021 16:18:32] SensorFactory::createSensor() → lidar.socket, device=OUSTER_OS2_128,ip=192.168.0.238,dip=192.168.0.225,port=7502,scan-frequency=10,hres=1024,force-set=true
[22-03-2021 16:18:32] LidarSocket::getParameters, return-mode not specified. Using default value DUAL
[22-03-2021 16:18:32] hfov-start is empty, default value 0 will be used
[22-03-2021 16:18:32] hfov-end is empty, default value 360 will be used
[22-03-2021 16:18:32] Destination IP address, Horizontal Resolution is used only by OUSTER Lidar
[22-03-2021 16:18:32] Return Mode is used only by Velodyne Lidar
[22-03-2021 16:18:32] Horizontal FOV is used only by Ouster and Velodyne Lidars
[22-03-2021 16:18:32] , Device magic number = 3235826430
[22-03-2021 16:18:32] initializeOuster: Current get_lidar_mode is “2048x10”, requested setting is “1024x10” !!!
[22-03-2021 16:18:32] initializeOuster: Current get_config_param active udp_port_lidar is “57779”, requested setting is “7502” !!!
[22-03-2021 16:18:32] Sensor INITIALIZING
[22-03-2021 16:18:32] Serial number = 992046000375
Firmware version = ousteros-image-prod-aries-v1.14.0-beta.12+ci+git+nvidia@aca4b1ca66-20200731174359-staging
[22-03-2021 16:18:33] Sensor INITIALIZING
[22-03-2021 16:18:33] Serial number = 992046000375
Firmware version = ousteros-image-prod-aries-v1.14.0-beta.12+ci+git+nvidia@aca4b1ca66-20200731174359-staging
[22-03-2021 16:18:34] Sensor INITIALIZING
[22-03-2021 16:18:34] Serial number = 992046000375
Firmware version = ousteros-image-prod-aries-v1.14.0-beta.12+ci+git+nvidia@aca4b1ca66-20200731174359-staging
[22-03-2021 16:18:35] Sensor INITIALIZING
[22-03-2021 16:18:35] Serial number = 992046000375
Firmware version = ousteros-image-prod-aries-v1.14.0-beta.12+ci+git+nvidia@aca4b1ca66-20200731174359-staging
[22-03-2021 16:18:36] Sensor INITIALIZING
[22-03-2021 16:18:36] Serial number = 992046000375
Firmware version = ousteros-image-prod-aries-v1.14.0-beta.12+ci+git+nvidia@aca4b1ca66-20200731174359-staging
[22-03-2021 16:18:41] Sensor INITIALIZING
[22-03-2021 16:18:41] Serial number = 992046000375
Firmware version = ousteros-image-prod-aries-v1.14.0-beta.12+ci+git+nvidia@aca4b1ca66-20200731174359-staging
[22-03-2021 16:18:42] Sensor RUNNING
[22-03-2021 16:18:42] Serial number = 992046000375
Firmware version = ousteros-image-prod-aries-v1.14.0-beta.12+ci+git+nvidia@aca4b1ca66-20200731174359-staging
[22-03-2021 16:18:42] LidarSocket::LidarSocket, connected to 192.168.0.238:7502
[22-03-2021 16:18:42] Initialize DriveWorks VisualizationSDK v3.5.75
[22-03-2021 16:18:42] Initialize DriveWorksGL SDK v3.5.75
[22-03-2021 16:18:42] LidarUdp: detected a gap in sensor timestamps: 643563 us, assuming 3317 packets dropped
[22-03-2021 16:18:42] LidarUdp: detected a gap in sensor timestamps: 45127 us, assuming 232 packets dropped
[22-03-2021 16:19:40] LidarUdp: detected a gap in sensor timestamps: 18578 us, assuming 95 packets dropped
[22-03-2021 16:19:46] LidarUdp: detected a gap in sensor timestamps: 110948 us, assuming 571 packets dropped
[22-03-2021 16:19:46] LidarUdp: detected a gap in sensor timestamps: 132623 us, assuming 683 packets dropped
[22-03-2021 16:20:14] LidarUdp: detected a gap in sensor timestamps: 56218 us, assuming 289 packets dropped
[22-03-2021 16:20:54] LidarUdp: detected a gap in sensor timestamps: 23258 us, assuming 119 packets dropped
[22-03-2021 16:20:54] LidarUdp: detected a gap in sensor timestamps: 145703 us, assuming 751 packets dropped
[22-03-2021 16:21:00] LidarUdp: detected a gap in sensor timestamps: 112297 us, assuming 578 packets dropped
[22-03-2021 16:21:06] LidarUdp: detected a gap in sensor timestamps: 34963 us, assuming 180 packets dropped
[22-03-2021 16:21:33] LidarUdp: detected a gap in sensor timestamps: 117063 us, assuming 603 packets dropped
[22-03-2021 16:22:07] LidarUdp: detected a gap in sensor timestamps: 62295 us, assuming 321 packets dropped
[22-03-2021 16:22:41] LidarUdp: detected a gap in sensor timestamps: 190996 us, assuming 984 packets dropped
[22-03-2021 16:22:52] LidarUdp: detected a gap in sensor timestamps: 118540 us, assuming 611 packets dropped
[22-03-2021 16:23:15] LidarUdp: detected a gap in sensor timestamps: 135169 us, assuming 696 packets dropped

get_config_txt:
{“udp_ip”: “192.168.0.225”, “udp_port_lidar”: 7502, “udp_port_imu”: 33972, “timestamp_mode”: “TIME_FROM_PTP_1588”, “sync_pulse_in_polarity”: “ACTIVE_HIGH”, “nmea_in_polarity”: “ACTIVE_HIGH”, “nmea_ignore_valid_char”: 0, “nmea_baud_rate”: “BAUD_9600”, “nmea_leap_seconds”: 0, “multipurpose_io_mode”: “OFF”, “sync_pulse_out_polarity”: “ACTIVE_HIGH”, “sync_pulse_out_frequency”: 1, “sync_pulse_out_angle”: 360, “sync_pulse_out_pulse_width”: 10, “auto_start_flag”: 1, “lidar_mode”: “1024x10”, “azimuth_window”: [0, 36000], “phase_lock_enable”: true, “phase_lock_offset”: 270000}

SivaRamaKrishnaNV · March 22, 2021, 12:34pm

Dear @daniel.vadranapu1,
Could you remove force-set=true from command line parameters and confirm if you notice the same output log.

SivaRamaKrishnaNV · March 23, 2021, 7:58am

Dear @daniel.vadranapu1,
Could you share cat /proc/interrupts | grep eth0.rx

pallav.bhalla · March 23, 2021, 8:10am

@SivaRamaKrishnaNV, sorry for replying from multiple accounts.
Here is the cat /proc/interrupts | grep eth0.rx output:
etho_interrupt

zyczkowski · March 23, 2021, 8:38am

Dear @VickNV

We’ve downgrade firmware from 2.0.0 to ousteros-image-prod-aries-v1.14.0-beta.1+ci+git+nvidia@850828ff2c+dev_fw_PR-793-20200128044023-staging and now we get only “sensor error” which didn’t occurs with a newer software:

./sample_lidar_replay --protocol=lidar.socket --params=device=OUSTER_OS2_128,ip=192.168.1.12,port=7502,scan-frequency=10,hres=1024,dip=192.168.1.10
Starting my sample application…
[06-03-2021 00:04:31] Platform: Detected DDPX - Tegra B
[06-03-2021 00:04:31] TimeSource: monotonic epoch time offset is 1614977981618627
[06-03-2021 00:04:31] TimeSource: Could not detect valid PTP time source at nvpps. Fallback to eth0
[06-03-2021 00:04:31] TimeSource Eth: Lost PTP time synchronizaton. Synchronized time will not be available from this timesource.
[06-03-2021 00:04:31] TimeSource: Could not detect valid PTP time source at ‘eth0’. Fallback to CLOCK_MONOTONIC.
[06-03-2021 00:04:31] Platform: number of GPU devices detected 1
[06-03-2021 00:04:31] Platform: currently selected GPU device integrated ID 0
[06-03-2021 00:04:31] Context::getDataPathFromSelfLocation DATA_ROOT found at: /usr/local/driveworks-3.5/data
[06-03-2021 00:04:31] SDK: No resources(.pak) mounted, some modules will not function properly
[06-03-2021 00:04:31] SDK: Create NvMediaDevice
[06-03-2021 00:04:31] SDK: Create NvMedia2D
[06-03-2021 00:04:31] SDK: use EGL display as provided
[06-03-2021 00:04:31] TimeSource: monotonic epoch time offset is 1614977981618627
[06-03-2021 00:04:31] TimeSource: Could not detect valid PTP time source at nvpps. Fallback to eth0
[06-03-2021 00:04:31] TimeSource Eth: Lost PTP time synchronizaton. Synchronized time will not be available from this timesource.
[06-03-2021 00:04:31] TimeSource: Could not detect valid PTP time source at ‘eth0’. Fallback to CLOCK_MONOTONIC.
[06-03-2021 00:04:31] Initialize DriveWorks SDK v3.5.75
[06-03-2021 00:04:31] Release build with GNU 7.3.1 from heads/buildbrain-branch-0-gc61a9a35bd0 against Drive PDK v5.2.0.0
[06-03-2021 00:04:31] SensorFactory::createSensor() → lidar.socket, device=OUSTER_OS2_128,ip=192.168.1.12,port=7502,scan-frequency=10,hres=1024,dip=192.168.1.10
[06-03-2021 00:04:31] LidarSocket::getParameters, return-mode not specified. Using default value DUAL
[06-03-2021 00:04:31] hfov-start is empty, default value 0 will be used
[06-03-2021 00:04:31] hfov-end is empty, default value 360 will be used
[06-03-2021 00:04:31] Destination IP address, Horizontal Resolution is used only by OUSTER Lidar
[06-03-2021 00:04:31] Return Mode is used only by Velodyne Lidar
[06-03-2021 00:04:31] Horizontal FOV is used only by Ouster and Velodyne Lidars
[06-03-2021 00:04:31] , Device magic number = 3235826430
[06-03-2021 00:04:31] Sensor ERROR
[06-03-2021 00:04:31] Serial number = 992029000682
Firmware version = ousteros-image-prod-aries-v1.14.0-beta.12+ci+git+nvidia@aca4b1ca66-20200731174359-staging
[06-03-2021 00:04:32] Sensor ERROR
[06-03-2021 00:04:32] Serial number = 992029000682
Firmware version = ousteros-image-prod-aries-v1.14.0-beta.12+ci+git+nvidia@aca4b1ca66-20200731174359-staging
[06-03-2021 00:04:33] Sensor ERROR
[06-03-2021 00:04:33] Serial number = 992029000682
Firmware version = ousteros-image-prod-aries-v1.14.0-beta.12+ci+git+nvidia@aca4b1ca66-20200731174359-staging
[06-03-2021 00:04:34] Sensor ERROR
[06-03-2021 00:04:34] Serial number = 992029000682
Firmware version = ousteros-image-prod-aries-v1.14.0-beta.12+ci+git+nvidia@aca4b1ca66-20200731174359-staging
[06-03-2021 00:04:35] Sensor ERROR
[06-03-2021 00:04:35] Serial number = 992029000682
Firmware version = ousteros-image-prod-aries-v1.14.0-beta.12+ci+git+nvidia@aca4b1ca66-20200731174359-staging
[06-03-2021 00:04:36] Sensor ERROR
[06-03-2021 00:04:36] Serial number = 992029000682
Firmware version = ousteros-image-prod-aries-v1.14.0-beta.12+ci+git+nvidia@aca4b1ca66-20200731174359-staging
[06-03-2021 00:04:37] Sensor ERROR
[06-03-2021 00:04:37] Serial number = 992029000682
Firmware version = ousteros-image-prod-aries-v1.14.0-beta.12+ci+git+nvidia@aca4b1ca66-20200731174359-staging
[06-03-2021 00:04:38] Sensor ERROR
[06-03-2021 00:04:38] Serial number = 992029000682
Firmware version = ousteros-image-prod-aries-v1.14.0-beta.12+ci+git+nvidia@aca4b1ca66-20200731174359-staging

pallav.bhalla · March 23, 2021, 8:50am

@zyczkowski, could you please share the lidar sensor configuration for reference?

SivaRamaKrishnaNV · March 23, 2021, 9:14am

Dear @zyczkowski ,
Sensor ERROR indicate sensor is not in running mode. Could you please check if lidar state?

SivaRamaKrishnaNV · March 25, 2021, 1:59am

Dear @zyczkowski,
Is there any update?

SivaRamaKrishnaNV · March 25, 2021, 2:09am

Dear @pallav.bhalla,

There could be multiple reasons for packet drop in your case. High system load, destination IP being set to broadcast or low kernel buffer size

Please try below things and let us know if helped.

Check setting dip to Tegra IP
Increase buffer size like below

sudo sysctl -w net.core.rmem_max=16777216
sudo sysctl -w net.core.rmem_default=16777216

zyczkowski · March 26, 2021, 2:44pm

Hi
I attach lidar config: error_log_ouster (5.3 KB)
We got a response from Ouster, that firmware: ousteros-image-prod-aries-v1.14.0-beta.1+ci+git+nvidia@850828ff2c+dev_fw_PR-793-20200128044023-staging is outdated and has some bugs so they cannot send it to us.

Thanks