Two Velodyne VLS128 Problem

Hello,

I have a problem with connecting two VLS128 and 8 Cameras ( 128 beam lidar from Velodyne ) to Drive AGX Pegasus with dGPUs (E3550) which has Drive 10.0 Software installed

Operating System:

Linux tegra-ubuntu 4.14.102-rt53-tegra #1 SMP PREEMPT RT Fri Sep 20 16:23:45 PDT 2019 aarch64 aarch64 aarch64 GNU/Linux

Aurix Firmware:

SW Version: DRIVE-V5.1.6-E3550-EB-Aurix-With3LSS-ForHyperion-StepA-3.05.04
Complation date: Jun 25 2019, 14:25:36

The detailed explanation of the problem, problem persists both for Tegra A and Tegra B:

  1. When we have 8 cameras plugged into Fakra-A and Fakra-B connectors of AGX Pegasus. I can visualize the cameras without a problem and I can ping Aurix from VLAN 200 ( 10.42.0.146 )

  2. When we have 8 cameras plugged into Fakra-A and Fakra-B and one VLS128 Lidar plugged into 1Gb/S Link 1&2 connector via ethernet dongle Channel A supplied from Nvidia ( shown as the connectors for lidar connections on Drive OS 10.0 documentation ), I can visualize the cameras and also the lidar ( 192.168.1.201, Port: 2368 )

  3. When we have 8 cameras plugged into Fakra-A and Fakra-B and two VLS128 Lidars plugged into 1Gb/S Link 1&2 connector via ethernet dongle Channel A and Channel B supplied from Nvidia ( shown as the connectors for lidar connections on Drive OS 10.0 documentation ), I can’t visualize the cameras but I can visualize the lidars! But there is a huge package loss when we ping lidars and Aurix through 10.42.0.146. We also tried to connecting two lidars to seperate 1Gb/s connectors, Link 1&2 and Link3&4 we also tried to connect lidars to 10Gb/s connectors separately too. The problem persisted.

Here is the error output from the visualizer command:

nvidia@tegra-ubuntu:~/samples$ ./sample_camera_gmsl --tegra-slave=1
[21-09-2019 01:14:18] Platform: Detected DDPX - Tegra B
[21-09-2019 01:14:18] TimeSource: monotonic epoch time offset is 1569027511537396
[21-09-2019 01:14:18] PTP Time is available from NVPPS Driver
[21-09-2019 01:14:18] Platform: number of GPU devices detected 2
[21-09-2019 01:14:18] Platform: currently selected GPU device discrete ID 0
[21-09-2019 01:14:18] SDK: Resource failed to mount from ‘…/./data/’: VirtualFileSystem: Failed to mount ‘…/./data/[.pak]’
[21-09-2019 01:14:18] SDK: Resources mounted from /usr/local/driveworks-2.2/data/
[21-09-2019 01:14:18] SDK: Create NvMediaDevice
[21-09-2019 01:14:18] SDK: use EGL display as provided
[21-09-2019 01:14:18] TimeSource: monotonic epoch time offset is 1569027511537396
[21-09-2019 01:14:18] PTP Time is available from NVPPS Driver
[21-09-2019 01:14:18] Initialize DriveWorks SDK v2.2.3136
[21-09-2019 01:14:18] Release build with GNU 7.3.1 from heads/buildbrain-branch-0-gca7b4b26e65 against Drive PDK v5.1.6.1
[21-09-2019 01:14:18] Initialize DriveWorks VisualizationSDK v2.2.3136
[21-09-2019 01:14:18] Initialize DriveWorksGL SDK v2.2.3136
[21-09-2019 01:14:20] SensorFactory::createSensor() → camera.gmsl, output-format=yuv,fifo-size=3,isp-mode=yuv420-uint8,camera-type=ar0231-rccb-bae-sf3324,camera-group=a,slave=1
[21-09-2019 01:14:20] CameraGMSL: Couldn’t set CameraGMSL setup thread sched params!
[21-09-2019 01:14:20] CameraGMSL: Create NvMediaIPPManager
[21-09-2019 01:14:20] CameraGMSL: required FPS = 30, resolution = 1920x1208
nvmedia isc: GetCameraPowerControlLevel: 936: Camera power control library: NVCCP
receive_from: Timeout Error.
Please check Ethernet ip configuration or
Update Aurix firmware version by the following command

sudo /bin/bash /etc/systemd/scripts/nv_aurix_check_fw.sh -auto_update

nvmedia isc: NvccpCamPowerControl: 544: NVCCP command failed with ret NVCCP_REQ_TIMEOUT
iscRootDevOpen: Failed to request ownership
NvMediaISCRootDeviceCreate: Unable to open root device
nvmedia: ERROR: Init: Failed to create NvMedia ISC root device
[21-09-2019 01:14:25] Driveworks exception thrown: DW_SAL_CANNOT_CREATE_SENSOR: CameraGMSL: cannot connect to camera

terminate called after throwing an instance of ‘std::runtime_error’
what(): [2019-09-21 01:14:25] DW Error DW_SAL_CANNOT_CREATE_SENSOR executing DW function:
dwSAL_createSensor(&m_camera, params, m_sal)
at /dvs/git/dirty/gitlab-master_av/dw/sdk/samples/sensors/camera/camera_gmsl/main.cpp:174
Aborted (core dumped)
nvidia@tegra-ubuntu:~/samples$

Pinging First VLS128:

nvidia@tegra-ubuntu:~/samples$ ping 192.168.1.201
PING 192.168.1.201 (192.168.1.201) 56(84) bytes of data.
64 bytes from 192.168.1.201: icmp_seq=10 ttl=64 time=1.26 ms
64 bytes from 192.168.1.201: icmp_seq=11 ttl=64 time=0.921 ms
64 bytes from 192.168.1.201: icmp_seq=15 ttl=64 time=0.553 ms
^C
— 192.168.1.201 ping statistics —
15 packets transmitted, 3 received, 80% packet loss, time 14256ms
rtt min/avg/max/mdev = 0.553/0.913/1.265/0.290 ms
nvidia@tegra-ubuntu:~/samples$

Pinging Second VLS128:

nvidia@tegra-ubuntu:~/samples$ ping 192.168.1.202
PING 192.168.1.202 (192.168.1.202) 56(84) bytes of data.
64 bytes from 192.168.1.202: icmp_seq=2 ttl=64 time=200 ms
64 bytes from 192.168.1.202: icmp_seq=4 ttl=64 time=0.498 ms
64 bytes from 192.168.1.202: icmp_seq=5 ttl=64 time=157 ms
64 bytes from 192.168.1.202: icmp_seq=7 ttl=64 time=0.460 ms
^C
— 192.168.1.202 ping statistics —
7 packets transmitted, 4 received, 42% packet loss, time 6064ms
rtt min/avg/max/mdev = 0.460/89.921/200.908/90.731 ms
nvidia@tegra-ubuntu:~/samples$

Pinging Aurix:

nvidia@tegra-ubuntu:~/samples$ ping 10.42.0.146
PING 10.42.0.146 (10.42.0.146) 56(84) bytes of data.
64 bytes from 10.42.0.146: icmp_seq=2 ttl=255 time=10.1 ms
64 bytes from 10.42.0.146: icmp_seq=4 ttl=255 time=9.32 ms
64 bytes from 10.42.0.146: icmp_seq=6 ttl=255 time=8.47 ms
64 bytes from 10.42.0.146: icmp_seq=7 ttl=255 time=6.88 ms
64 bytes from 10.42.0.146: icmp_seq=8 ttl=255 time=9.92 ms
^C
— 10.42.0.146 ping statistics —
9 packets transmitted, 5 received, 44% packet loss, time 8046ms
rtt min/avg/max/mdev = 6.883/8.944/10.112/1.178 ms
nvidia@tegra-ubuntu:~/samples$

We can connect two VLP16s without a problem with cameras without a problem in either of the configuration explained above.

Best Regards.

Dear csari3,
We will check reproducing this issue and get back to you.

Dear csari3,
Could you run ./sample_camera_gmsl only on tegra A in the above set up(8 cameras+2lidars) and confirm you see the same error? Make sure no application is using these sensors on Tegra B.

I can confirm that I get the same errors on Tegra A while there are no sample applications running on Tegra B.

Do you want me to shutdown Tegra B completely ?

Dear csari3,
Please restart the machine and share the above logs on Tegra A.

Hello,

Here is the log from Tegra A with the configuration: 8 cameras connected to Fakra A and Fakra B, Lidars are connected to 1Gb/s Link 1&2 Channel A and Channel B

Sample Camera GMSL Log:

nvidia@tegra-ubuntu:/usr/local/driveworks/bin$ ./sample_camera_gmsl
[21-09-2019 00:59:38] Platform: Detected DDPX - Tegra A
[21-09-2019 00:59:38] TimeSource: monotonic epoch time offset is 1569027511484399
[21-09-2019 00:59:38] PTP Time is available from NVPPS Driver
[21-09-2019 00:59:38] Platform: number of GPU devices detected 2
[21-09-2019 00:59:38] Platform: currently selected GPU device discrete ID 0
[21-09-2019 00:59:38] SDK: Resources mounted from …/./data/
[21-09-2019 00:59:38] SDK: Create NvMediaDevice
[21-09-2019 00:59:38] SDK: use EGL display as provided
[21-09-2019 00:59:38] TimeSource: monotonic epoch time offset is 1569027511484399
[21-09-2019 00:59:38] PTP Time is available from NVPPS Driver
[21-09-2019 00:59:38] Initialize DriveWorks SDK v2.2.3136
[21-09-2019 00:59:38] Release build with GNU 7.3.1 from heads/buildbrain-branch-0-gca7b4b26e65 against Drive PDK v5.1.6.1
[21-09-2019 00:59:38] Initialize DriveWorks VisualizationSDK v2.2.3136
[21-09-2019 00:59:38] Initialize DriveWorksGL SDK v2.2.3136
[21-09-2019 00:59:38] SensorFactory::createSensor() → camera.gmsl, output-format=yuv,fifo-size=3,isp-mode=yuv420-uint8,camera-type=ar0231-rccb-bae-sf3324,camera-group=a,slave=0
[21-09-2019 00:59:38] CameraGMSL: Couldn’t set CameraGMSL setup thread sched params!
[21-09-2019 00:59:38] CameraGMSL: Create NvMediaIPPManager
[21-09-2019 00:59:38] CameraGMSL: required FPS = 30, resolution = 1920x1208
nvmedia isc: GetCameraPowerControlLevel: 936: Camera power control library: NVCCP
receive_from: Timeout Error.
Please check Ethernet ip configuration or
Update Aurix firmware version by the following command

sudo /bin/bash /etc/systemd/scripts/nv_aurix_check_fw.sh -auto_update

nvmedia isc: NvccpCamPowerControl: 602: NVCCP command failed with ret NVCCP_REQ_TIMEOUT
iscRootDevOpen: Failed to power on the deserializer
receive_from: Timeout Error.
Please check Ethernet ip configuration or
Update Aurix firmware version by the following command

sudo /bin/bash /etc/systemd/scripts/nv_aurix_check_fw.sh -auto_update

nvmedia isc: NvccpCamPowerControl: 555: NVCCP command failed with ret NVCCP_REQ_TIMEOUT
iscRootDevOpen: Failed to release ownership
NvMediaISCRootDeviceCreate: Unable to open root device
nvmedia: ERROR: Init: Failed to create NvMedia ISC root device
[21-09-2019 00:59:49] Driveworks exception thrown: DW_SAL_CANNOT_CREATE_SENSOR: CameraGMSL: cannot connect to camera

terminate called after throwing an instance of ‘std::runtime_error’
what(): [2019-09-21 00:59:49] DW Error DW_SAL_CANNOT_CREATE_SENSOR executing DW function:
dwSAL_createSensor(&m_camera, params, m_sal)
at /dvs/git/dirty/gitlab-master_av/dw/sdk/samples/sensors/camera/camera_gmsl/main.cpp:174
Aborted (core dumped)

Aurix Ping:

nvidia@tegra-ubuntu:/usr/local/driveworks/bin$ ping 10.42.0.146
PING 10.42.0.146 (10.42.0.146) 56(84) bytes of data.
64 bytes from 10.42.0.146: icmp_seq=2 ttl=255 time=9.13 ms
64 bytes from 10.42.0.146: icmp_seq=7 ttl=255 time=6.68 ms
64 bytes from 10.42.0.146: icmp_seq=9 ttl=255 time=10.8 ms
64 bytes from 10.42.0.146: icmp_seq=11 ttl=255 time=11.0 ms
^C
— 10.42.0.146 ping statistics —
11 packets transmitted, 4 received, 63% packet loss, time 10147ms
rtt min/avg/max/mdev = 6.684/9.434/11.046/1.753 ms

Lidar 1 Ping:

nvidia@tegra-ubuntu:/usr/local/driveworks/bin$ ping 192.168.1.201
PING 192.168.1.201 (192.168.1.201) 56(84) bytes of data.
From 192.168.1.200 icmp_seq=1 Destination Host Unreachable
64 bytes from 192.168.1.201: icmp_seq=8 ttl=64 time=293 ms
64 bytes from 192.168.1.201: icmp_seq=11 ttl=64 time=134 ms
64 bytes from 192.168.1.201: icmp_seq=13 ttl=64 time=1.21 ms
64 bytes from 192.168.1.201: icmp_seq=15 ttl=64 time=0.500 ms
64 bytes from 192.168.1.201: icmp_seq=19 ttl=64 time=7.97 ms
^C
— 192.168.1.201 ping statistics —
19 packets transmitted, 5 received, +1 errors, 73% packet loss, time 18337ms
rtt min/avg/max/mdev = 0.500/87.634/293.582/114.939 ms, pipe 4

Lidar 2 Ping:

nvidia@tegra-ubuntu:/usr/local/driveworks/bin$ ping 192.168.1.202
PING 192.168.1.202 (192.168.1.202) 56(84) bytes of data.
64 bytes from 192.168.1.202: icmp_seq=1 ttl=64 time=2.06 ms
64 bytes from 192.168.1.202: icmp_seq=7 ttl=64 time=0.467 ms
64 bytes from 192.168.1.202: icmp_seq=8 ttl=64 time=438 ms
64 bytes from 192.168.1.202: icmp_seq=9 ttl=64 time=1.34 ms
64 bytes from 192.168.1.202: icmp_seq=10 ttl=64 time=1.14 ms
64 bytes from 192.168.1.202: icmp_seq=11 ttl=64 time=69.2 ms
^C
— 192.168.1.202 ping statistics —
11 packets transmitted, 6 received, 45% packet loss, time 10125ms
rtt min/avg/max/mdev = 0.467/85.489/438.658/159.883 ms

I have also the serial logs from Aurix, Tegra A and Tegra B via serial link ( minicom )

Dear csari3,
You do not see this issue with sample_camera_gmsl when 2 VLP 16 lidars+8 cameras are connected.
You notice issue when you replaces two VLP16 with VLP128 Lidars and you do not see any issue when using 8cameras + 1 Lidar.

Could you check with two VLP32 lidars(if you have) and give feedback.
Could you attach the dmesg logs?
Is the lidar is set to unicast or broadcast? could you change it and check once?
What is the IP and port values for both VLP128 lidars. Please check changing IP and port values

  1. You do not see this issue with sample_camera_gmsl when 2 VLP 16 lidars+8 cameras are connected.

Correct

  1. You notice issue when you replaces two VLP16 with VLP128 Lidars and you do not see any issue when using 8cameras + 1 Lidar.

Correct

  1. Could you check with two VLP32 lidars(if you have) and give feedback.

We don’t have VLP32 Lidars

  1. Is the lidar is set to unicast or broadcast? could you change it and check once?

It is the default setting

  1. What is the IP and port values for both VLP128 lidars. Please check changing IP and port values

First Lidar: 192.168.1.201, Port 2368, Telemetry Port is different from Second Lidar’s
Second Lidar: 192.168.1.202, Port 2369 Telemetry Port is different from First Lidar’s
tegraBboot.txt (70 KB)
tegraAboot.txt (72.8 KB)

Hello,

The problem has been solved changing broadcasting to Host IP. While broadcasting two VLS128 lidars effects the network when we set host ip for Tegra B as 192.168.203, the problem is solved.

Best Regards,