I plan to use Nvidia Orin dev kit to control Franka robot FR3. The robot requires Linux OS with PREEMPT_RT. So I have flashed the kit with Jetson Linux 35.3.1 with PREEMPT_RT patched by following the steps in No display with PREEMPT_RT patches.
Then, I connected the kit with the robot FCI box directly via ethernet port. I called ping to check the communication speed. The round-trip time is about 0.67 ms, which is pretty large for real-time control of the robot at 1kHz. In fact, this sometime made the robot halts because more than 20 successive control commands takes more than 1 ms to reach the robot. For comparison, I used a typical x86 computer to ping the robot. It takes only about 0.12 ms.
I have browsed through many related topics and followed their solutions, such as maximize power mode and cpu clock. Some suggested increasing MTU. But none solves my problem. I have looked at ethtool information and found that eth0 has 1000Mb/s bandwidth, which should be sufficient to attain 0.12ms round-trip time for small size packets.
So I would like to ask about the way to reduce the communication time delay through Ethernet port. Especially, the ping round-trip time should be less than 0.12 ms for direct connection as compared to typical x86 desktop. Thank you very much in advance.
Hi,
Could you try to connect to a Ubuntu PC and run iperf3 command? Would like to know if the performance issue happens in this setup. Or only when connecting to the robot device.
I hooked up an Ubuntu PC with Nvidia Orin dev kit and run iperf3. If the PC is the server, the average transfer bit rate is about 942 Mbit/s. On the other hand, if the Orin is the server, the average transfer bit rate falls to 913 Mbit/s.
I also ping between the machines using command
sudo ping 172.31.1.111 -i 0.001 -D -c 10000 -s 1200
Either side would produce rtt min/avg/max/mdev about the same of 0.088/0.649/1.283/0.117 ms. So the problem of slow communication speed with small consecutive packets is not due to the robot device. It is the problem of the Ethernet port of Orin, either hardware or software setup.
I also did an additional experiment by running the above command via using USB-c to Ethernet port adapter on the Nvidia Orin. Surprisingly, the rtt numbers drops to 0.144/0.200/0.777/0.090 ms, which is at least 3 times faster than through the actual Ethernet port. These numbers are the same as when Orin pings to the robot via USB-c to Ethernet port adapter. Still, the PC real Ethernet port pinging to the robot is faster (0.12 ms on average).
I think there must be something wrong with the real Ethernet port of Nvidia Orin. Could you or somebody do the same test as I have described to verify this is not just the defect of my purchase only. Thank you very much.
With ethtool shows the result as 1000T, the performance of 9xxMb/s seems a valid result. Even jeston to jetson direct ethernet connection shall give similar result.
If you want some enhancements, please enable jetson_clocks and adjust the MTU size. Though I think it might only give limited help because the original performance is already normal one.
Orin has no problem in achieving 1000 Mb/s throughput since I guess it sends/receives large-size packets for a second-order interval. That is, a millisecond-order communication overhead (0.6 ms) does not show up. However, the problem I found and that matters my work is that when sending/receiving small-size packets (such as what ping command does, or real-time control command), the communication time delay becomes larger in proportion to the actual data size being transmitted. The ratio is too large in a sense that it makes round-trip time of ping command at least 3 times slower than typical Ubuntu x86 desktop or even a notebook. More surprisingly, a fake Ethernet port via USB-c to Ethernet adapter is faster!!
I already tried jetson_clocks and adjusting the MTU size to like 9000. The result does not change.
Hi,
Please test on Orin developer kit and check if the issue is present. If you can observe it on developer kit, please share the full steps so that we can set up to replicate the issue.
And please try without PREEMPT_RT. To confirm if it occurs only when PREEMPT_RT is
enabled.