TX2 ethernet high latency

while using ssh and nfs to work on TX2, I find the console command very lagged. Then I pinged another ip of the same LAN from TX2, it shows the latency is around a few hundred milliseconds to over 1 second. This is very high compared to TX1 in the same network (around 2~3 ms)

Anyone ever had the same issue?

The Jetson SDK version is 28.1

Hi xliu,

Is nfs necessary to reproduce this issue? Would this happen when only use ssh?

no need.
just ping any other ip, the round trip time shows it. For TX2, a few hundred ms to 1000ms, for TX1, 2~3ms.
I’ve done it after a fresh reboot, without using ssh or nfs.

Hi xliu,

This is my test result.

TX1
64 bytes from 10.19.106.138: icmp_seq=26 ttl=64 time=0.791 ms
64 bytes from 10.19.106.138: icmp_seq=27 ttl=64 time=0.881 ms
64 bytes from 10.19.106.138: icmp_seq=28 ttl=64 time=1.09 ms
64 bytes from 10.19.106.138: icmp_seq=29 ttl=64 time=1.09 ms
64 bytes from 10.19.106.138: icmp_seq=30 ttl=64 time=0.734 ms
64 bytes from 10.19.106.138: icmp_seq=31 ttl=64 time=0.662 ms
64 bytes from 10.19.106.138: icmp_seq=32 ttl=64 time=0.924 ms

TX2
64 bytes from 10.19.106.138: icmp_seq=26 ttl=64 time=1.74 ms
64 bytes from 10.19.106.138: icmp_seq=27 ttl=64 time=1.89 ms
64 bytes from 10.19.106.138: icmp_seq=28 ttl=64 time=1.66 ms
64 bytes from 10.19.106.138: icmp_seq=29 ttl=64 time=1.79 ms
64 bytes from 10.19.106.138: icmp_seq=30 ttl=64 time=2.24 ms
64 bytes from 10.19.106.138: icmp_seq=31 ttl=64 time=1.34 ms
64 bytes from 10.19.106.138: icmp_seq=32 ttl=64 time=1.73 ms

It does not have abnormal value like 1000ms. Could you check if there is any error message in “dmesg” while pinging.

Does latency improve if you run the “sudo ~/jetson_clocks.sh” script and “sudo nvpmodel -m 0”?

yes, it improves. Now it’s never more than 300ms. What does this mean?

64 bytes from 192.168.31.80: icmp_seq=13 ttl=64 time=246 ms
64 bytes from 192.168.31.80: icmp_seq=14 ttl=64 time=141 ms
64 bytes from 192.168.31.80: icmp_seq=15 ttl=64 time=221 ms
64 bytes from 192.168.31.80: icmp_seq=16 ttl=64 time=249 ms
64 bytes from 192.168.31.80: icmp_seq=17 ttl=64 time=247 ms
64 bytes from 192.168.31.80: icmp_seq=18 ttl=64 time=31.5 ms
64 bytes from 192.168.31.80: icmp_seq=19 ttl=64 time=155 ms
64 bytes from 192.168.31.80: icmp_seq=20 ttl=64 time=5.03 ms
64 bytes from 192.168.31.80: icmp_seq=21 ttl=64 time=3.96 ms
64 bytes from 192.168.31.80: icmp_seq=22 ttl=64 time=194 ms
64 bytes from 192.168.31.80: icmp_seq=23 ttl=64 time=55.1 ms
64 bytes from 192.168.31.80: icmp_seq=24 ttl=64 time=26.3 ms
64 bytes from 192.168.31.80: icmp_seq=25 ttl=64 time=14.0 ms
64 bytes from 192.168.31.80: icmp_seq=26 ttl=64 time=249 ms

Please try these steps separately and see if it is affected by jetson_clock.sh or nvpmodel.

Hi Wayne,
Just did it again with them separately and jointly, I see no improvement. Maybe it was just luck in my last post.

Also, 240 milliseconds ping to a local network node is still terrible.
There’s something going on on your network, or perhaps on your machine, that causes significant degradation.

Here’s pings from my machine in low power mode:

Note the times that are 1,000 times faster than the times you are seeing!

I changed my router with another, then it comes back to normal (1~2ms).
The old router is MI-3 router.

1~2 ms is normal for gigabit. Looks like the router was the problem.

Yes, thank you guys.

See also https://devtalk.nvidia.com/default/topic/995819/jetson-tx1/jetson-tx1-wifi-speed-and-latency/post/5102535/#5102535 for how to disable the power-saving of some chip, to improve the ping latency dramatically.

Hi,

I also have a network speed issue. I tried the wifi speed suggestion and it did not help. I have a couple tx2 on the wifi lan. When I ping a fast laptop I get:

nvidia@george:~/synccom$ ping spacely.local
PING spacely.local (192.168.10.198) 56(84) bytes of data.
64 bytes from 192.168.10.198: icmp_seq=1 ttl=64 time=210 ms
64 bytes from 192.168.10.198: icmp_seq=2 ttl=64 time=438 ms
64 bytes from 192.168.10.198: icmp_seq=3 ttl=64 time=9.44 ms
64 bytes from 192.168.10.198: icmp_seq=4 ttl=64 time=64.8 ms
64 bytes from 192.168.10.198: icmp_seq=5 ttl=64 time=35.2 ms
64 bytes from 192.168.10.198: icmp_seq=6 ttl=64 time=317 ms
64 bytes from 192.168.10.198: icmp_seq=7 ttl=64 time=13.5 ms
64 bytes from 192.168.10.198: icmp_seq=8 ttl=64 time=162 ms
64 bytes from 192.168.10.198: icmp_seq=9 ttl=64 time=182 ms
64 bytes from 192.168.10.198: icmp_seq=10 ttl=64 time=432 ms
64 bytes from 192.168.10.198: icmp_seq=11 ttl=64 time=230 ms
64 bytes from 192.168.10.198: icmp_seq=12 ttl=64 time=251 ms

or from another tx2:

nvidia@em7:~/synccom$ ping spacely.local
PING spacely.local (192.168.10.198) 56(84) bytes of data.
64 bytes from 192.168.10.198: icmp_seq=1 ttl=64 time=2.26 ms
64 bytes from 192.168.10.198: icmp_seq=2 ttl=64 time=1014 ms
64 bytes from 192.168.10.198: icmp_seq=3 ttl=64 time=275 ms
64 bytes from 192.168.10.198: icmp_seq=4 ttl=64 time=50.2 ms
64 bytes from 192.168.10.198: icmp_seq=5 ttl=64 time=490 ms
64 bytes from 192.168.10.198: icmp_seq=6 ttl=64 time=102 ms
64 bytes from 192.168.10.198: icmp_seq=7 ttl=64 time=536 ms
64 bytes from 192.168.10.198: icmp_seq=8 ttl=64 time=9.72 ms
64 bytes from 192.168.10.198: icmp_seq=9 ttl=64 time=168 ms
64 bytes from 192.168.10.198: icmp_seq=10 ttl=64 time=199 ms
64 bytes from 192.168.10.198: icmp_seq=11 ttl=64 time=633 ms
64 bytes from 192.168.10.198: icmp_seq=12 ttl=64 time=62.5 ms
64 bytes from 192.168.10.198: icmp_seq=13 ttl=64 time=672 ms
64 bytes from 192.168.10.198: icmp_seq=14 ttl=64 time=76.6 ms

But on the same lan if I ping laptop to laptop:

steve@spacely:~/synccom$ ping trillium.local
PING trillium.local (192.168.10.199) 56(84) bytes of data.
64 bytes from 192.168.10.199: icmp_seq=1 ttl=64 time=44.9 ms
64 bytes from 192.168.10.199: icmp_seq=2 ttl=64 time=3.76 ms
64 bytes from 192.168.10.199: icmp_seq=3 ttl=64 time=4.68 ms
64 bytes from 192.168.10.199: icmp_seq=4 ttl=64 time=3.99 ms
64 bytes from 192.168.10.199: icmp_seq=5 ttl=64 time=3.04 ms
64 bytes from 192.168.10.199: icmp_seq=6 ttl=64 time=10.9 ms
64 bytes from 192.168.10.199: icmp_seq=7 ttl=64 time=2.73 ms
64 bytes from 192.168.10.199: icmp_seq=8 ttl=64 time=3.68 ms
64 bytes from 192.168.10.199: icmp_seq=9 ttl=64 time=3.44 ms
64 bytes from 192.168.10.199: icmp_seq=10 ttl=64 time=4.14 ms
64 bytes from 192.168.10.199: icmp_seq=11 ttl=64 time=3.38 ms
64 bytes from 192.168.10.199: icmp_seq=12 ttl=64 time=2.22 ms
64 bytes from 192.168.10.199: icmp_seq=13 ttl=64 time=7.45 ms

Still not blazing speed, but more consistent and about 10x the jetson speed.

ssh speed to the tx2s has noticable network lag, too.

Some environment info too:

nvidia@george:~/synccom$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.5 LTS
Release:	16.04
Codename:	xenial
nvidia@george:~/synccom$ uname -a
Linux george 4.4.38-tegra #1 SMP PREEMPT Fri Jul 28 09:55:22 PDT 2017 aarch64 aarch64 aarch64 GNU/Linux

From a Jetson which is having issues, what is the output from “route” and “ifconfig”?

here is the info you asked for:

nvidia@george:~/synccom$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         Hera-pfSense.he 0.0.0.0         UG    600    0        0 wlan0
link-local      *               255.255.0.0     U     1000   0        0 wlan0
192.168.10.0    *               255.255.255.0   U     600    0        0 wlan0
nvidia@george:~/synccom$ ifconfig
eth0      Link encap:Ethernet  HWaddr 00:04:4b:c4:f8:4b  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:42 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:3401 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3401 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:336619 (336.6 KB)  TX bytes:336619 (336.6 KB)

wlan0     Link encap:Ethernet  HWaddr 00:04:4b:c4:f8:49  
          inet addr:192.168.10.200  Bcast:192.168.10.255  Mask:255.255.255.0
          inet6 addr: fe80::c1ee:9db5:3c34:bde4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:414369 errors:0 dropped:0 overruns:0 frame:0
          TX packets:206562 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:93640203 (93.6 MB)  TX bytes:32573399 (32.5 MB)

nvidia@george:~/synccom$ ping spacely.local
PING spacely.local (192.168.10.198) 56(84) bytes of data.
64 bytes from 192.168.10.198: icmp_seq=1 ttl=64 time=19.2 ms
64 bytes from 192.168.10.198: icmp_seq=2 ttl=64 time=432 ms
64 bytes from 192.168.10.198: icmp_seq=3 ttl=64 time=40.5 ms
64 bytes from 192.168.10.198: icmp_seq=4 ttl=64 time=302 ms
64 bytes from 192.168.10.198: icmp_seq=5 ttl=64 time=88.8 ms
64 bytes from 192.168.10.198: icmp_seq=6 ttl=64 time=108 ms
64 bytes from 192.168.10.198: icmp_seq=7 ttl=64 time=341 ms
64 bytes from 192.168.10.198: icmp_seq=8 ttl=64 time=159 ms
^C
--- spacely.local ping statistics ---
8 packets transmitted, 8 received, 0% packet loss, time 7008ms
rtt min/avg/max/mdev = 19.211/186.733/432.660/143.123 ms

I don’t see any outright errors or failures. On the other hand, I see this is purely over WiFi. I’m not all that great at debugging WiFi, but there are a lot of things which can get in the way (antenna placement, noise in the area, wiring in walls…just about anything).

You might want to test when the Jetson is physically very close to the WiFi router, and again when further away. This would be a cheap way to test if noise might be a problem since signal to noise goes up when they sit side by side, or gets worse when far apart. Should the ping and latency (and especially how variable it is) not change for the better when the Jetson is sitting next to the router it is more likely a software issue or hardware issue and not environmental.

Ok, I will have to find out where the router is!

But in any case the laptop PC is only about 3 feet from the jetson, so I would expect similar speeds. I’ll let you know when/if I find out anything about router proximity.

Steve

This is a common issue with Jetsons, the wifi has a power saving mode you can change with this command:

$ iw dev wlan0 set power_save off

you can restore this by

$ iw dev wlan0 get power_save