while using ssh and nfs to work on TX2, I find the console command very lagged. Then I pinged another ip of the same LAN from TX2, it shows the latency is around a few hundred milliseconds to over 1 second. This is very high compared to TX1 in the same network (around 2~3 ms)
Anyone ever had the same issue?
The Jetson SDK version is 28.1
Hi xliu,
Is nfs necessary to reproduce this issue? Would this happen when only use ssh?
no need.
just ping any other ip, the round trip time shows it. For TX2, a few hundred ms to 1000ms, for TX1, 2~3ms.
I’ve done it after a fresh reboot, without using ssh or nfs.
Hi xliu,
This is my test result.
TX1
64 bytes from 10.19.106.138: icmp_seq=26 ttl=64 time=0.791 ms
64 bytes from 10.19.106.138: icmp_seq=27 ttl=64 time=0.881 ms
64 bytes from 10.19.106.138: icmp_seq=28 ttl=64 time=1.09 ms
64 bytes from 10.19.106.138: icmp_seq=29 ttl=64 time=1.09 ms
64 bytes from 10.19.106.138: icmp_seq=30 ttl=64 time=0.734 ms
64 bytes from 10.19.106.138: icmp_seq=31 ttl=64 time=0.662 ms
64 bytes from 10.19.106.138: icmp_seq=32 ttl=64 time=0.924 ms
TX2
64 bytes from 10.19.106.138: icmp_seq=26 ttl=64 time=1.74 ms
64 bytes from 10.19.106.138: icmp_seq=27 ttl=64 time=1.89 ms
64 bytes from 10.19.106.138: icmp_seq=28 ttl=64 time=1.66 ms
64 bytes from 10.19.106.138: icmp_seq=29 ttl=64 time=1.79 ms
64 bytes from 10.19.106.138: icmp_seq=30 ttl=64 time=2.24 ms
64 bytes from 10.19.106.138: icmp_seq=31 ttl=64 time=1.34 ms
64 bytes from 10.19.106.138: icmp_seq=32 ttl=64 time=1.73 ms
It does not have abnormal value like 1000ms. Could you check if there is any error message in “dmesg” while pinging.
Does latency improve if you run the “sudo ~/jetson_clocks.sh” script and “sudo nvpmodel -m 0”?
yes, it improves. Now it’s never more than 300ms. What does this mean?
64 bytes from 192.168.31.80: icmp_seq=13 ttl=64 time=246 ms
64 bytes from 192.168.31.80: icmp_seq=14 ttl=64 time=141 ms
64 bytes from 192.168.31.80: icmp_seq=15 ttl=64 time=221 ms
64 bytes from 192.168.31.80: icmp_seq=16 ttl=64 time=249 ms
64 bytes from 192.168.31.80: icmp_seq=17 ttl=64 time=247 ms
64 bytes from 192.168.31.80: icmp_seq=18 ttl=64 time=31.5 ms
64 bytes from 192.168.31.80: icmp_seq=19 ttl=64 time=155 ms
64 bytes from 192.168.31.80: icmp_seq=20 ttl=64 time=5.03 ms
64 bytes from 192.168.31.80: icmp_seq=21 ttl=64 time=3.96 ms
64 bytes from 192.168.31.80: icmp_seq=22 ttl=64 time=194 ms
64 bytes from 192.168.31.80: icmp_seq=23 ttl=64 time=55.1 ms
64 bytes from 192.168.31.80: icmp_seq=24 ttl=64 time=26.3 ms
64 bytes from 192.168.31.80: icmp_seq=25 ttl=64 time=14.0 ms
64 bytes from 192.168.31.80: icmp_seq=26 ttl=64 time=249 ms
Please try these steps separately and see if it is affected by jetson_clock.sh or nvpmodel.
Hi Wayne,
Just did it again with them separately and jointly, I see no improvement. Maybe it was just luck in my last post.
Also, 240 milliseconds ping to a local network node is still terrible.
There’s something going on on your network, or perhaps on your machine, that causes significant degradation.
Here’s pings from my machine in low power mode:
Note the times that are 1,000 times faster than the times you are seeing!
I changed my router with another, then it comes back to normal (1~2ms).
The old router is MI-3 router.
1~2 ms is normal for gigabit. Looks like the router was the problem.
Yes, thank you guys.
See also [url]https://devtalk.nvidia.com/default/topic/995819/jetson-tx1/jetson-tx1-wifi-speed-and-latency/post/5102535/#5102535[/url] for how to disable the power-saving of some chip, to improve the ping latency dramatically.
Hi,
I also have a network speed issue. I tried the wifi speed suggestion and it did not help. I have a couple tx2 on the wifi lan. When I ping a fast laptop I get:
nvidia@george:~/synccom$ ping spacely.local
PING spacely.local (192.168.10.198) 56(84) bytes of data.
64 bytes from 192.168.10.198: icmp_seq=1 ttl=64 time=210 ms
64 bytes from 192.168.10.198: icmp_seq=2 ttl=64 time=438 ms
64 bytes from 192.168.10.198: icmp_seq=3 ttl=64 time=9.44 ms
64 bytes from 192.168.10.198: icmp_seq=4 ttl=64 time=64.8 ms
64 bytes from 192.168.10.198: icmp_seq=5 ttl=64 time=35.2 ms
64 bytes from 192.168.10.198: icmp_seq=6 ttl=64 time=317 ms
64 bytes from 192.168.10.198: icmp_seq=7 ttl=64 time=13.5 ms
64 bytes from 192.168.10.198: icmp_seq=8 ttl=64 time=162 ms
64 bytes from 192.168.10.198: icmp_seq=9 ttl=64 time=182 ms
64 bytes from 192.168.10.198: icmp_seq=10 ttl=64 time=432 ms
64 bytes from 192.168.10.198: icmp_seq=11 ttl=64 time=230 ms
64 bytes from 192.168.10.198: icmp_seq=12 ttl=64 time=251 ms
or from another tx2:
nvidia@em7:~/synccom$ ping spacely.local
PING spacely.local (192.168.10.198) 56(84) bytes of data.
64 bytes from 192.168.10.198: icmp_seq=1 ttl=64 time=2.26 ms
64 bytes from 192.168.10.198: icmp_seq=2 ttl=64 time=1014 ms
64 bytes from 192.168.10.198: icmp_seq=3 ttl=64 time=275 ms
64 bytes from 192.168.10.198: icmp_seq=4 ttl=64 time=50.2 ms
64 bytes from 192.168.10.198: icmp_seq=5 ttl=64 time=490 ms
64 bytes from 192.168.10.198: icmp_seq=6 ttl=64 time=102 ms
64 bytes from 192.168.10.198: icmp_seq=7 ttl=64 time=536 ms
64 bytes from 192.168.10.198: icmp_seq=8 ttl=64 time=9.72 ms
64 bytes from 192.168.10.198: icmp_seq=9 ttl=64 time=168 ms
64 bytes from 192.168.10.198: icmp_seq=10 ttl=64 time=199 ms
64 bytes from 192.168.10.198: icmp_seq=11 ttl=64 time=633 ms
64 bytes from 192.168.10.198: icmp_seq=12 ttl=64 time=62.5 ms
64 bytes from 192.168.10.198: icmp_seq=13 ttl=64 time=672 ms
64 bytes from 192.168.10.198: icmp_seq=14 ttl=64 time=76.6 ms
But on the same lan if I ping laptop to laptop:
steve@spacely:~/synccom$ ping trillium.local
PING trillium.local (192.168.10.199) 56(84) bytes of data.
64 bytes from 192.168.10.199: icmp_seq=1 ttl=64 time=44.9 ms
64 bytes from 192.168.10.199: icmp_seq=2 ttl=64 time=3.76 ms
64 bytes from 192.168.10.199: icmp_seq=3 ttl=64 time=4.68 ms
64 bytes from 192.168.10.199: icmp_seq=4 ttl=64 time=3.99 ms
64 bytes from 192.168.10.199: icmp_seq=5 ttl=64 time=3.04 ms
64 bytes from 192.168.10.199: icmp_seq=6 ttl=64 time=10.9 ms
64 bytes from 192.168.10.199: icmp_seq=7 ttl=64 time=2.73 ms
64 bytes from 192.168.10.199: icmp_seq=8 ttl=64 time=3.68 ms
64 bytes from 192.168.10.199: icmp_seq=9 ttl=64 time=3.44 ms
64 bytes from 192.168.10.199: icmp_seq=10 ttl=64 time=4.14 ms
64 bytes from 192.168.10.199: icmp_seq=11 ttl=64 time=3.38 ms
64 bytes from 192.168.10.199: icmp_seq=12 ttl=64 time=2.22 ms
64 bytes from 192.168.10.199: icmp_seq=13 ttl=64 time=7.45 ms
Still not blazing speed, but more consistent and about 10x the jetson speed.
ssh speed to the tx2s has noticable network lag, too.
Some environment info too:
nvidia@george:~/synccom$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.5 LTS
Release: 16.04
Codename: xenial
nvidia@george:~/synccom$ uname -a
Linux george 4.4.38-tegra #1 SMP PREEMPT Fri Jul 28 09:55:22 PDT 2017 aarch64 aarch64 aarch64 GNU/Linux
From a Jetson which is having issues, what is the output from “route” and “ifconfig”?
here is the info you asked for:
nvidia@george:~/synccom$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default Hera-pfSense.he 0.0.0.0 UG 600 0 0 wlan0
link-local * 255.255.0.0 U 1000 0 0 wlan0
192.168.10.0 * 255.255.255.0 U 600 0 0 wlan0
nvidia@george:~/synccom$ ifconfig
eth0 Link encap:Ethernet HWaddr 00:04:4b:c4:f8:4b
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Interrupt:42
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:3401 errors:0 dropped:0 overruns:0 frame:0
TX packets:3401 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:336619 (336.6 KB) TX bytes:336619 (336.6 KB)
wlan0 Link encap:Ethernet HWaddr 00:04:4b:c4:f8:49
inet addr:192.168.10.200 Bcast:192.168.10.255 Mask:255.255.255.0
inet6 addr: fe80::c1ee:9db5:3c34:bde4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:414369 errors:0 dropped:0 overruns:0 frame:0
TX packets:206562 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:93640203 (93.6 MB) TX bytes:32573399 (32.5 MB)
nvidia@george:~/synccom$ ping spacely.local
PING spacely.local (192.168.10.198) 56(84) bytes of data.
64 bytes from 192.168.10.198: icmp_seq=1 ttl=64 time=19.2 ms
64 bytes from 192.168.10.198: icmp_seq=2 ttl=64 time=432 ms
64 bytes from 192.168.10.198: icmp_seq=3 ttl=64 time=40.5 ms
64 bytes from 192.168.10.198: icmp_seq=4 ttl=64 time=302 ms
64 bytes from 192.168.10.198: icmp_seq=5 ttl=64 time=88.8 ms
64 bytes from 192.168.10.198: icmp_seq=6 ttl=64 time=108 ms
64 bytes from 192.168.10.198: icmp_seq=7 ttl=64 time=341 ms
64 bytes from 192.168.10.198: icmp_seq=8 ttl=64 time=159 ms
^C
--- spacely.local ping statistics ---
8 packets transmitted, 8 received, 0% packet loss, time 7008ms
rtt min/avg/max/mdev = 19.211/186.733/432.660/143.123 ms
I don’t see any outright errors or failures. On the other hand, I see this is purely over WiFi. I’m not all that great at debugging WiFi, but there are a lot of things which can get in the way (antenna placement, noise in the area, wiring in walls…just about anything).
You might want to test when the Jetson is physically very close to the WiFi router, and again when further away. This would be a cheap way to test if noise might be a problem since signal to noise goes up when they sit side by side, or gets worse when far apart. Should the ping and latency (and especially how variable it is) not change for the better when the Jetson is sitting next to the router it is more likely a software issue or hardware issue and not environmental.
Ok, I will have to find out where the router is!
But in any case the laptop PC is only about 3 feet from the jetson, so I would expect similar speeds. I’ll let you know when/if I find out anything about router proximity.
Steve
This is a common issue with Jetsons, the wifi has a power saving mode you can change with this command:
$ iw dev wlan0 set power_save off
you can restore this by
$ iw dev wlan0 get power_save