xavier cannot initiate the connection to ports.ubuntu.com to install CUDA (Solved)

Hi, everyone. I have several xaviers and have flashed three xaviers with jetpack successfully, but there is something wrong with one xavier while installing CUDA on target using jetpack.

Jetpack installer on the host reports:

Err: http://ports.ubuntu.com/ubuntu-ports bionic InRelease
  Cannot initialte the connection to ports.ubuntu.com:80 (2001:67c:1360:8001::22)
  - connect (101: Network is unreachable) Could not connect to ports.ubuntu.com:80 (91.189.88.150), connection timed out.

My network is fine both on host and xavier:

ping ports.ubuntu.com
PING ports.ubuntu.com (91.189.88.150) 56(84) bytes of data.
64 bytes from astomi.canonical.com (91.189.88.150): icmp_seq=1 ttl=49 time=227 ms
64 bytes from astomi.canonical.com (91.189.88.150): icmp_seq=2 ttl=49 time=227 ms
64 bytes from astomi.canonical.com (91.189.88.150): icmp_seq=3 ttl=49 time=227 ms
64 bytes from astomi.canonical.com (91.189.88.150): icmp_seq=4 ttl=49 time=227 ms
64 bytes from astomi.canonical.com (91.189.88.150): icmp_seq=5 ttl=49 time=227 ms
64 bytes from astomi.canonical.com (91.189.88.150): icmp_seq=6 ttl=49 time=228 ms
64 bytes from astomi.canonical.com (91.189.88.150): icmp_seq=7 ttl=49 time=227 ms
64 bytes from astomi.canonical.com (91.189.88.150): icmp_seq=8 ttl=49 time=227 ms
64 bytes from astomi.canonical.com (91.189.88.150): icmp_seq=9 ttl=49 time=227 ms
64 bytes from astomi.canonical.com (91.189.88.150): icmp_seq=10 ttl=49 time=227 ms
64 bytes from astomi.canonical.com (91.189.88.150): icmp_seq=11 ttl=49 time=227 ms
64 bytes from astomi.canonical.com (91.189.88.150): icmp_seq=12 ttl=49 time=227 ms

And ‘apt-get update’ doesn’t work on xavier as following:

nvidia@jetson-0423718017049:~$ sudo apt-get update
[sudo] password for nvidia: 
Err:1 http://ports.ubuntu.com/ubuntu-ports bionic InRelease 
  Cannot initiate the connection to ports.ubuntu.com:80 (2001:67c:1360:8001::22). - connect (101: Network is unreachable) Could not connect to ports.ubuntu.com:80 (91.189.88.150), connection timed out
Err:2 http://ports.ubuntu.com/ubuntu-ports bionic-updates InRelease
  Cannot initiate the connection to ports.ubuntu.com:80 (2001:67c:1360:8001::22). - connect (101: Network is unreachable)
Err:3 http://ports.ubuntu.com/ubuntu-ports bionic-backports InRelease
  Cannot initiate the connection to ports.ubuntu.com:80 (2001:67c:1360:8001::22). - connect (101: Network is unreachable)
Err:4 http://ports.ubuntu.com/ubuntu-ports bionic-security InRelease
  Cannot initiate the connection to ports.ubuntu.com:80 (2001:67c:1360:8001::22). - connect (101: Network is unreachable)
Reading package lists... Done
W: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/bionic/InRelease  Cannot initiate the connection to ports.ubuntu.com:80 (2001:67c:1360:8001::22). - connect (101: Network is unreachable) Could not connect to ports.ubuntu.com:80 (91.189.88.150), connection timed out
W: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/bionic-updates/InRelease  Cannot initiate the connection to ports.ubuntu.com:80 (2001:67c:1360:8001::22). - connect (101: Network is unreachable)
W: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/bionic-backports/InRelease  Cannot initiate the connection to ports.ubuntu.com:80 (2001:67c:1360:8001::22). - connect (101: Network is unreachable)
W: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/bionic-security/InRelease  Cannot initiate the connection to ports.ubuntu.com:80 (2001:67c:1360:8001::22). - connect (101: Network is unreachable)
W: Some index files failed to download. They have been ignored, or old ones used instead.

Does anyone has any idea about this problem?

If “Network is unreachable”, then perhaps there is another firewall or proxy. Some parts of the world may also block this content.

Are you using wired or WiFi? If wired, then it becomes much easier to figure out. Generally speaking though, can you ping the outside world, e.g., “ping google.com”?

I am using a wired network, and i also question the firewall or proxy, but the same network and Jetpack installer worked on the other xaviers and “apt-get update” succeeded on the host when it failed on my “problem” xavier.

What does the failing Xavier say from these commands:

host ports.ubuntu.com
route
ifconfig

Here are the outputs of these commands and “apt-get update” on Xavier:

nvidia@jetson-0423718017049:~$ host ports.ubuntu.com
ports.ubuntu.com has address 91.189.88.150
ports.ubuntu.com has IPv6 address 2001:67c:1360:8001::22
nvidia@jetson-0423718017049:~$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         _gateway        0.0.0.0         UG    0      0        0 eth0
default         _gateway        0.0.0.0         UG    20100  0        0 eth0
link-local      0.0.0.0         255.255.0.0     U     1000   0        0 l4tbr0
172.30.**.0     0.0.0.0         255.255.255.0   U     100    0        0 eth0
192.168.**.0    0.0.0.0         255.255.255.0   U     0      0        0 l4tbr0
nvidia@jetson-0423718017049:~$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.30.*.*  netmask 255.255.255.0  broadcast 172.30.*.255
        inet6 fe80::3aef:3213:*:*  prefixlen 64  scopeid 0x20<link>
        ether 00:04:4b:*:*:*  txqueuelen 1000  (Ethernet)
        RX packets 297579  bytes 334153920 (334.1 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2926  bytes 898506 (898.5 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 40  

l4tbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.*.*  netmask 255.255.255.0  broadcast 192.168.*.255
        inet6 fe80::f425:16ff:*:*  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::1  prefixlen 128  scopeid 0x20<link>
        ether 62:08:40:4c:*:*  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 9  bytes 1668 (1.6 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 459  bytes 33530 (33.5 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 459  bytes 33530 (33.5 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

rndis0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 62:08:40:4c:a3:*  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

usb0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether ca:da:34:d6:ec:*  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
nvidia@jetson-0423718017049:~$ sudo apt-get update
[sudo] password for nvidia: 
Get:1 file:/var/cuda-repo-10-0-local-10.0.117  InRelease
Ign:1 file:/var/cuda-repo-10-0-local-10.0.117  InRelease
Get:2 file:/var/cuda-repo-10-0-local-10.0.117  Release [574 B]
Get:2 file:/var/cuda-repo-10-0-local-10.0.117  Release [574 B]
Err:4 http://ports.ubuntu.com/ubuntu-ports bionic InRelease                    
  Cannot initiate the connection to ports.ubuntu.com:80 (2001:67c:1360:8001::22). - connect (101: Network is unreachable) Could not connect to ports.ubuntu.com:80 (91.189.88.150), connection timed out
Err:5 http://ports.ubuntu.com/ubuntu-ports bionic-updates InRelease
  Cannot initiate the connection to ports.ubuntu.com:80 (2001:67c:1360:8001::22). - connect (101: Network is unreachable)
Err:6 http://ports.ubuntu.com/ubuntu-ports bionic-backports InRelease
  Cannot initiate the connection to ports.ubuntu.com:80 (2001:67c:1360:8001::22). - connect (101: Network is unreachable)
Err:7 http://ports.ubuntu.com/ubuntu-ports bionic-security InRelease
  Cannot initiate the connection to ports.ubuntu.com:80 (2001:67c:1360:8001::22). - connect (101: Network is unreachable)
Reading package lists... Done                               
W: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/bionic/InRelease  Cannot initiate the connection to ports.ubuntu.com:80 (2001:67c:1360:8001::22). - connect (101: Network is unreachable) Could not connect to ports.ubuntu.com:80 (91.189.88.150), connection timed out
W: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/bionic-updates/InRelease  Cannot initiate the connection to ports.ubuntu.com:80 (2001:67c:1360:8001::22). - connect (101: Network is unreachable)
W: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/bionic-backports/InRelease  Cannot initiate the connection to ports.ubuntu.com:80 (2001:67c:1360:8001::22). - connect (101: Network is unreachable)
W: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/bionic-security/InRelease  Cannot initiate the connection to ports.ubuntu.com:80 (2001:67c:1360:8001::22). - connect (101: Network is unreachable)
W: Some index files failed to download. They have been ignored, or old ones used instead.

You have two gateway routes through wired eth0…other than metric they are the same:

default         _gateway        0.0.0.0         UG    0      0        0 eth0
default         _gateway        0.0.0.0         UG    20100  0        0 eth0

I don’t know what will happen with this…perhaps nothing will happen…but it is quite odd. How this would happen I don’t know.

I don’t see the full addresses, I assume you just masked those, which should be ok (if this is the literal output though, then there is a problem).

I also see DNS works:

ports.ubuntu.com has address 91.189.88.150

Since 91.189.88.150 is not in the 172.30.**.0/255.255.255.0 subnet it does mean that the gateway must provide the final connection. What is your gateway? Is this from a router appliance?

For particular testing, does this work?

ping 91.189.88.150

If not, then where does it fail? Keep in mind that ping and traceroute are ICMP and not all gateways will support ICMP, so some hops might not show results, but see if there is an obvious cut off point…I am thinking it will be your router. Use this command to find out:

traceroute 91.189.88.150

The problem disappeared this morning, “apt-get update” worked fine and Jetpack installer succeeded to install CUDA on the “problem” xavier.

It seems a problem with the route table which is automatically generated by the system or our network. There were two default gateway routes yesterday, but one default gateway route today.

Thanks a lot, linuxdev.

Here is what I ran into:

$ ping google.com
< ping: google.com: Name or service not known >
ping ports.ubuntu.com
< cannot connect to ports.ubuntu.com >

So now I am going to get a new IP address

$ sudo dhclient -r
$ sudo dhclient

$ ping google.com
PING google.com (xxx.xx.xxx.xx) bytes of data.
64 bytes from xxxx (xxx.xx.xxx.xx): icmp_seq=1 ttl=57 time=14.5 ms

now it works!

dns propagation

Whenever dns is changed, it takes a while for dns servers across the internet to sync up. Some ISPs are horrendous at this. If you use google dns on your router, or cloudflare, you will never run into this problem.