How to add PCIe bridge to increase number of PCIe slots?

Auvidea · November 29, 2018, 1:20pm

We have interfaced a Pericom PI7C9X2G1616PR 16 lane PCIe bridge to the TX2. 4 Lanes are connected upstream to the TX2. Downstream 4 lanes are connected to an M.2 NVME slot and 8 lanes are connected to 8 PCIe x1 slots. Two of these are populated with i210 GbE modules.

The problem is that these 3 devices are not always initialised correctly after boot up. Below is a non working case: eth0 and eth1 are only available after remove and rescan. The SSD is not available at all. We suspect that uBoot does not scan the busses after the bridge correctly in all cases. This seems to be random. uBoot should scan those buses to find out all PCI/PCIe device and do enumeration (assign I/O, Memory and IRQ resources for all PCI/PCIe devices). How can we ensure that?

Non working case (ok after remove & rescan)

ubuntu@tegra-ubuntu:~$ lspci
00:01.0 PCI bridge: NVIDIA Corporation Device 0fae (rev a1)
01:00.0 PCI bridge: Pericom Semiconductor Device 8619
01:00.1 System peripheral: Pericom Semiconductor Device 8619
02:01.0 PCI bridge: Pericom Semiconductor Device 8619
02:02.0 PCI bridge: Pericom Semiconductor Device 8619
02:03.0 PCI bridge: Pericom Semiconductor Device 8619
02:0a.0 PCI bridge: Pericom Semiconductor Device 8619
02:0b.0 PCI bridge: Pericom Semiconductor Device 8619
02:0c.0 PCI bridge: Pericom Semiconductor Device 8619
02:0d.0 PCI bridge: Pericom Semiconductor Device 8619
02:0e.0 PCI bridge: Pericom Semiconductor Device 8619
02:0f.0 PCI bridge: Pericom Semiconductor Device 8619
06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
0a:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
ubuntu@tegra-ubuntu:~$ ifconfig 
enx00044b7aba6d Link encap:Ethernet  HWaddr 00:04:4b:7a:ba:6d  
          inet addr:192.168.0.27  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::204:4bff:fe7a:ba6d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:847 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1797 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:143386 (143.3 KB)  TX bytes:1471106 (1.4 MB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:903 errors:0 dropped:0 overruns:0 frame:0
          TX packets:903 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:110001 (110.0 KB)  TX bytes:110001 (110.0 KB)

wlan0     Link encap:Ethernet  HWaddr 00:04:4b:7a:ba:6b  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

ubuntu@tegra-ubuntu:~$ echo 1 > /sys/bus/pci/devices/0000\:06\:00.0/remove 
ubuntu@tegra-ubuntu:~$ echo 1 > /sys/bus/pci/devices/0000\:0a\:00.0/remove 
ubuntu@tegra-ubuntu:~$ echo 1 > /sys/bus/pci/rescan 
ubuntu@tegra-ubuntu:~$ lspci
00:01.0 PCI bridge: NVIDIA Corporation Device 0fae (rev a1)
01:00.0 PCI bridge: Pericom Semiconductor Device 8619
01:00.1 System peripheral: Pericom Semiconductor Device 8619
02:01.0 PCI bridge: Pericom Semiconductor Device 8619
02:02.0 PCI bridge: Pericom Semiconductor Device 8619
02:03.0 PCI bridge: Pericom Semiconductor Device 8619
02:0a.0 PCI bridge: Pericom Semiconductor Device 8619
02:0b.0 PCI bridge: Pericom Semiconductor Device 8619
02:0c.0 PCI bridge: Pericom Semiconductor Device 8619
02:0d.0 PCI bridge: Pericom Semiconductor Device 8619
02:0e.0 PCI bridge: Pericom Semiconductor Device 8619
02:0f.0 PCI bridge: Pericom Semiconductor Device 8619
06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
0a:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
ubuntu@tegra-ubuntu:~$ ifconfig 
enx00044b7aba6d Link encap:Ethernet  HWaddr 00:04:4b:7a:ba:6d  
          inet addr:192.168.0.27  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::204:4bff:fe7a:ba6d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2093 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6195 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:392946 (392.9 KB)  TX bytes:5706533 (5.7 MB)

eth0      Link encap:Ethernet  HWaddr 00:a0:c9:00:00:01  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Memory:13000000-13100000 

eth1      Link encap:Ethernet  HWaddr 00:a0:c9:00:00:00  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Memory:13200000-13300000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:1137 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1137 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:127207 (127.2 KB)  TX bytes:127207 (127.2 KB)

wlan0     Link encap:Ethernet  HWaddr 00:04:4b:7a:ba:6b  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

ubuntu@tegra-ubuntu:~$

Here is a working case: eth0, eth1 and the SSD straight away after boot of the system (without taking any special action)

ubuntu@tegra-ubuntu:~$ lspci
00:01.0 PCI bridge: NVIDIA Corporation Device 0fae (rev a1)
01:00.0 PCI bridge: Pericom Semiconductor Device 8619
01:00.1 System peripheral: Pericom Semiconductor Device 8619
02:01.0 PCI bridge: Pericom Semiconductor Device 8619
02:02.0 PCI bridge: Pericom Semiconductor Device 8619
02:03.0 PCI bridge: Pericom Semiconductor Device 8619
02:0a.0 PCI bridge: Pericom Semiconductor Device 8619
02:0b.0 PCI bridge: Pericom Semiconductor Device 8619
02:0c.0 PCI bridge: Pericom Semiconductor Device 8619
02:0d.0 PCI bridge: Pericom Semiconductor Device 8619
02:0e.0 PCI bridge: Pericom Semiconductor Device 8619
02:0f.0 PCI bridge: Pericom Semiconductor Device 8619
03:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a804
06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
0a:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
ubuntu@tegra-ubuntu:~$ ifconfig 
enx00044b7aba6d Link encap:Ethernet  HWaddr 00:04:4b:7a:ba:6d  
          inet addr:192.168.0.27  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::204:4bff:fe7a:ba6d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4453 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8151 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:513691 (513.6 KB)  TX bytes:5790657 (5.7 MB)

eth1      Link encap:Ethernet  HWaddr 00:a0:c9:00:00:01  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Memory:13100000-13200000 

eth2      Link encap:Ethernet  HWaddr 00:a0:c9:00:00:00  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Memory:13300000-13400000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:1164 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1164 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:128738 (128.7 KB)  TX bytes:128738 (128.7 KB)

wlan0     Link encap:Ethernet  HWaddr 00:04:4b:7a:ba:6b  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

ubuntu@tegra-ubuntu:~$ df
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/mmcblk0p1  14318640 12904004    664252  96% /
none             2033572        0   2033572   0% /dev
tmpfs            2045372      592   2044780   1% /dev/shm
tmpfs            2045372    17744   2027628   1% /run
tmpfs               5120        4      5116   1% /run/lock
tmpfs            2045372        0   2045372   0% /sys/fs/cgroup
tmpfs             409076       64    409012   1% /run/user/1000
/dev/nvme0n1   240234168    60564 227947292   1% /media/ubuntu/CorsairSSD
ubuntu@tegra-ubuntu:~$

vidyas · November 30, 2018, 4:40am

Can you please confirm that there is no issue in terms of PCIe devices getting enumerated? I mean, in both working and non-working cases, you do get all PCIe devices enumerated as per the expectations right?

vidyas · November 30, 2018, 4:42am

Also, can you please share full logs in both working and non-working cases?

kayccc · December 6, 2018, 3:14am

Hi Auvidea,

Please help to provide further information to help to move this issue forward.
Or you have clarified the cause and resolved the problem?

Thanks

Auvidea · December 14, 2018, 3:55pm

The Pericom bridge is still not working. Today we have tested it with a new carrier board - the J220. Here debugging is easier, as it has a standard HDMI port. Previously we tested this on a J94, which is a headless carrier board with no HDMI monitor.

But the results are still the same. The system works very badly, if booted at warm or hot temperature. If it is called down to -20°C while booting, it at least detect all PCIe devices connected: 8x i210 Ethernet controller and 1x M.2 NVME SSD card.

Here are the errors the igb driver puts out:

ubuntu@tegra-ubuntu:~$ journalctl -k | grep igb
Feb 11 17:28:00 tegra-ubuntu kernel: igb: Intel(R) Gigabit Ethernet Network Driver - version 5.0.3-k
Feb 11 17:28:00 tegra-ubuntu kernel: igb: Copyright (c) 2007-2013 Intel Corporation.
Feb 11 17:28:00 tegra-ubuntu kernel: igb 0000:04:00.0: Refused to change power state, currently in D3
Feb 11 17:28:00 tegra-ubuntu kernel: igb 0000:04:00.0: PHY reset is blocked due to SOL/IDER session.
Feb 11 17:28:00 tegra-ubuntu kernel: igb 0000:04:00.0: Invalid MAC Address
Feb 11 17:28:00 tegra-ubuntu kernel: igb: probe of 0000:04:00.0 failed with error -5
Feb 11 17:28:00 tegra-ubuntu kernel: igb 0000:05:00.0: PHY reset is blocked due to SOL/IDER session.
Feb 11 17:28:00 tegra-ubuntu kernel: igb 0000:05:00.0: Invalid MAC Address
Feb 11 17:28:00 tegra-ubuntu kernel: igb: probe of 0000:05:00.0 failed with error -5
Feb 11 17:28:00 tegra-ubuntu kernel: igb 0000:06:00.0: PHY reset is blocked due to SOL/IDER session.
Feb 11 17:28:00 tegra-ubuntu kernel: igb 0000:06:00.0: Invalid MAC Address
Feb 11 17:28:00 tegra-ubuntu kernel: igb: probe of 0000:06:00.0 failed with error -5
Feb 11 17:28:00 tegra-ubuntu kernel: igb 0000:07:00.0: PHY reset is blocked due to SOL/IDER session.
Feb 11 17:28:00 tegra-ubuntu kernel: igb 0000:07:00.0: Invalid MAC Address
Feb 11 17:28:00 tegra-ubuntu kernel: igb: probe of 0000:07:00.0 failed with error -5
Feb 11 17:28:00 tegra-ubuntu kernel: igb 0000:08:00.0: PHY reset is blocked due to SOL/IDER session.
Feb 11 17:28:00 tegra-ubuntu kernel: igb 0000:08:00.0: Invalid MAC Address
Feb 11 17:28:00 tegra-ubuntu kernel: igb: probe of 0000:08:00.0 failed with error -5
Feb 11 17:28:00 tegra-ubuntu kernel: igb 0000:09:00.0: PHY reset is blocked due to SOL/IDER session.
Feb 11 17:28:00 tegra-ubuntu kernel: igb 0000:09:00.0: Invalid MAC Address
Feb 11 17:28:00 tegra-ubuntu kernel: igb: probe of 0000:09:00.0 failed with error -5
Feb 11 17:28:00 tegra-ubuntu kernel: igb 0000:0a:00.0: PHY reset is blocked due to SOL/IDER session.
Feb 11 17:28:00 tegra-ubuntu kernel: igb 0000:0a:00.0: Invalid MAC Address
Feb 11 17:28:00 tegra-ubuntu kernel: igb: probe of 0000:0a:00.0 failed with error -5
Feb 11 17:28:00 tegra-ubuntu kernel: igb 0000:0b:00.0: PHY reset is blocked due to SOL/IDER session.
Feb 11 17:28:00 tegra-ubuntu kernel: igb 0000:0b:00.0: Invalid MAC Address
Feb 11 17:28:00 tegra-ubuntu kernel: igb: probe of 0000:0b:00.0 failed with error -5

Here is the complete list produced by “journalctl -k” because dmesg ran out of buffer space.

errors.txt (2.37 KB)

Auvidea · December 14, 2018, 3:56pm

I have tried to put the complete log into a code box, but the log was too large. So I have attached it as a file.

The tests above were performed on the TX1. We also tried a TX2. Here the error messages and the logs are different.

nvidia@tegra-ubuntu:~$ lspci
00:01.0 PCI bridge: NVIDIA Corporation Device 10e5 (rev a1)
01:00.0 PCI bridge: Pericom Semiconductor Device 8619
01:00.1 System peripheral: Pericom Semiconductor Device 8619
02:01.0 PCI bridge: Pericom Semiconductor Device 8619
02:02.0 PCI bridge: Pericom Semiconductor Device 8619
02:03.0 PCI bridge: Pericom Semiconductor Device 8619
02:0a.0 PCI bridge: Pericom Semiconductor Device 8619
02:0b.0 PCI bridge: Pericom Semiconductor Device 8619
02:0c.0 PCI bridge: Pericom Semiconductor Device 8619
02:0d.0 PCI bridge: Pericom Semiconductor Device 8619
02:0e.0 PCI bridge: Pericom Semiconductor Device 8619
02:0f.0 PCI bridge: Pericom Semiconductor Device 8619
04:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
07:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
08:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
09:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
0a:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
0b:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
nvidia@tegra-ubuntu:~$ ifconfig
eth0      Link encap:Ethernet  HWaddr 00:04:4b:8c:9a:de  
          inet addr:192.168.0.43  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::6641:1ea2:f2bd:9557/64 Scope:Link
          inet6 addr: fd00::9ed:3595:8b81:8258/64 Scope:Global
          inet6 addr: fd00::8da9:e815:f3c0:b570/64 Scope:Global
          inet6 addr: 2001:a61:344e:d700:9ed:3595:8b81:8258/64 Scope:Global
          inet6 addr: 2001:a61:344e:d700:1f75:15ef:50f9:2259/64 Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:202 errors:0 dropped:0 overruns:0 frame:0
          TX packets:132 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:35111 (35.1 KB)  TX bytes:22969 (22.9 KB)
          Interrupt:43 

eth1      Link encap:Ethernet  HWaddr 00:a0:c9:00:00:00  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Memory:50200000-502fffff 

eth2      Link encap:Ethernet  HWaddr 00:a0:c9:00:00:02  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Memory:50500000-505fffff 

eth3      Link encap:Ethernet  HWaddr 00:a0:c9:00:00:01  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Memory:50800000-508fffff 

eth4      Link encap:Ethernet  HWaddr 00:a0:c9:00:00:06  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Memory:50b00000-50bfffff 

eth5      Link encap:Ethernet  HWaddr 00:a0:c9:00:00:05  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Memory:50e00000-50efffff 

eth6      Link encap:Ethernet  HWaddr 00:a0:c9:00:00:07  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Memory:51100000-511fffff 

eth7      Link encap:Ethernet  HWaddr 00:a0:c9:00:00:03  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Memory:51400000-514fffff 

eth8      Link encap:Ethernet  HWaddr 00:a0:c9:00:00:04  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Memory:51700000-517fffff 

l4tbr0    Link encap:Ethernet  HWaddr 02:da:45:2a:a9:98  
          inet addr:192.168.55.1  Bcast:192.168.55.255  Mask:255.255.255.0
          inet6 addr: fe80::505d:a8ff:fefd:6a52/64 Scope:Link
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:704 (704.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:234 errors:0 dropped:0 overruns:0 frame:0
          TX packets:234 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:17606 (17.6 KB)  TX bytes:17606 (17.6 KB)

usb0      Link encap:Ethernet  HWaddr 02:da:45:2a:a9:98  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

usb1      Link encap:Ethernet  HWaddr 4e:9a:50:ca:13:78  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

wlan0     Link encap:Ethernet  HWaddr 00:04:4b:8c:9a:dc  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

nvidia@tegra-ubuntu:~$ dmesg | grep igb
[    2.661988] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.3.0-k
[    2.661990] igb: Copyright (c) 2007-2014 Intel Corporation.
[    8.988515] igb 0000:04:00.0: enabling device (0000 -> 0002)
[    9.021911] igb 0000:04:00.0: added PHC on eth1
[    9.021914] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
[    9.021918] igb 0000:04:00.0: eth1: (PCIe:2.5Gb/s:Width x1) 00:a0:c9:00:00:00
[    9.022016] igb 0000:04:00.0: eth1: PBA No: 000300-000
[    9.022019] igb 0000:04:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[    9.022220] igb 0000:05:00.0: enabling device (0000 -> 0002)
[    9.056797] igb 0000:05:00.0: added PHC on eth2
[    9.056800] igb 0000:05:00.0: Intel(R) Gigabit Ethernet Network Connection
[    9.056804] igb 0000:05:00.0: eth2: (PCIe:2.5Gb/s:Width x1) 00:a0:c9:00:00:02
[    9.056895] igb 0000:05:00.0: eth2: PBA No: 000300-000
[    9.056899] igb 0000:05:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[    9.057106] igb 0000:06:00.0: enabling device (0000 -> 0002)
[    9.091316] igb 0000:06:00.0: added PHC on eth3
[    9.091318] igb 0000:06:00.0: Intel(R) Gigabit Ethernet Network Connection
[    9.091322] igb 0000:06:00.0: eth3: (PCIe:2.5Gb/s:Width x1) 00:a0:c9:00:00:01
[    9.091412] igb 0000:06:00.0: eth3: PBA No: 000300-000
[    9.091415] igb 0000:06:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[    9.091576] igb 0000:07:00.0: enabling device (0000 -> 0002)
[    9.132251] igb 0000:07:00.0: added PHC on eth4
[    9.132254] igb 0000:07:00.0: Intel(R) Gigabit Ethernet Network Connection
[    9.132258] igb 0000:07:00.0: eth4: (PCIe:2.5Gb/s:Width x1) 00:a0:c9:00:00:06
[    9.132347] igb 0000:07:00.0: eth4: PBA No: 000300-000
[    9.132350] igb 0000:07:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[    9.132550] igb 0000:08:00.0: enabling device (0000 -> 0002)
[    9.164487] igb 0000:08:00.0: added PHC on eth5
[    9.164490] igb 0000:08:00.0: Intel(R) Gigabit Ethernet Network Connection
[    9.164494] igb 0000:08:00.0: eth5: (PCIe:2.5Gb/s:Width x1) 00:a0:c9:00:00:05
[    9.164598] igb 0000:08:00.0: eth5: PBA No: 000300-000
[    9.164601] igb 0000:08:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[    9.164788] igb 0000:09:00.0: enabling device (0000 -> 0002)
[    9.198576] igb 0000:09:00.0: added PHC on eth6
[    9.198579] igb 0000:09:00.0: Intel(R) Gigabit Ethernet Network Connection
[    9.198603] igb 0000:09:00.0: eth6: (PCIe:2.5Gb/s:Width x1) 00:a0:c9:00:00:07
[    9.198693] igb 0000:09:00.0: eth6: PBA No: 000300-000
[    9.198696] igb 0000:09:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[    9.198896] igb 0000:0a:00.0: enabling device (0000 -> 0002)
[    9.232130] igb 0000:0a:00.0: added PHC on eth7
[    9.232132] igb 0000:0a:00.0: Intel(R) Gigabit Ethernet Network Connection
[    9.232136] igb 0000:0a:00.0: eth7: (PCIe:2.5Gb/s:Width x1) 00:a0:c9:00:00:03
[    9.232225] igb 0000:0a:00.0: eth7: PBA No: 000300-000
[    9.232228] igb 0000:0a:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[    9.232390] igb 0000:0b:00.0: enabling device (0000 -> 0002)
[    9.266234] igb 0000:0b:00.0: added PHC on eth8
[    9.266236] igb 0000:0b:00.0: Intel(R) Gigabit Ethernet Network Connection
[    9.266241] igb 0000:0b:00.0: eth8: (PCIe:2.5Gb/s:Width x1) 00:a0:c9:00:00:04
[    9.266355] igb 0000:0b:00.0: eth8: PBA No: 000300-000
[    9.266359] igb 0000:0b:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[   28.766393] igb 0000:04:00.0 eth1: PCIe link lost, device now detached
[   28.781487] igb 0000:0b:00.0 eth8: PCIe link lost, device now detached
[   28.826327] igb 0000:05:00.0 eth2: PCIe link lost, device now detached
[   29.112805] igb 0000:0a:00.0 eth7: PCIe link lost, device now detached
[   29.158605] igb 0000:06:00.0 eth3: PCIe link lost, device now detached
[   29.427119] igb 0000:09:00.0 eth6: PCIe link lost, device now detached
[   29.741281] igb 0000:08:00.0 eth5: PCIe link lost, device now detached
[   30.055445] igb 0000:07:00.0 eth4: PCIe link lost, device now detached

The complete log is attached as a file.

log_TX2.txt (89.2 KB)

vidyas · September 15, 2019, 9:25am

Apologies for not understanding comment #1 correctly. NVMe device is missing in first lspci output clearly.
Is this issue resolved yet?
to me, it looks like there is something wrong with the switch operation as the device that is not getting enumerated (NVMe) is downstream of the switch. Could it be that switch is getting overheated? (Typically it is the case with PCIe switches). Did you try keeping a fan on it?