PCI-e Quad Port NIC problem, same but different

Autonomous Machines Jetson & Embedded Systems Jetson TX1
1

We have a number of these Intel Quad NICs on our Jetson PCI-e slots that we have been working with. The first ones (Intel PT EXPI9404PT) were very problematic; we never did get lspci to recognize them even with different kernels as suggested here. The other ones we tried (the VTs) seem to work a little better. Here is lspci -vvv output…

03:00.0 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
Subsystem: Intel Corporation Gigabit VT Quad Port Server Adapter
03:00.1 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
Subsystem: Intel Corporation Gigabit VT Quad Port Server Adapter
04:00.0 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
Subsystem: Intel Corporation Gigabit VT Quad Port Server Adapter
04:00.1 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
Subsystem: Intel Corporation Gigabit VT Quad Port Server Adapter

Loads the module fine and takes an address

[ 9.942002] Intel(R) Gigabit Ethernet Network Driver - version 5.3.5.4
[ 9.942009] Copyright (c) 2007-2015 Intel Corporation.
[ 10.218406] igb 0000:03:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 10.488666] igb 0000:03:00.1: Intel(R) Gigabit Ethernet Network Connection
[ 10.758666] igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 11.040018] igb 0000:04:00.1: Intel(R) Gigabit Ethernet Network Connection

but then it suffers from TX hangs…

[ 2036.025490] igb 0000:04:00.0: Detected Tx Unit Hang
Tx Queue <0>
TDH <0>
TDT <1>
next_to_use <1>
next_to_clean <0>
buffer_info[next_to_clean]
time_stamp <10002a387>
next_to_watch
jiffies <10002a61e>
desc.status <1ac000>

Looking into driver solutions but figured I would ask here as well…

2

Note that lspci will not actually give you verbose output without “sudo”. Assuming lspci shows verbose output there will likely be a line near the end showing “Kernel modules: …”. Assuming a module is used (versus integrated feature) this would be useful to see, along with the “Kernel driver in use” line.

Should “ifconfig” show any output for the device this too would be nice to see (unlikely if the driver doesn’t load properly, otherwise it might give some error information).

3

This is typical lspci output…

03:00.0 Ethernet controller: Intel Corporation 82575GB Gigabit Network Connection (rev 02)
Subsystem: Intel Corporation Gigabit VT Quad Port Server Adapter
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 130
Region 0: Memory at 13400000 (32-bit, non-prefetchable)
Region 1: Memory at 13000000 (32-bit, non-prefetchable)
Region 2: I/O ports at 1000 [disabled]
Region 3: Memory at 13440000 (32-bit, non-prefetchable)
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [60] MSI-X: Enable+ Count=10 Masked-
Vector table: BAR=3 offset=00000000
PBA: BAR=3 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #2, Speed 2.5GT/s, Width x4, ASPM not supported, Exit Latency L0s <4us, L1 <64us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 16ms to 55ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr+ BadTLP+ BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Capabilities: [140 v1] Device Serial Number 00-1b-21-ff-ff-24-05-f8
Kernel driver in use: igb
Kernel modules: igb

I have tried it with a slightly older igb compiled into the kernel as well with similar effect…

Here is some ifconfig info but no luck with ping… and the link lights clearly indicate resetting… I can ping myself but nothing else on that subnet, even non-Jetsons / non-Intel NICs…

root@tegra:~# ifconfig eth3
eth3 Link encap:Ethernet HWaddr 00:1b:21:24:05:fc
inet addr:192.168.5.5 Bcast:192.168.5.255 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

4

The lspci indicates PCIe sees the device as gen. 1 on 4 lanes and it operates at gen. 1 speeds on 4 lanes. I don’t think any PCIe errors were fatal.

It does seem that only the igb driver is involved for the rest of the NIC function.

ifconfig does not indicate any error…it also does not indicate any packets even attempted to use the interface. I’m wondering if there is a route issue. The “Detected Tx Unit Hang” message could be something that shows up before a packet ever gets a chance to use the interface. What happens if you ping the address of that interface (in this case 192.168.5.5)? You could also attempt to ssh from that machine to its 192.168.5.5 interface to see what TCP does (versus ping/ICMP). After a ping do any more error messages show up in dmesg, and does “RX packets” or “TX packets” of ifconfig change? Also, what does “sudo route” show?

5

I will try setting a couple of static routes widely separated… at least the 4 NICs are consistent in their behavior. And they are enumerating and assigning correctly. Will post an update in the am.

6

Routing table ( all 4 boards identical) looks pretty correct…

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.4.1 0.0.0.0 UG 0 0 0 eth4
192.168.0.0 * 255.255.255.0 U 0 0 0 eth0
192.168.4.0 * 255.255.255.0 U 0 0 0 eth4
192.168.1.0 * 255.255.255.0 U 0 0 0 eth1
192.168.2.0 * 255.255.255.0 U 0 0 0 eth2
192.168.3.0 * 255.255.255.0 U 0 0 0 eth3

Here are 2 NICs I am trying to pair…

Node 5 eth4

eth4 Link encap:Ethernet HWaddr 00:1b:21:24:05:fd
inet addr:192.168.4.5 Bcast:192.168.4.255 Mask:255.255.255.0
inet6 addr: fe80::21b:21ff:fe24:5fd/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:15033 overruns:15033 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

Node 4 eth 4

eth4 Link encap:Ethernet HWaddr 00:1b:21:24:0a:bd
inet addr:192.168.4.4 Bcast:192.168.4.255 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

I did notice that lsmod never says igb is being used…

root@Node4:~# lsmod
Module Size Used by
igb 189049 0

maybe I can explicitly attach it to eth4 in modules.conf…

7

This particular error from eth4 strongly suggests a configuration issue:

RX packets:0 errors:0 dropped:15033 overruns:15033 frame:0

If there were a hardware error you might see “errors” or “frame” go up, but these are zero. For whatever reason every single packet received in eth4 was dropped…apparently because the buffer filled and nothing took the data. If another machine separate from this computer were misconfigured you would also likely see collisions, but with no collisions, it appears to be an error limited to this one computer.

Note that eth4 is the default route, so any address not explicitly falling to an exact interface and also not otherwise having a matching network/netmask would go here (meaning that packets designated for a particular interface would go there, perhaps even to eth4 if this was the proper network/netmask combination, but packets to a missing network/netmask combination also go to eth4).

I can’t be sure, but I suspect your routing table is part (but not all) of the problem. Notice that typically two separate NICs would tend to be on separate subnets (an out of date term, but it is still correct). So a combination of a network (such as 192.168.4.5) and a netmask (such as 255.255.255.0 or “/24”) leaves a range of addresses to which the NIC will respond (such as 192.168.4.0 through 192.168.4.255 for eth4). Other NICs should not overlap that range, though an exact address match for exactly one address should work.

The trouble is that all of those addresses are “255.255.255.0” or “/24” netmask within the same subnet, which means every single NIC is configured to handle all traffic in that network range. Outside of the 192.168.4.x subnet the default route gets the traffic, e.g., if pointing at google.com’s 216.58.194.206 address…within the eth4 network/netmask every single NIC is configured to handle the traffic…but there can be only one.

I believe the hardware is probably working correctly. As an experiment you can manually (temporarily) disable the other NIC interfaces on that card while leaving the interface with the default route up. Under “route” the “default” interface should be left alone, and on the others, use ifconfig to disable those interfaces (I’m hoping something like NetworkManager doesn’t get in the way):

sudo -s
# if the address is inside of 192.168.4.x...
ifdown eth0
ifdown eth1
ifdown eth2
ifdown eth3
...
# verify only one NIC is on that subnet and that the NIC is the default route handler:
ifconfig
route
exit

You may find that the NIC starts working if there is an outside computer in the 192.168.4.x subnet. While figuring out your addressing you could also purposely move each of the other NICs to their own “/24” subnet, e.g., to “192.168.5.1/255.255.255.0”, “192.168.6.1/255.255.255.0”, “192.168.7.1/255.255.255.0”, so on. You could also split handling of the same subnet to a series of smaller subnets, e.g., something within “192.168.4.x”, but with netmask of “255.255.255.64” in combination with four new unique non-overlapping addresses to cover the full 32-bits of that network using each of the four NICs without conflicting overlaps.

8

I have all of the eth4s on one switch, all of the eth3s on another and so on… am going to attempt a number of different network experiments eventually (like channel bonding)… I am going to try different ethernet cables and switches. Maybe I should try a layer 2 tool to see if there is any activity there…

9

Regardless of whether those interfaces go to different switches or not the routing must be valid. Inside the kernel there is still invalid configuration if the networks/netmasks overlap on one machine.

I believe bonding is a special case and requires a virtual interface, and then addresses and netmasks are assigned to the virtual interface (the eth0 through ethN participating in the bond would not have an IP address nor a netmask…only the MAC address or device special file would be useful to refer to the particular port). To succeed I strongly recommend getting each of the four interfaces up with only one active port at a time, removing IP address and netmask configuration of all others for each of the four physical ports to be tested…this would verify whether the NIC is functioning correctly and prove whether it is a NIC/driver issue versus configuration issue. Once each individual port works one at a time with IP/netmask remove all of eth0 through ethN IP/netmask configurations and create the bonding configuration. Once ports lack an IP address the virtual interface be assigned IP/netmask.

For reference, see:
[url]https://www.kernel.org/doc/Documentation/networking/bonding.txt[/url]

In particular, look in that URL at “4.2 Network configuration”. If you look at this example the physical ports (eth0 and eth1) are never assigned an IP address or netmask. Only the virtual interface (bond0) is assigned an IP address or netmask.

10

There is no network activity in any address or netmask scenario. Turning off all other NICs (including the onboard) and just attempting to ping between 2 jacks via a direct connect bears no dividends. No link light flashes, NIC resetting every couple of seconds, card really warm… :) These VT cards do work on another MoBo and should be pretty performing if I get them up. I will try some other possible drivers and see if I can at least get some diagnostic info. Like I say, lsmod never said igb was being used, so I will see if i can track that error down. Any other ideas welcome but no worries, we will keep you posted.

11

lsmod only shows a driver if the support was as a module. If the module fails to load lsmod will also not show this. lspci shows kernel driver for PCIe devices, and this will be the associated driver regardless of whether it is a module or not…and if a module, regardless of whether the module loads or not (lsmod looks at the module name…lspci kernel driver looks at the driver name).

One point about conflicting configuration is that it can prevent packets from ever touching the NIC. There can also be a race condition…one that consumes power and heats up the NIC. As for the MoBo the NIC works with…I’d bet configuration differs. 2 jacks with conflicting config would cause this, so unless the two jacks were configured correctly this would be expected to show no packets.