Hi!
I work on a stereo system based on two GbE cameras and Jetson TX2. I use Connect Tech Elroy carrier and MPG104 mini PCIe card with two additional GbE ports. This, in theory, should give me three GbE ports, two on the PCIe card are being used for the cameras and the third one (on the carrier board) works as an interface to another computer. When I test the bandwidth using iperf all three ports have full bandwidth, however when the cameras are working I loose bandwidth on the third port. It looks like even though these are separate cards, the bandwidth is shared between them. Have you encountered this problem? How to fix it?
Thanks!
Tomasz
How many lanes are the added ports using? What do you see from:
sudo lspci -vvv
…need to know how many PCIe lanes are being used, and at what speed.
Also, make sure you have performance maxed:
sudo nvpmodel -m 0
sudo jetson_clocks
The above commands might differ if you are using an older release. Check your release via:
head -n 1 /etc/nv_tegra_release
I have Jetpack 3.3 installed, it runs in nvpmodel 0 and with jetson_clocks.sh.
Here is the output from the console:
nvidia@tegra-ubuntu:~$ sudo lspci -vvv
00:01.0 PCI bridge: NVIDIA Corporation Device 10e5 (rev a1) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 388
Bus: primary=00, secondary=01, subordinate=04, sec-latency=0
I/O behind bridge: 00001000-00002fff
Memory behind bridge: 50100000-502fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Subsystem: NVIDIA Corporation Device 0000
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/2 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
Mapping Address Base: 00000000fee00000
Capabilities: [80] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Off, PwrInd On, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet+ LinkState+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Kernel driver in use: pcieport
01:00.0 PCI bridge: Pericom Semiconductor Device 2303 (rev 05) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Bus: primary=01, secondary=02, subordinate=04, sec-latency=0
I/O behind bridge: 00001000-00002fff
Memory behind bridge: 50100000-502fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [5c] Vital Product Data
Unknown small resource type 00, will not decode more.
Capabilities: [64] Vendor Specific Information: Len=34 <?>
Capabilities: [b0] Subsystem: Device 0000:0000
Capabilities: [c0] Express (v2) Upstream Port, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM not supported, Exit Latency L0s <512ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=4
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=04 MaxTimeSlots=64 RejSnoopTrans-
Arb: Fixed+ WRR32- WRR64- WRR128+ TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Port Arbitration Table <?>
Capabilities: [20c v1] Power Budgeting <?>
Capabilities: [230 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Kernel driver in use: pcieport
02:01.0 PCI bridge: Pericom Semiconductor Device 2303 (rev 05) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin ? routed to IRQ 448
Bus: primary=02, secondary=03, subordinate=03, sec-latency=0
I/O behind bridge: 00001000-00001fff
Memory behind bridge: 50100000-501fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [4c] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 000000026b4cc000 Data: 0000
Capabilities: [64] Vendor Specific Information: Len=34 <?>
Capabilities: [b0] Subsystem: Device 0000:0000
Capabilities: [c0] Express (v2) Downstream Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #1, Speed 2.5GT/s, Width x1, ASPM not supported, Exit Latency L0s <512ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot+ ASPMOptComp-
LnkCtl: ASPM Disabled; Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Off, PwrInd Off, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet- LinkState+
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -3.5dB
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=4
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=04 MaxTimeSlots=64 RejSnoopTrans-
Arb: Fixed+ WRR32- WRR64- WRR128+ TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Port Arbitration Table <?>
Capabilities: [20c v1] Power Budgeting <?>
Capabilities: [220 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Kernel driver in use: pcieport
02:02.0 PCI bridge: Pericom Semiconductor Device 2303 (rev 05) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin ? routed to IRQ 449
Bus: primary=02, secondary=04, subordinate=04, sec-latency=0
I/O behind bridge: 00002000-00002fff
Memory behind bridge: 50200000-502fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [4c] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 000000026b4cc000 Data: 0001
Capabilities: [64] Vendor Specific Information: Len=34 <?>
Capabilities: [b0] Subsystem: Device 0000:0000
Capabilities: [c0] Express (v2) Downstream Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #2, Speed 2.5GT/s, Width x1, ASPM not supported, Exit Latency L0s <512ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot+ ASPMOptComp-
LnkCtl: ASPM Disabled; Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Off, PwrInd Off, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet- LinkState+
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -3.5dB
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=4
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=04 MaxTimeSlots=64 RejSnoopTrans-
Arb: Fixed+ WRR32- WRR64- WRR128+ TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Port Arbitration Table <?>
Capabilities: [20c v1] Power Budgeting <?>
Capabilities: [220 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Kernel driver in use: pcieport
03:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
Subsystem: Intel Corporation 82574L Gigabit Network Connection
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 388
Region 0: Memory at 50100000 (32-bit, non-prefetchable)
Region 2: I/O ports at 1000 [disabled]
Region 3: Memory at 50120000 (32-bit, non-prefetchable)
Capabilities: [c8] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [e0] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #1, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <128ns, L1 <64us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
Vector table: BAR=3 offset=00000000
PBA: BAR=3 offset=00002000
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Capabilities: [140 v1] Device Serial Number 00-0c-8b-ff-ff-55-06-35
Kernel driver in use: e1000e
04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
Subsystem: Intel Corporation 82574L Gigabit Network Connection
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 388
Region 0: Memory at 50200000 (32-bit, non-prefetchable)
Region 2: I/O ports at 2000 [disabled]
Region 3: Memory at 50220000 (32-bit, non-prefetchable)
Capabilities: [c8] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [e0] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #2, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <128ns, L1 <64us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
Vector table: BAR=3 offset=00000000
PBA: BAR=3 offset=00002000
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Capabilities: [140 v1] Device Serial Number 00-0c-8b-ff-ff-54-06-35
Kernel driver in use: e1000e
To also see the layout what is the output from “lspci” and “lspci -t” (the original “sudo lspci -vvv” provides device details, “lspci -t” will show branches sharing bandwidth, but “lspci” will be needed to say which abbreviation/number listed in “lspci -t” is which)?
There are a number of bridges listed not taken into account yet (each running without any listed PCIe error, but unknown as yet how they are participating in the chain of devices). For the e1000e driver, each of the two NICs are listed as using a single lane with gen. 1 speeds (2.5GT/s implies a theoretical 2GT/s over the one lane after considering the 8b10b encoding scheme, but not other overhead). Assuming the two are not bridged to the same single lane there should be no issue with achieving 1Gbit/s (if both ran from a single 2.5GT/s lane then there would be some bottleneck to performance, but probably not a huge hit if everything is running at 100%).
Additionally, no PCIe errors are reported for the two gigabit controllers. Could you also post the output of “ifconfig” so we can tell if there are network errors? If neither errors on ethernet, nor PCIe, then it comes down to purely throughput (such as sharing a PCIe lane) or whatever consumes the data (e.g., a CPU bottleneck). Generally speaking, the Intel controllers are about as good as it gets for not loading the CPU, but these could still be limited by configuration or any network switch.
Is there a network switch involved for the cameras? Or do they go straight to the NICs?
Here is the output from the lscpi and lspci -t:
nvidia@tegra-ubuntu:~$ lspci
00:01.0 PCI bridge: NVIDIA Corporation Device 10e5 (rev a1)
01:00.0 PCI bridge: Pericom Semiconductor Device 2303 (rev 05)
02:01.0 PCI bridge: Pericom Semiconductor Device 2303 (rev 05)
02:02.0 PCI bridge: Pericom Semiconductor Device 2303 (rev 05)
03:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
nvidia@tegra-ubuntu:~$ lspci -t
-[0000:00]---01.0-[01-04]----00.0-[02-04]--+-01.0-[03]----00.0
\-02.0-[04]----00.0
nvidia@tegra-ubuntu:~$
And here is ifconfig:
nvidia@tegra-ubuntu:~$ ifconfig
docker0 Link encap:Ethernet HWaddr 02:42:2f:b3:80:e2
inet addr:172.17.0.1 Bcast:172.17.255.255 Mask:255.255.0.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
eth0 Link encap:Ethernet HWaddr 00:0c:8b:55:06:35
inet addr:169.254.77.90 Bcast:169.254.77.255 Mask:255.255.255.0
inet6 addr: fe80::82f:26c:938e:52b3/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:6 errors:0 dropped:0 overruns:0 frame:0
TX packets:59 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:384 (384.0 B) TX bytes:6995 (6.9 KB)
Interrupt:132 Memory:50100000-50120000
eth1 Link encap:Ethernet HWaddr 00:0c:8b:54:06:35
inet addr:169.254.78.92 Bcast:169.254.78.255 Mask:255.255.255.0
inet6 addr: fe80::50bd:35eb:413d:3303/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:6 errors:0 dropped:0 overruns:0 frame:0
TX packets:31 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:384 (384.0 B) TX bytes:3487 (3.4 KB)
Interrupt:132 Memory:50200000-50220000
eth2 Link encap:Ethernet HWaddr 00:04:4b:dd:5f:d7
inet addr:169.254.140.94 Bcast:169.254.140.255 Mask:255.255.255.0
inet6 addr: fe80::8b5:6704:9a85:fb67/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:120 errors:0 dropped:0 overruns:0 frame:0
TX packets:103 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:13267 (13.2 KB) TX bytes:13795 (13.7 KB)
Interrupt:42
l4tbr0 Link encap:Ethernet HWaddr aa:02:4e:e6:1f:08
inet addr:192.168.55.1 Bcast:192.168.55.255 Mask:255.255.255.0
inet6 addr: fe80::c4b0:96ff:fe53:732e/64 Scope:Link
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:744 (744.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:1956 errors:0 dropped:0 overruns:0 frame:0
TX packets:1956 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:120704 (120.7 KB) TX bytes:120704 (120.7 KB)
usb0 Link encap:Ethernet HWaddr aa:02:4e:e6:1f:08
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
usb1 Link encap:Ethernet HWaddr ae:3a:20:08:04:04
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
wlan0 Link encap:Ethernet HWaddr 00:04:4b:dd:5f:d5
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
eth0 and eth1 are the cameras, eth2 is the communication link.
There is no switch in the system, cameras are connected directly and so is the other computer. When both cameras are working there is plenty of resources left on the CPU so the bottleneck must be somewhere else.
This is a bit long, I’ll add a question summary at the end. Skip to that if you get bored :P
It looks like all PCIe is working (performance might or might not suffer, but there is no outright failure).
Now we should look at the output of the “route” command…add that information. Also, add the output of “ethtool eth0”, “ethtool eth1”, and “ethtool eth2”. If you don’t have ethtool, then “sudo apt-get install ethtool”.
Also, give exact details on what ports have routers, and which ports go through a network switch. Give details on any information you have about how the cameras get their IP address, and whether you know of options where the camera acts as a router, or if address must be assigned by the TX2. Are there any special commands for address assignment mandated by the cameras? I’ll be answering some questions below on the assumption that there is a router, but the cameras may change the situation. The cameras could just be an end point such that the ethernet would need address assignment via a router, or the cameras might actually provide DHCP and assign the address to the port.
If cameras can act as a DHCP server to assign address, then probably the cameras have some sort of option which needs to be used for this. If no such option exists on the cameras, then you would need router setup (indications are that all router setup failed…explained below via the 169.254.x.x address comments…those comments are only true if the cameras themselves did not assign the 169.254.x.x addresses). I have no knowledge of what options these cameras have related to DHCP and address assignment.
Comments on the MTU: The 9000 MTU is perfectly acceptable, and could help performance, but not all network devices will work with this. For devices which do work with this, then I would expect improved performance for large packets (such as from a camera…although low resolution devices with smaller network packets would have reduced performance while waiting to fill up the 9000 bytes before sending…I can’t imagine a camera that low performance, but hardware compression could actually reduce packet size below something triggered at 9000 bytes). If some component in the network does not work with this (e.g., not all switches will support 9000), then this would be a cause of failure or degraded performance. Did you manually set to 9000? I ask because the default is normally 1500. If you have a way to adjust this, then you might try with 1500, but if 9000 works without total failure, then perhaps it isn’t even needed to try with 1500. Do these cameras support hardware compression?
Comments on the ifconfig: No errors of any kind appear. The actual driver for the PHY has a small amount of traffic working as it should. eth0 and eth1 share IRQ 132, and are apparently the ethernet on the add-on card (this confirms, through IRQ, that drivers agree eth0 and eth1 are the cameras). This means PCIe is functioning and no aspect of performance reduction is from an outright error throughout the entire chain of camera->PCIe->ethernet driver. However, the amount of bytes of traffic on those two interfaces is quite low. The amount of traffic is approximately consistent with DHCP traffic assigning an address, and then no traffic. If the ifconfig was given before you had tried to use the cameras, then you might run the cameras for a minute and post the new “ifconfig” to see what happens with more traffic.
However, normally the assigned 169.254.x.x address range indicates there was a failure to actually get an address. The attempt was made, and perhaps something even replied, but if there was a reply, then the reply was a denial of address. This might not be the case for you, explained below.
The 169.254.x.x address indication of failure is also true of eth2, and eth2 is the network which is part of the default module. However, this has more traffic. This is apparently the default route and failed traffic went through this ethernet PHY, but never got a response (unless the camera assigned this).
I am unsure about the lspci tree, but the Elroy carrier board probably changes this relative to what I’m used to on the devel carrier. I’m going to suggest a bit more verbose version, “lspci -tv”, though I doubt this will add any new information.
Question Summary:
- Let the TX2 run for a minute to add to net traffic (use the camera for a short time to cause more traffic statistics). Then run "ifconfig" again.
- After the new "ifconfig", post the content of "route".
- Post the content of the more verbose "lspci -tv".
- Describe all switch and router wiring.
- Describe how the cameras get their address, or if the cameras are acting as a router.
- Do any routers involved require devices be on a MAC address list to allow address assignment (doesn't apply if you don't have a router)?
- Was the 9000 MTU something you adjusted? Do the cameras have hardware compression? What resolution are the cameras?
- Do you know if the 169.254.x.x addresses were intentional?
- Do any network switches involved not support jumbo frames?
Thanks a lot for the detailed answer!
Regarding the wiring: there are no switches or routers in the system. The cameras are connected directly to the ethernet cards. The computer is also directly connected to the ethernet port on the carrier board.
All IPs are set manually, cameras require addresses in the 169.254.x.x range. MTUs are set to 9000 manually as this was recommended by the camera manufacturer. These are 5MP cameras with no compression, so ~20 Hz saturates the GbE channel.
ifconfig:
nvidia@tegra-ubuntu:~$ ifconfig
docker0 Link encap:Ethernet HWaddr 02:42:78:b8:b7:b4
inet addr:172.17.0.1 Bcast:172.17.255.255 Mask:255.255.0.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
eth0 Link encap:Ethernet HWaddr 00:0c:8b:55:06:35
inet addr:169.254.77.90 Bcast:169.254.77.255 Mask:255.255.255.0
inet6 addr: fe80::82f:26c:938e:52b3/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:4667138 errors:0 dropped:0 overruns:0 frame:0
TX packets:926 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:41879514010 (41.8 GB) TX bytes:60669 (60.6 KB)
Interrupt:132 Memory:50100000-50120000
eth1 Link encap:Ethernet HWaddr 00:0c:8b:54:06:35
inet addr:169.254.78.92 Bcast:169.254.78.255 Mask:255.255.255.0
inet6 addr: fe80::50bd:35eb:413d:3303/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:4671188 errors:3144 dropped:0 overruns:0 frame:1572
TX packets:2529 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:41911932664 (41.9 GB) TX bytes:168887 (168.8 KB)
Interrupt:132 Memory:50200000-50220000
eth2 Link encap:Ethernet HWaddr 00:04:4b:dd:5f:d7
inet addr:169.254.140.94 Bcast:169.254.140.255 Mask:255.255.255.0
inet6 addr: fe80::8b5:6704:9a85:fb67/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:1608393 errors:0 dropped:0 overruns:0 frame:0
TX packets:489650 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:106181023 (106.1 MB) TX bytes:61870686351 (61.8 GB)
Interrupt:42
l4tbr0 Link encap:Ethernet HWaddr 12:40:9b:62:cc:ab
inet addr:192.168.55.1 Bcast:192.168.55.255 Mask:255.255.255.0
inet6 addr: fe80::34a1:7fff:fe14:2fe5/64 Scope:Link
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:857 (857.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:3033 errors:0 dropped:0 overruns:0 frame:0
TX packets:3033 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:427087 (427.0 KB) TX bytes:427087 (427.0 KB)
usb0 Link encap:Ethernet HWaddr 12:8d:5e:20:bd:4f
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
usb1 Link encap:Ethernet HWaddr 12:40:9b:62:cc:ab
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
wlan0 Link encap:Ethernet HWaddr 00:04:4b:dd:5f:d5
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
route:
nvidia@tegra-ubuntu:~$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default tomasz-XPS-15-9 0.0.0.0 UG 100 0 0 eth2
link-local * 255.255.0.0 U 1000 0 0 l4tbr0
169.254.77.0 * 255.255.255.0 U 100 0 0 eth0
169.254.78.0 * 255.255.255.0 U 100 0 0 eth1
169.254.140.0 * 255.255.255.0 U 100 0 0 eth2
172.17.0.0 * 255.255.0.0 U 0 0 0 docker0
192.168.55.0 * 255.255.255.0 U 0 0 0 l4tbr0
lspci -tv:
nvidia@tegra-ubuntu:~$ lspci -tv
-[0000:00]---01.0-[01-04]----00.0-[02-04]--+-01.0-[03]----00.0 Intel Corporation 82574L Gigabit Network Connection
\-02.0-[04]----00.0 Intel Corporation 82574L Gigabit Network Connection
ehtool output:
nvidia@tegra-ubuntu:~$ ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: off (auto)
Cannot get wake-on-lan settings: Operation not permitted
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
nvidia@tegra-ubuntu:~$ ethtool eth1
Settings for eth1:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: on (auto)
Cannot get wake-on-lan settings: Operation not permitted
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
nvidia@tegra-ubuntu:~$ ethtool eth2
Settings for eth2:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Link partner advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Link partner advertised pause frame use: Symmetric Receive-only
Link partner advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 0
Transceiver: external
Auto-negotiation: on
Cannot get wake-on-lan settings: Operation not permitted
Link detected: yes
I’ve also changed the buffer sizes (this was also recommended by the manufacturer):
nvidia@tegra-ubuntu:~$ ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
Current hardware settings:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 256
nvidia@tegra-ubuntu:~$ ethtool -g eth1
Ring parameters for eth1:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
Current hardware settings:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 256
This camera has a problem:
eth1 Link encap:Ethernet HWaddr 00:0c:8b:54:06:35
inet addr:169.254.78.92 Bcast:169.254.78.255 Mask:255.255.255.0
inet6 addr: fe80::50bd:35eb:413d:3303/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:4671188 <u><i><b>errors:3144</b></i></u> dropped:0 overruns:0 <i><u><b>frame:1572</b></u></i>
TX packets:2529 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:41911932664 (41.9 GB) TX bytes:168887 (168.8 KB)
Interrupt:132 Memory:50200000-50220000
“errors:3144” means it isn’t a network throughput issue, the problem is something else which is more serious. You’re receiving malformed frames at the Jetson. This could be something the camera sends, or it could be the cable. A cable can be perfectly good to DC, or even to actual networking at shorter distances, and then go bad at longer distances…some higher quality cables can work at longer distances than some shorter lower quality cables. There are no TX errors, this is strictly a reception from camera to Jetson. A bent connector pin might even make poor contact.
Buffer size is the same on both, so this isn’t an issue since it works on one. All adapter settings are the same for the Intel NIC (using the same driver with the same IRQ), and I see no sign of PCIe issue.
From this point forward your interest will only be for eth1. Now if you swap cameras on the same cables (we’re trying to see if it follows a particular component, so don’t switch both cables and camera at the same time), does the issue follow the camera? Or does the issue stay with the particular interface? If swapping cameras keeps the errors on a given interface, then swap the cameras back, verify that interface maintains the error, and then swap the cables themselves.
One more suggestion: Closely examine all connectors to look for damage.
Hi,
I’m not sure if this is the case: all cables are new, cat6 and only 20cm long. Furthermore, the errors appeared last time when the cameras were running at 10Hz and the output was transmitted over eth2. Assuming that the bandwidth is shared this exceeds 1GbE. When the cameras are running at 5Hz the output is:
eth0 Link encap:Ethernet HWaddr 00:0c:8b:55:06:35
inet addr:169.254.77.90 Bcast:169.254.77.255 Mask:255.255.255.0
inet6 addr: fe80::82f:26c:938e:52b3/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:1276846 errors:0 dropped:0 overruns:0 frame:0
TX packets:654 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:11454767352 (11.4 GB) TX bytes:45164 (45.1 KB)
Interrupt:132 Memory:50100000-50120000
eth1 Link encap:Ethernet HWaddr 00:0c:8b:54:06:35
inet addr:169.254.78.92 Bcast:169.254.78.255 Mask:255.255.255.0
inet6 addr: fe80::50bd:35eb:413d:3303/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:1281905 errors:1 dropped:0 overruns:0 frame:1
TX packets:617 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:11497935116 (11.4 GB) TX bytes:42718 (42.7 KB)
Interrupt:132 Memory:50200000-50220000
eth2 Link encap:Ethernet HWaddr 00:04:4b:dd:5f:d7
inet addr:169.254.140.94 Bcast:169.254.140.255 Mask:255.255.255.0
inet6 addr: fe80::8b5:6704:9a85:fb67/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:807182 errors:0 dropped:0 overruns:0 frame:0
TX packets:237642 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:53330705 (53.3 MB) TX bytes:29860843759 (29.8 GB)
Interrupt:42
When only one camera (the one connected to eth1) is running, I can run it at 20Hz without errors:
eth1 Link encap:Ethernet HWaddr 00:0c:8b:54:06:35
inet addr:169.254.78.92 Bcast:169.254.78.255 Mask:255.255.255.0
inet6 addr: fe80::50bd:35eb:413d:3303/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:3793718 errors:1 dropped:0 overruns:0 frame:1
TX packets:2810 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:34032104532 (34.0 GB) TX bytes:185838 (185.8 KB)
Interrupt:132 Memory:50200000-50220000
Exceeding bandwidth might produce drops or overruns, but it won’t produce errors. This is definitely not just a case of bandwidth being insufficient.
The case of one camera at 20Hz was not without error:
RX packets:3793718 <b>errors:1</b> dropped:0 overruns:0 <b>frame:1</b>
You had a framing error. On an ethernet under heavy load running for a very long time this should be “0”. In the case of statistics where something fails once in a while, I would expect to see fewer errors when there are fewer samples…but this still guarantees there is something wrong.
Are you able to swap cameras without changing cables? Are you able to swap cables without changing cameras? I would suggest doing both, running under a significant load, and checking where the errors go (or where they don’t go). More stress will imply less testing needed, on average, to see where the error goes.
Note that the receive (RX) was always the fail. Look closely for RX error or frame counts other than “0”.
I checked all the connections and looks like the framing errors are gone. I didn’t really change anything but after re-plugging all the cables it’s ok.
I’ve also installed the irqbalance
apt-get install irqbalance
and the bandwidth issues are also gone.
I think that this topic may be closed now.
Thanks a lot for help!