Third party PCI-E USB 3.0 Expansion card for Xavier Board

Can you suggest any PCI-E USB 3.0 expansion cards supported for Xavier board?

Hi nikhiltagalpallewar,

Our internal is using below card for testing:
[url]USB 3.0 Cards

Thank you for this suggestion. But my requirement is 10 Gbps per port. But the above port is gives me only 5Gbps bandwidth per port.

Can you suggest some other card which satisfies my requirement?

I read Two card from your forum,

  1. HighPoint Technologies RocketU 1344A

https://devtalk.nvidia.com/default/topic/1043831/jetson-agx-xavier/xavier-not-recognizing-pcie-usb-3-1-card/0

and

  1. ASUS USB3.1 card

https://devtalk.nvidia.com/default/topic/1046654/jetson-agx-xavier/jetson-xavier-pcie-add-on-cards-tested/post/5311763/#5311763

Is this cards are compatible with XAVIER?

Hi nikhiltagalpallewar,

Yes, they are verified on Xavier:
[url]https://devtalk.nvidia.com/default/topic/1046654/jetson-agx-xavier/jetson-xavier-pcie-add-on-cards-tested/post/5311763/#5311763[/url]

Our internal is using below card for testing:
4 Port PCIe USB 3.0 Card w/ 4 Channels - USB 3.0 Cards | Add-on Cards & Peripherals | StarTech.com

I’m using this exact StarTech card on Xavier. Frequently, upon boot, this card fails to enumerate on PCIe, lspci shows no Renesas controllers. Attempts to rescan (echo 1 > /sys/bus/pci/rescan) don’t help - the Renesas controllers do show up in PCI but their internal registers can’t be accessed by RC. the only solution is to reboot, sometimes have to do it twice. Once the card enumerates properly - it works just fine.

I was wondering if your internal testers have seen this problem and if they did - how did they they mitigated it ? It is relatively easy to reproduce, just reboot in a loop and check lspci output. I have it failed on average every 8 times or so.

This card may have issues with low power states. Please add ‘pcie_aspm=off’ to the kernel command line.

Hello,

I have recently purchased an Xavier Dev Kit and the PCIe card mentioned above (4 Port PCIe USB 3.0 Card w/ 4 Channels - USB 3.0 Cards) but I am quite new to setting up these systems. The documentation for the card is lacking for setup with a Linux system and while I attempted to plug and play, the Jetson did not pick up that the card was there. Had the card powered by an external supply but this also showed no response.

Would you be able to tell me how to set up this card with the Jetson or refer me to a site with some instructions?

Thanks in advance

Any updates on this topic? I saw that NVIDIA uses this card internally with the Xavier but I am having a similar issue as jgoldenzqq5m.

The card comes up as “PCI bridge: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch” when using lspci. I can’t connect any usb devices to it and use them.

@cmcgu019: Reply at https://devtalk.nvidia.com/default/topic/1072884/jetson-agx-xavier/pcie-card-pexusb3s44v-with-xavier/?offset=2#5435310. Information still useful for the other people of this thread, but better to not mix multiple threads.

FYI, if the particular card is visible with “lspci”, then PCIe is itself working correctly. Normally this should work even if the driver for the particular card is missing. In some cases PCIe cards which are slow to boot won’t be ready in time for the enumeration, but this is almost always with an FPGA or other PCIe card with a full operating system…this should not be an issue for USB expander cards.

If lspci sees the device, but the device is not running, then it means you need to add the driver. Usually this is a simple file copy to the right place in “/lib/modules/$(uname -r)/”, but you’d need to know which driver, and build the module to copy in. Seeing “PCI bridge: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch” pretty much guarantees PCIe is working as intended, and that only the driver is required, but there might be other issues.

When you list your device with lspci you will see a slot designation on the left side of the line. As a contrived example, the slot ID might look like “00.01.1”. Substitute your actual ID, and show the output of this:

sudo lspci -s 00.01.1 -vvv

This will give a verbose output (and you’ll need sudo for full information), and it would be useful to see what that verbose information is (for example, it might mention a driver or bus errors).

Rebooting my Xavier I now see this when I type lspci

0000:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad0 (rev a1)
0000:01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981
0001:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad2 (rev a1)
0001:01:00.0 SATA controller: Marvell Technology Group Ltd. Device 9171 (rev 13)
0005:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad0 (rev a1)
0005:01:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch
0005:02:01.0 PCI bridge: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch
0005:02:02.0 PCI bridge: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch
0005:02:03.0 PCI bridge: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch
0005:02:04.0 PCI bridge: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch
0005:03:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
0005:04:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
0005:05:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
0005:06:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)

Still the USB3.0 ports on the PCIe card do not work with mouse/keyboard.

running what linuxdev suggested outputs this:

sudo lspci -s 0005:01:00.0 -vvv
0005:01:00.0 PCI bridge: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Bus: primary=01, secondary=02, subordinate=06, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: 40000000-403fffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [5c] Vital Product Data
pcilib: sysfs_read_vpd: read failed: Input/output error
		Not readable
	Capabilities: [64] Vendor Specific Information: Len=34 <?>
	Capabilities: [b0] Subsystem: Pericom Semiconductor PI7C9X2G608GP PCIe2 6-Port/8-Lane Packet Switch
	Capabilities: [c0] Express (v2) Upstream Port, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x4, ASPM not supported, Exit Latency L0s <512ns, L1 <1us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [140 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=4
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		Port Arbitration Table [170] <?>
		VC0:	Caps:	PATOffset=04 MaxTimeSlots=64 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128+ TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
			Port Arbitration Table <?>
		VC1:	Caps:	PATOffset=08 MaxTimeSlots=64 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128+ TWRR128+ WRR256-
			Ctrl:	Enable- ID=1 ArbSelect=Fixed TC/VC=00
			Status:	NegoPending- InProgress-
			Port Arbitration Table <?>
	Capabilities: [20c v1] Power Budgeting <?>
	Capabilities: [230 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Capabilities: [240 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1- L1_PM_Substates+
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-

		L1SubCtl2:
	Kernel driver in use: pcieport

running the same command on the USB controller: Renesas Technology corp…etc

sudo lspci -s 0005:03:00.0 -vvv
0005:03:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) (prog-if 30 [XHCI])
	Subsystem: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 38
	Region 0: Memory at 1f40000000 (64-bit, non-prefetchable) 
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [70] MSI: Enable- Count=1/8 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [90] MSI-X: Enable+ Count=8 Masked-
		Vector table: BAR=0 offset=00001000
		PBA: BAR=0 offset=00001080
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 unlimited
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR+, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [140 v1] Device Serial Number 13-00-00-00-92-43-14-08
	Capabilities: [150 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Kernel driver in use: xhci_hcd

I can post the terminal outputs for each slot designation, but not sure if that would be helpful. I did do some googling and it seems that the PEXUSB3S44V uses two drivers which may be why I am seeing the two different outputs when using lspci.

What stands out to me is line 17 in my second block of code where it says “pcilib: sysfs_read_vpd: read failed: Input/output error”.

Thanks for the help so far!

Edit: Update
I contacted StarTech and they were unsure of what to do. The recommended trying what NVIDIA suggested,

“This card may have issues with low power states. Please add ‘pcie_aspm=off’ to the kernel command line.”

How would I go about doing this?

Before you do more advanced debugging, the basics of what follows is that things are working correctly up to drivers being installed. What do you see from “lsusb -t” when your devices are connected? Perhaps it is just the software using the device which is failing.

Regarding the lspci output…

A PCIe bridge might be part of the USB card, and is operating correctly at PCIe gen. 2 speeds (and this is the maximum speed the card is capable of…everything is working correctly without limitations).

Similarly, the USB controllers are designed to talk to PCIe at PCIe gen. 2 speeds, and these too are operating without error at their full speed. Within the context of PCIe, everything is working correctly at full abilities.

The assigned driver for the USB side is the xhci_hcd driver. This is talking to the USB port through the PCIe as a kind of data tunnel. This driver would not have been assigned if PCIe had failed. The gist is that the hardware announced itself to the hot plug layer, and xhci_hcd identified the hardware as something it could work with, and then got assigned to the USB chips. This implies other software is capable of using the USB ports of the card.

Indeed, low power states are intended to reduce power consumption, but to still provide enough power that certain activities can “wake” the power state, but have problems. There are many devices which behave badly from low power states. It can be hard for companies to test all conditions of low power states and wake, and it is common for disabling low power states to fix issues (if the device never goes into low power state, then it can’t break from a badly handled wake event). The “pcie_aspm=off” could fix this if that is the case.

I am going to suggest that before you do any kind of flash operation it is good to have a backup clone for times when it goes bad. However, in Xavier (without U-Boot) you can expect to add the kernel command line parameter for “pcie_aspm=off” via an edit to the device tree. Within the device tree is a “chosen” node, and the “bootargs” entry is basically the argument list passed to the kernel. Earlier boot stages may edit this slightly, but compare the two commands:

# The actual booted kernel's arguments:
cat /proc/cmdline
# The chosen->bootargs node of the device tree:
cat /proc/device-tree/chosen/bootargs

Within the exception of some minor edits from earlier boot stages these match. If you were to append a space (it is space delimited) and “pcie_aspm=off” to the device tree “chosen->bootargs”, then this would disable aspm.

There may be alternate methods to do this, e.g., through sysctl.conf, but I am unfamiliar with how to actually do so. Any reference you see on the web for GRUB will be invalid since GRUB is only for PCs, and any reference to U-Boot will also be invalid since Xavier does not use U-Boot. Anyone else know how this kernel command line might be appended without flashing a device tree? If not, then you are going to need to flash the device tree with modification of " pcie_aspm=off".

Running lsusb -t with a USB2 mouse connected

agx1@agx1:~$ lsusb -t
/:  Bus 10.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 10000M
    |__ Port 4: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
/:  Bus 09.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 480M
    |__ Port 4: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 1: Dev 3, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
/:  Bus 08.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 5000M
/:  Bus 07.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
/:  Bus 06.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 5000M
/:  Bus 05.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M

When I run lsusb -t with the mouse connected to my USB3 Hub I am able to see another connection under

/:  Bus 09.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 480M
    |__ Port 4: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 1: Dev 3, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 4: Dev 5, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M

…so the mouse DOES NOT show up when connected to the PCIe USB3.0 card.

Thanks for the explanation, that makes sense.

Running cat /proc/cmdline

agx1@agx1:~$ cat /proc/cmdline 
root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4 video=tegrafb no_console_suspend=1 earlycon=tegra_comb_uart,mmio32,0x0c168000 gpt tegra_fbmem=0x800000@0xa069b000 lut_mem=0x2008@0xa0696000 usbcore.old_scheme_first=1 tegraid=19.1.2.0.0 maxcpus=8 boot.slot_suffix= boot.ratchetvalues=0.4.2 vpr_resize sdhci_tegra.en_boot_part_access=1    quiet usbcore.usbfs_memory_mb=1000

Running cat /proc/device-tree/chosen/bootargs

agx1@agx1:~$ cat /proc/device-tree/chosen/bootargs 
root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4 video=tegrafb no_console_suspend=1 earlycon=tegra_comb_uart,mmio32,0x0c168000 gpt tegra_fbmem=0x800000@0xa069b000 lut_mem=0x2008@0xa0696000 usbcore.old_scheme_first=1 tegraid=19.1.2.0.0 maxcpus=8 boot.slot_suffix= boot.ratchetvalues=0.4.2 vpr_resize sdhci_tegra.en_boot_part_access=1    quiet usbcore.usbfs_memory_mb=1000

As a side note, I am using FLIR BlackFly S cameras with the Xavier and they suggested making the change that you see at the end of the cat ouput “usbcore.usbfs_memory_mb=1000” I did this by using sudo gedit /boot/extlinux/extlinux.conf. Could this be a possible way to change the pcie_aspm=off?


Edit:
I rebooted this card without external power connected to it (only connected to the AGX through PCIe slot) and it magically worked (…I swear I’ve already tried this…). I was able to connect a thumbdrive/mouse/and USB3 camera to it. All devices worked for me. When using “dmesg --follow” and plugging back in the external power I see a message “USB over-current condition” and all USB ports on the card stop working. The card only recovers if I unplug external power and reboot the system.

Here is the output from “dmesg --follow” when I plug in external power to the PEXUSB3S44V

[   79.228274] usb usb3-port1: over-current condition
[   79.228422] usb usb7-port1: over-current condition
[   79.228560] usb usb2-port1: over-current condition
[   79.228716] usb usb3: usb_suspend_both: status 0
[   79.228762] usb usb7: usb_suspend_both: status 0
[   79.228818] usb usb2: usb_suspend_both: status 0
[   79.236289] usb usb8-port1: over-current condition
[   79.236492] usb usb5-port1: over-current condition
[   79.236518] usb usb6-port1: over-current condition
[   79.236529] usb usb4-port1: over-current condition
[   79.236920] usb usb5: usb_suspend_both: status 0
[   79.237003] usb usb6: usb_suspend_both: status 0
[   79.237097] usb usb8: usb_suspend_both: status 0
[   79.237117] usb usb4: usb_suspend_both: status 0
[   79.396133] usb usb1-port1: over-current condition
[   79.396311] usb usb1: usb_suspend_both: status 0

Is it correct that in the first lsusb the single HID device is not from the mouse in question? Indeed, I see a mouse only once there (the expansion card, USB3.1 gen. 2), but twice in the second lsusb (one of the native USB2 Jetson ports).

What I find of interest is that the expansion USB card does show up correctly, and indeed at least some USB devices work on it. This guarantees that at least some of the basics are working, and the chain of PCIe through USB end device is functioning to some degree.

If extlinux.conf APPEND works, then this might be a way to add pcie_aspm=off, but I can’t guarantee it will work as expected. It is easy to try and test (if it doesn’t work any risk to boot failing is near zero), but if there is a need for earlier boot stages to see this, then it might not work (but it might!).

This indeed is fatal to function:

[   79.228274] usb usb3-port1: over-current condition

If the over-current is an attempt to use more than the low power mode current, then the pcie_aspm=off will probably fix the issue (it would be a case of a badly behaving peripheral during a power savings mode, and not using that mode would fix the issue). If it is an actual excess current draw relative to what the power bus can supply, this this won’t be fixable without providing another power source (the PCIe itself could be drawing more power than the regulators can provide).

Another possibility (but one I doubt is the problem) is that USB3 tends to be backwards compatible and make earlier devices (USB1 through USB2) available by routing to a different legacy controller. If there is a lack of a fallback to a legacy controller, then the USB3 port would have no way to function with legacy devices. There is one legacy HID device showing in the first lsusb, and so this is probably unrelated (however, someone with similar issues might find that information useful for their case).

There is yet another test you can perform if you have a USB3.1 gen. 2 externally powered HUB, but this seems unlikely to matter in your case. Just to be thorough, the explanation is that an externally powered HUB avoids drawing power from the host PC (the Xavier). Externally powered HUBs also increase the rated power versus drawing from a parent root HUB. In the case of a mouse or keyboard the current is so tiny it wouldn’t matter anyway, but this would be important for external hard drives, and sometimes for cameras.

Does this USB3.1 gen. 2 PCIe expansion card have a connector available for supplemental power beyond what is delivered through the PCIe slot?

I have 2 quite old devices, a mouse and keyboard connected. That is probably what you are seeing.

Unfortunately I do not have a USB3.1 gen. 2 externally powered HUB. Editing extlinux.conf the way that I mentioned and appending “pcie_aspm=off” did not solve my problem. I can’t tell if it actually implemented the change, or if the change just did not solve my problem…

This PCIe card does have a 4-pin Molex connector and a SATA power connector on the back in order to supply power. The over-current condition problem only occurs when I supply power through either of these connectors. When I boot the system without external power on the PCIe card everything works fine.

If you run command “cat /proc/cmdline”, and if the “pcie_aspm=off” shows up in this with space delimiting, then likely the parameter was available to the kernel. I do not know if earlier boot stages are also required to see this, and they might since PCIe has some setup prior to the kernel ever running, but odds are reasonably good that cmdline reflects whether the argument is used.

The other question about whether having “pcie_aspm=off” is sufficient or not for your case I can’t answer, and it is a valid question.

Any over-current will shut down a port or a power rail, depending on where the over-current is. Because of other details appearing to work correctly, my gut feeling is that this is a power delivery issue. Any over-current in the logs is basically a guaranteed failure of some part of the USB. It seems quite odd that the external power delivery to the card causes an over-current, and tends to make me wonder if there is a hardware fault somewhere. I will suggest that any manufacturer details on the use of the external power to the card be examined as this should never cause over-current. Perhaps it is something as simple as that external supply producing a ground loop.

Running “cat /proc/cmdline”,

agx1@agx1:~$ cat /proc/cmdline 
root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4 video=tegrafb no_console_suspend=1 earlycon=tegra_comb_uart,mmio32,0x0c168000 gpt tegra_fbmem=0x800000@0xa069b000 lut_mem=0x2008@0xa0696000 usbcore.old_scheme_first=1 tegraid=19.1.2.0.0 maxcpus=8 boot.slot_suffix= boot.ratchetvalues=0.4.2 vpr_resize sdhci_tegra.en_boot_part_access=1    quiet pcie_aspm=off usbcore.usbfs_memory_mb=1000

So it would seem that my change was implemented, however, it did not solve the problem.
I will try the card on my Windows PC and see if I get the same error.

Just updating to let everyone know this issue was resolved. It ended up being a problem with the power supply that I was using. The card now works fine with my new power supply.

Thanks for the update.

As per the site stereolabs

Some other cards are

Hi All,

I just purchased a new PEXUSB3S44V (Rev:3) card that had PCIe problems on boot with my Jetson TX2. After previously using an earlier version (Rev:PCom1) with great success, I was expecting simple plug and play. However, this was not the case with the latest version.

Editing the /boot/extlinux/extlinux.conf file to turn off active state power management fixed things for me, and I was able to run pipelines with multiple cameras simultaneously again. Here’s a simple way to do it:

$ sudo gedit /boot/extlinux/extlinux.conf

Here is what my extlinux.conf file looks like after saving. Note: ‘pcie_aspm=off’ was the only parameter that I added, so everything else below should be default on Jetson TX2.

TIMEOUT 30
DEFAULT primary

MENU TITLE L4T boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      INITRD /boot/initrd
      APPEND ${cbootargs} quiet pcie_aspm=off

This response might be a bit excessive on this particular thread, as it addresses a specific version of the PEXUSB3S44V card on a Jetson TX2 system. I wanted this to provide clarity to the solution add ‘pcie_aspm=off’ to the kernel command line on the Jetson platform.

Kudos to remko.lems for posting the solution here https://forums.developer.nvidia.com/t/pcie-driver-fails-after-moving-to-latest-l4t/81439/8.

Thanks!