USB stick failing at startup

Autonomous Machines Jetson & Embedded Systems Jetson Nano
,
1

Hi,
I have some trouble with usb stick on jetson nano and I’d like to get some help.

Configuration:
Jetson nano production module
Jetpack 4.6 (L4T 32.6.1)

More often than not, we get the following problem at startup:
The usb stick cannot be mounted, and our data cannot be saved.
This is even more problematic as the jetson are not on site and changing usb stick can be complicated.

This is the messages we get
$dmesg | grep usb

[    0.000000] Kernel command line: tegraid=21.1.2.0.0 ddr_die=4096M@2048M section=512M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 console=ttyS0,115200n8 debug_uartport=lsport,4 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=0x1000@0xff780000 core_edp_mv=1075 core_edp_ma=4000 gpt  earlycon=uart8250,mmio32,0x70006000  root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 sdhci_tegra.en_boot_part_access=1 quiet root=/dev/mmcblk0p1 ro rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 sdhci_tegra.en_boot_part_access=1 usbcore.autosuspend=-1
[    0.435182] iommu: Adding device 70090000.xusb to group 2
[    0.507023] vdd-usb-vbus: 5000 mV 
[    0.507261] vdd-usb-vbus: supplied by vdd-5v0-sys
[    0.507620] vdd-usb-vbus2: 5000 mV 
[    0.554398] usbcore: registered new interface driver usbfs
[    0.554450] usbcore: registered new interface driver hub
[    0.554491] usbcore: registered new device driver usb
[    0.621163] vdd-usb-vbus2: supplied by vdd-3v3-sys
[    0.623081] vdd-usb-hub-en: 5000 mV 
[    0.623407] vdd-usb-hub-en: supplied by vdd-1v8-sys
[    1.033078] tegra-xusb-padctl 7009f000.xusb_padctl: TEGRA_FUSE_SKU_CALIB_0 = 0x71c7c0e
[    1.033086] tegra-xusb-padctl 7009f000.xusb_padctl: TEGRA_FUSE_USB_CALIB_EXT_0 = 0x4
[    1.034728] tegra-xusb-padctl 7009f000.xusb_padctl: dev = phy-usb2.0, lane = usb2-0, function = xusb
[    1.034837] tegra-xusb-padctl 7009f000.xusb_padctl: dev = phy-usb2.1, lane = usb2-1, function = xusb
[    1.034929] tegra-xusb-padctl 7009f000.xusb_padctl: dev = phy-usb2.2, lane = usb2-2, function = xusb
[    1.035120] tegra-xusb-padctl 7009f000.xusb_padctl: dev = phy-pcie.3, lane = pcie-0, function = pcie-x1
[    1.035206] tegra-xusb-padctl 7009f000.xusb_padctl: dev = phy-pcie.4, lane = pcie-1, function = pcie-x4
[    1.035297] tegra-xusb-padctl 7009f000.xusb_padctl: dev = phy-pcie.5, lane = pcie-2, function = pcie-x4
[    1.035383] tegra-xusb-padctl 7009f000.xusb_padctl: dev = phy-pcie.6, lane = pcie-3, function = pcie-x4
[    1.035473] tegra-xusb-padctl 7009f000.xusb_padctl: dev = phy-pcie.7, lane = pcie-4, function = pcie-x4
[    1.035560] tegra-xusb-padctl 7009f000.xusb_padctl: dev = phy-pcie.8, lane = pcie-5, function = xusb
[    1.035651] tegra-xusb-padctl 7009f000.xusb_padctl: dev = phy-pcie.9, lane = pcie-6, function = xusb
[    1.654041] usbcore: registered new interface driver r8152
[    1.654083] usbcore: registered new interface driver asix
[    1.654121] usbcore: registered new interface driver ax88179_178a
[    1.654156] usbcore: registered new interface driver cdc_ether
[    1.654188] usbcore: registered new interface driver net1080
[    1.654220] usbcore: registered new interface driver cdc_subset
[    1.654252] usbcore: registered new interface driver zaurus
[    1.654298] usbcore: registered new interface driver cdc_ncm
[    1.660812] tegra-xusb 70090000.xusb: USB2 port 0 has OTG_CAP
[    1.661373] tegra-xusb-padctl 7009f000.xusb_padctl: enabled OTG on UTMI pad 0
[    1.662425] tegra-xusb 70090000.xusb: extcon 0: ffffffc0fa5d2c00 id
[    1.662498] tegra-xusb 70090000.xusb: Direct firmware load for tegra21x_xusb_firmware failed with error -2
[    1.662503] tegra-xusb 70090000.xusb: Falling back to user helper
[    1.663644] usbcore: registered new interface driver uas
[    1.663690] usbcore: registered new interface driver usb-storage
[    1.663766] usbcore: registered new interface driver usbserial
[    1.667881] usbcore: registered new interface driver xpad
[    1.904390] usbcore: registered new interface driver usbhid
[    1.904393] usbhid: USB HID core driver
[    1.909350] usbcore: registered new interface driver snd-usb-audio
[    2.352127] tegra-xusb 70090000.xusb: cannot find firmware....retry after 1 second
[    3.383816] tegra-xusb 70090000.xusb: Firmware timestamp: 2020-07-31 09:33:12 UTC, Version: 50.26 release
[    3.383849] tegra-xusb 70090000.xusb: xHCI Host Controller
[    3.383867] tegra-xusb 70090000.xusb: new USB bus registered, assigned bus number 1
[    3.384554] tegra-xusb 70090000.xusb: hcc params 0x0184f525 hci version 0x100 quirks 0x00050010
[    3.384588] tegra-xusb 70090000.xusb: irq 61, io mem 0x70090000
[    3.384710] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    3.384715] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.384719] usb usb1: Product: xHCI Host Controller
[    3.384722] usb usb1: Manufacturer: Linux 4.9.253-tegra xhci-hcd
[    3.384725] usb usb1: SerialNumber: 70090000.xusb
[    3.385465] tegra-xusb 70090000.xusb: xHCI Host Controller
[    3.385474] tegra-xusb 70090000.xusb: new USB bus registered, assigned bus number 2
[    3.385536] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[    3.385604] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003
[    3.385608] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.385611] usb usb2: Product: xHCI Host Controller
[    3.385614] usb usb2: Manufacturer: Linux 4.9.253-tegra xhci-hcd
[    3.385617] usb usb2: SerialNumber: 70090000.xusb
[    3.386448] tegra-xusb 70090000.xusb: Upgrade port 0 to USB3.0
[    3.386454] tegra-xusb 70090000.xusb: Upgrade port 1 to USB3.0
[    3.486781] tegra-xusb-padctl 7009f000.xusb_padctl: power on UTMI pads 1
[    3.486880] tegra-xusb-padctl 7009f000.xusb_padctl: power on UTMI pads 2
[    3.710727] usb 1-2: new high-speed USB device number 2 using tegra-xusb
[    3.733591] usb 1-2: New USB device found, idVendor=2109, idProduct=2817
[    3.733597] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    3.733601] usb 1-2: Product: USB2.0 Hub             
[    3.733604] usb 1-2: Manufacturer: VIA Labs, Inc.         
[    3.857815] usb 2-1: new SuperSpeed USB device number 2 using tegra-xusb
[    3.877025] usb 2-1: New USB device found, idVendor=2109, idProduct=0817
[    3.877032] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    3.877035] usb 2-1: Product: USB3.0 Hub             
[    3.877039] usb 2-1: Manufacturer: VIA Labs, Inc.         
[    3.994713] usb 1-3: new full-speed USB device number 3 using tegra-xusb
[    4.016687] usb 1-3: New USB device found, idVendor=8087, idProduct=0029
[    4.016694] usb 1-3: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    4.450709] usb 1-2.2: new full-speed USB device number 4 using tegra-xusb
[    4.475653] usb 1-2.2: New USB device found, idVendor=1546, idProduct=01a9
[    4.475660] usb 1-2.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    4.475664] usb 1-2.2: Product: u-blox GNSS receiver
[    4.475668] usb 1-2.2: Manufacturer: u-blox AG - www.u-blox.com
[    4.566989] usb 2-1.1: Device not responding to setup address.
[    4.790972] usb 2-1.1: Device not responding to setup address.
[    5.002726] usb 2-1.1: device not accepting address 3, error -71
[    5.346983] usb 2-1.1: Device not responding to setup address.
[    5.570995] usb 2-1.1: Device not responding to setup address.
[    5.663292] usb0: HOST MAC ea:4f:23:d8:a6:d2
[    5.663322] usb0: MAC ea:4f:23:d8:a6:d3
[    5.731973] l4tbr0: port 2(usb0) entered blocking state
[    5.731977] l4tbr0: port 2(usb0) entered disabled state
[    5.732984] device usb0 entered promiscuous mode
[    5.778734] usb 2-1.1: device not accepting address 4, error -71
[    5.784985] usb 2-1-port1: attempt power cycle
[    5.791200] IPv6: ADDRCONF(NETDEV_UP): usb0: link is not ready
[    6.991004] usb 2-1.1: Device not responding to setup address.
[    7.214979] usb 2-1.1: Device not responding to setup address.
[    7.426772] usb 2-1.1: device not accepting address 5, error -71
[    7.774970] usb 2-1.1: Device not responding to setup address.
[    7.998993] usb 2-1.1: Device not responding to setup address.
[    8.210727] usb 2-1.1: device not accepting address 6, error -71
[    8.217019] usb 2-1-port1: unable to enumerate USB device
[    8.314992] usb 2-1.4: Device not responding to setup address.
[    8.534962] usb 2-1.4: Device not responding to setup address.
[    8.746726] usb 2-1.4: device not accepting address 7, error -71
[    9.095087] usb 2-1.4: Device not responding to setup address.
[    9.318977] usb 2-1.4: Device not responding to setup address.
[    9.530722] usb 2-1.4: device not accepting address 8, error -71
[    9.536978] usb 2-1-port4: attempt power cycle
[   10.734974] usb 2-1.4: Device not responding to setup address.
[   10.958970] usb 2-1.4: Device not responding to setup address.
[   11.170716] usb 2-1.4: device not accepting address 9, error -71
[   11.519073] usb 2-1.4: Device not responding to setup address.
[   11.742933] usb 2-1.4: Device not responding to setup address.
[   11.954722] usb 2-1.4: device not accepting address 10, error -71
[   11.961147] usb 2-1-port4: unable to enumerate USB device
[   12.025255] usbcore: registered new interface driver btusb
[   12.064570] usbcore: registered new interface driver cdc_acm
[   33.906910] vdd-usb-vbus: disabling
[   33.906913] vdd-usb-vbus2: disabling
[   33.907007] vdd-usb-hub-en: disabling

The problem seems to appear randomly, on multiple jetson with different usb stick. (SanDisk and samsung)
On the same Jetson, it can be working a day but not the following.
I also couldn’t find the meaning of the “-71” error number.

I tried the following things found here and there on internet, and it didn’t work so far, including
*set use_both_schemes=y
*set usbcore.autosuspend=-1

I hope you will be able to help me,
I stay available for any further information.

Thank you in advance for your help.
Leo.

3

Hi,

are you using a custom board or our official DevKit?

4

Hi DaveYYY, we encountered the issue with both official and unofficial DevKits.

5

Hi,

if it happens on custom boards, then some users found it related to the battery power supply on the board:

, but if it also happens on our DevKit, we may need more time to investigate.

8

Hi,

can you try with different USB sticks/devices and see if it can always be re-produced?
Also upgrade to our latest BSP. (32.7.4)

9

Thanks for the reply, I will investigate these 2 options and get back to you as soon as possible.

10

Hi Dave,
latest BSP cannot be installed on distant machine, we would like to fix the problem remotely, neither we use RTC.
We are still investigating and we suppose it could come from the use of usb3.2.
Do you know how i can force the device to use USB 2.0 driver instead of the USB 3?
If possible we would like to avoid editing the device tree.
Thank you.

11

Hi,

you can try with a USB hub that only supports USB 2.0; otherwise, I don’t think it’s possible without changes in device trees.

12

A useful example is the use of a USB1.1 protocol analyzer: Some devices normally run at USB2 or faster, but if the analyzer has only USB1.1 or 1.0, then it is a problem. The solution is in fact to plug the device into a USB1.1 hub, which forces USB1.1 mode even though the base device is USB2. USB1.1 hubs are surprisingly expensive though. I suggest what @DaveYYY suggests because it is inexpensive and very reliable (some devices do not support reverting to slower speeds, but most do). The hub itself is better at causing an automatic driver switch.

I suppose there are USB3 wires you could cut.

13

Thanks for the replies Dave and linuxdev,
Could’nt force USB2.0 from device tree, it is either everything is working (including usb 3.0) or nothing is working. Even though I will keep investigate that way.
I will try the hub method I just wanted to find a remote fix, but if it is the only option I’ll do it.
Also I am not ready to cut the pcb trace yet, I still got some hope to fix the issue in a more conventional way.

Thank for your support.

14

Do you know how I should edit the device tree on my jetson nano to force USB2 ? as I could not make it work. I have tried to disable many different port unsuccessfully so far.

15

Just to clarify

If this is “-71” error from usb driver and your board is custom board, then it is most likely signal quality issue.

Also, device tree change is the basic one when making custom board but it may not be directly related to -71 issue. For example, some users’ board have correct DTB but -71 issue still comes due to hardware problem.

16

It is not a custom board per se, it is a waveshare Development / Expansion Kit.
I know that it is probably a hardware related bug, but I would like to test if the same thing happend using usb2.0 devices. As it could be an quick and temporary fix.

17

Hi,

Actually everything which is not NV devkit is custom board to us…

Also, are you the board vendor or not? If not, then do not change the device tree by yourself.

18

I hear that, I am not the vendor but we are doing development, so we’re kind of forced to bring modification at some point if we want our system to be working properly.

19

Hi,

Basically what you can do now is try more kinds of usb drive and usb cable.

If you want to force speed to usb2, then just connect a cable or hub which only supports USB2 speed.

Or if you have equipment, dump the usb bus trace when you connect the device which can trigger -71 error.

Also, if possible, try to reproduce your issue on devkit. If you cannot reproduce on devkit, please also contact board vendor and let them handle this issue with us.

20

Thank Wayne, I’ll get back to you after more testing.

21

If you really want to test USB2, then put it through a USB2 HUB which is not capable of USB3. It will switch to the legacy USB2 controller without modifying the device tree, this is how the design works (the USB3 controller has no legacy mode ability, so it reroutes to the USB2 controller). You’d know then if USB2 works, unless it is a signal quality issue (the HUB can act as a signal adapter/buffer, and USB2 runs slower, so the signal quality does not need to be as good; but the controller and protocols and speeds are forced to the USB2 side).

22

Hi, some update on the situation,

To put it simply:
The issue seems to appear randomly and we did not succeed to reproduce it at the office. Without powering down the jetson, a simple unplug replug sometimes do the trick. A cold reboot can also fix the issue sometimes, but a hot reboot never fixes the issue.
We encounter this issue on both official nvidia devboard and on a custom board from a nvidia partner. On the same jetson with 2 pluged usb flash drive, we can have addressing issues on both sticks, only one or none. The problem does not seam to appear on newly installed products but after few months of usage. We have other devices connected using USB2, and never had issues with them. It only concerns the USB3 Samsung USB Sticks.

The only way we could reproduce the issue at the office is by cutting one lane of the USB3 (transmit-, transmit+, receive+ or receive-). We didn’t try with Data-, Data+.

Note that there is a USB 3.0 hub on the nvidia dev board(RTS5411S) and an other one on the partner board(VL817-Q7 3.0 hub ).

It happens on both boards with different hub so it is very unlikely that it comes from the hub, it’s random so it shouldn’t be coming from the driver either. Moreover I don’t think it comes from the jetson as there is a hub between the jetson and the ports, and we can have one device OK and the other one NOK.

So it is very likely to come from the usb stick or the connection between the USB stick and the dev board.
Our idea is to force the hub to use USB2 and not USB3. USB2 is known to be more reliable than USB3 because there is less cables and frequencies are lower. USB2 speed is way enough for us. We changed the device tree by disabling lanes and ports without success, neither USB2 or USB3 was working. I think the jetson have to setup the hub to use only USB2. I tried to find the driver of the RTS5411S on the kernel sources but couldn’t find it.

We have an USB2 external hub and a USB2 male/female cable, the usb stick are working with them but also with direct connection on USB3.1.

So it is not possible to put an external hub for testing if usb2 fix the issue as it is very random. We already can’t reproduce it without hub.

So the question is how to totally disable USB3 on jetson nano dev board to force devices to use USB2?

Thanks, Leo.

23

You can remove the usb3 setting from the device tree.

A document here that explains how that works.
https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-3273/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/adaptation_and_bringup_nano.html#wwpID0E06L0HA

->usb structure