Ever since upgrading to 24.1 (from 23.2), I have had periodic issues with my USB ports. I am currently still using the 32 bit user space as I haven’t been able to re-compile all libraries yet.
Often when the TX1 boots, my USB ports work correctly and I am able to use keyboard, mouse (via usb hub) on the USB2 port and a camera on the USB3 port. Moving devices around to different ports work as expected.
Other times, nothing will get recognized on either port. In this scenario, I boot up with keyboard and mouse attached via a hub on the USB2 port and neither will respond. If I connect anything (hub, keyboard or mouse) into the USB2 port, I get the following messages on the serial console and the device does not work:
[ 435.820259] regulator_get() failed for (tegra-ehci.0,usb_vbus), -19
[ 435.828299] usb_phy: failed regulator_get vdd_vbus_usb:-19, inst:0
[ 436.863280] Could not add tegra-ehci.0 to power domain using device tree
If I plug anything into the USB3 port, the device does not work but I do not get any messages on the serial console.
If I type “lsusb -t” on the serial console, the program hangs.
Warm rebooting rarely fixes this condition and I usually have to remove power from the board for ~10 seconds before it will recover and USB will work correctly.
One of the first things I advise someone to do with an unreliable USB device is to consider using an externally powered HUB. I would say this even if I hadn’t see the “regulator_get” error. Consider that anything not self-powered, including a HUB itself, adds up to some cumulative amount of power which would otherwise need to be powered by the host port itself. You could chain lots of HUBs together and devices, and end up with many dozens of devices which individually do not seem like much of a power drain, but which must all be powered simultaneously by the single host port the HUBs and devices eventually trickle down to. Power delivery circuits in USB root HUBs also drastically differ in how they detect and deal with power delivery (a cost issue), so non-powered devices may or may not run correctly near the margins of power requirements from one host to the next…unless power is delivered externally.
So…is your HUB powered or unpowered? I suggest grabbing a powered HUB and remove the Jetson from the power delivery part of the hardware. It is possible the power delivery is dealt with differently in R23.2 versus R24.1 (I don’t know, but the USB2 versus USB3 software itself allows higher current draw for USB3 versus USB2…only the hardware is constant to some extent…possibly software PINMUX for port owner is related to different power delivery hardware).
Additionally, keyboards and mice do not draw much, but many of the camera or sensor devices commonly used on a Jetson draw far more power. Example: Does a USB3 camera running in USB2 mode cause the camera to limit its current draw to USB2 values? It would depend on how much of the current draw is due to USB mode versus normal operation of hardware such as sensor…
Thank you for taking the time to supply an detailed answer.
My HUB is powered. The serial console messages occur with both the HUB or just a single USB Keyboard or mouse is on the USB2 port so I don’t believe I am drawing too much power. Nothing (keyboard, mouse, camera or powered hub) work on the USB3 port when the error condition.
I just went and looked closely at boot messages from one where things worked normally (R24.1 64-bit), and it turns out the log messages you saw also occur when things work. What stands out was “using device tree”, so I’m thinking this is not necessarily an issue if setup works from non-firmware files instead. So one very important detail to ask…do those messages occur each time you change/plug/unplug a USB device during failure? Or does this occur just once during boot? If this occurs during each plug/unplug/change?
The messages will occur anytime in plug into the USB2 port. Nothing is displayed on unplug from that port. The message will be re-displayed when a new device (anything) is plugged into that port again.
By “USB2 port”, do you mean the micro-USB port (probably what you mean)? Or the full-sized port when configured as USB2 (less likely to be what you mean)?
Also, how does the issue change if the entire HUB is unplugged and plugged, versus just one item on the HUB?
Yes, by USB2 I mean the micro-USB port, not the full sized port. The micro-USB port behaves exactly the same if you plug in a single device, just the HUB or the HUB with devices attached. The same messages are shown for all cases and you only get one set of messages, not one per device.
The full sized port does nothing regardless if you use a USB2 or USB3 device.
For reference, my testing was on R24.1 64-bit sample rootfs. You said above you were using R24.1 32-bit, so maybe testing under 64-bit would be good, but I don’t know if you are bound to a particular 32/64-bit L4T version for some mandatory reason or not.
Admittedly I do not have a large array of USB devices to test with, and it is possible a bug in some particular part of USB is hitting which my devices do not reach, but I’ve been unable reproduce this (remember though that I’ve only tested 64-bit). Intermittent issues which are borderline between showing up or not are the hardest thing to debug.
Is the 32-bit required for your purposes? It may not be worth debugging if future releases are all 64-bit and if the problem does not exist on 64-bit.
In case it’s useful to either of you at all, I’ve been able to crash the USB driver resulting in no USB devices being recognized until the next reboot simply by copying files from a USB drive to the TX1 eMMC. I have also crashed the OS in the same manner. I’ve observed this behavior in both R24.1 64-bit and R24.1 32-bit, (no USB hub involved, just the development board’s main USB port to onboard memory). I am curious if you have seen this behavior at all.
On R24.1 64-bit I was able to copy anywhere from 100MB up to 36GB via dd to either /dev/null or to a file on eMMC without error (after about 5GB I went to /dev/null since I didn’t want to fill up eMMC). The drive is an unpowered external USB3 case with a laptop hard drive in it (although lsusb verifies the drive was in USB3 mode actual data throughput would have been slow…it’s something like 5400 RPM). My use of dd would have bypassed any file system bugs, but tested USB3 and eMMC.
Can you run fsck on your drive to see if the partition used has any defects? I doubt it does, but that would be required to be thorough. Assuming your USB hard drive is /dev/sda, you could try this test to bypass file system type issues:
cat /dev/sda > /dev/null
More information on the USB drive itself may also help, e.g., SSD versus old style hard drive changes things.
I was able to fully reproduce my issue on the 64-bit OS. On my third warm boot after flash, no USB devices can be recognized and the same errors are shown on the serial console when a device is connected to the micro-USB port. As before, no errors are shown when a device is connected to the USB3 Type A port.
How are you performing restart? Is the power being removed between restarts? Is it a shutdown by software followed by power button? Is it a software command shutdown with reboot where no power or reset button is used? Any details on how shutdown is performed and exact timing of add/remove USB connections or power connector may be useful.
I have seen issues with both software initiated restarts (i.e. reboot command with no power button pressed) and pressing the “Reset” button (also not requiring a power button press). In both cases, power is not being removed between restarts (hence warm boot).
I have not tested extensively by pressing power button or by fully disconnecting the 19V power from board between restarts.
No USB connections or disconnections are made between when the board is working correctly and when the board is reset and fully reboots to the Ubuntu login screen. Once at the login screen, I can test if the keyboard and mouse are working, if not, then I will disconnect and re-connect to see the error messages above.
Do remember that hitting the reset button on a running system causes the need for the file system to either recover via journal or repair…in the case of journal recovery a small amount of changes to the file system from the last run will be lost…in the case of fsck repair a much larger chunk of the file system can be lost. I doubt this is an issue for USB detection, and certainly the case of software initiated restarts shows the issue is not dependent upon file system issues…but for the subset of issues reported after a running system has the reset button hit you have to throw out that data.
Just because I didn’t see the issue does not mean it doesn’t exist, and in case it does, here is an observation possibly related to this…
Normally there is a USB2 driver loaded first from u-boot. This allows boot loading from a USB hard drive. As soon as the kernel loads, this original USB2 driver goes away, and then the kernel loads its own driver. This is why booting to USB drive will fail if the full-size USB connector is configured as USB3…hand off from u-boot USB2 to kernel USB2 does not throw out the existing USB enumeration, or at least it does not prevent USB2 from operating continuously. Going from u-boot USB2 to kernel USB3 requires losing everything in USB before it re-enumerates, which breaks USB3 as a USB hard disk for boot (it’s a different topic, but it would be worthwhile to enable USB3 within u-boot).
This is just speculation, but I’m wondering if perhaps the u-boot version of the USB2 driver servicing both micro and full-size USB ports might be affected by the kernel essentially forcing part or all of USB2 to unload and reload (micro port stays loaded as USB2, full-size port gets wiped out and reloaded as USB3). The two ports and drivers should be independent under the Linux kernel, but at the moment of going from u-boot operation to kernel operation I could see how having something attached to USB3 or not might change the timing of enumeration on both ports (firmware is changing USB port layout, there may be code which does not execute until something is plugged into the full-size port).
One possible test would be to configure the full-size port to run as USB2 and see if the issue goes away.
I am aware of the dangers of using the reset button on a flash file system. I have simply used it for testing this issue and have no intentions of exposing the button on my actual application. And as you have stated, I do not believe the issue is related to file system corruption (since it goes away after another reset).
However, I am planning on using the USB3 port in my application so reliable booting with USB3 support is important.
If you can tell me how to set the USB3 port into USB2 only mode, I will test and see if the issue is solved.
This used to be possible via the extlinux.conf “usb_port_owner_info”, but does not seem to work now (at least under R24.1 64-bit). In the kernel source “arch/arm64/mach-tegra/board-t210ref.c” it seems as if setting “usb_port_owner_info” to “1” (equals define UTMI1_PORT_OWNER_XUSB 0x1) would do the trick, but apparently this is not passed along from the extlinux.conf.
For testing purposes, does anyone here know if USB3 can be forced to USB2 via extlinux.conf in R24.1?
For the original issue, you mentioned that it happens in R24.1 and works well in R23.2, right?
Can you describe the detailed steps to reproduce that issue?
The steps to reproduce are pretty much as detailed in the first post:
Connect USB keyboard and mouse to powered USB HUB
Connect HUB to micro-USB port on Jetson TX1
Power on the TX1. Note that USB Keyboard and mouse work as expected
Reboot the TX1 either by pressing the reset button or using “sudo reboot”. After a few attempts (usually 2 - 3 reboots), notice the USB Keyboard and mouse no longer work.
Once USB Keyboard and mouse stop working, unplug the USB hub from micro USB port.
Connect HUB to USB type A port (USB3 port). Notice the keyboard and mouse still don’t work. No messages on serial console
connect keyboard only to Micro-USB port. Notice keyboard does not work and the serial console prints out messages:
[ 435.820259] regulator_get() failed for (tegra-ehci.0,usb_vbus), -19
[ 435.828299] usb_phy: failed regulator_get vdd_vbus_usb:-19, inst:0
[ 436.863280] Could not add tegra-ehci.0 to power domain using device tree
Reboot the system. Sometimes TX1 will recover, sometimes it will still not have working USB. It seems fairly random if USB will work or not on each reboot.
Hello, mleonhardt:
I tried your steps 1-4 for several times, and every time USB mouse works well. (my USB hub is powerless. Also I tried USB mouse directly connecting to mini USB port for several times rebooting, and it also works well.)
Would you please try with USB mouse directly without hub and check whether the issue happens? When the error happens, would you please paste the full kernel log, or any error message you noticed?
for the regulator_get failure, that’s more like a warning. VBUS is on when that port works in host mode.