USB 3 Vision devices sometimes not detected

Autonomous Machines Jetson & Embedded Systems Jetson Xavier NX
, ,
1

Hello,

we have a custom board which is powered by a Xavier NX using Jetson Linux 35.5.0. The NX is connected to a USB 3 Vision device.

Sometimes this device won’t show up in the system. The camera is not seen by lsusb even no entry in dmesg is available.

When the camera is not found following entry is in the dmesg log:
[ 9.695838] tegra-xusb 3610000.xhci: entering ELPG done

This entry is not there when the camera was found.

Looking in the kernel sources reveals this entry have to do with power management settings like suspend.
But this entry is created during boot so no suspend should be triggered.

Even disconnecting the camera from the USB port will not solve the problem.
The port itself is still functional. When connecting a keyboard or other devices the will recognized.
Connecting other superspeed device to another USB port will also not recognized.

How can i get the tegra-xhci back into running state? Or can i disable the ELPG?

Thank you

2

Probably the hardware design is wrong since the beginning but as USB has some error tolerance capability, it is still working from time to time.

You better checking if your vbus is always-on. If it is, change to use a gpio + load switch to control it. After doing this change, device tree software change is required too.

3

Thanks for your reply.

It’s unlikely that the design is the reason because we run also the Nano with the same PCB. In this case we can let run a test that boots frequently for days and the camera is found every time.
We also used Jetson Linux 32.4.4 before and i think this problem does not occur with this version.

Is it possible to disable the ELPG to see if this will fix the Problem? Preferably without compiling the Kernel.

4

Hi,

Let me clarify some points first…

  1. Did you configure any software on your carrier board for that USB port before? I mean the device tree.

  2. The point to tell a Jetson nano is working fine on that board does not indicate the software running on Jetson Xavier NX is matching for that board. Thus, it is back to point (1) again.

  3. You are not the first one reporting such “my USB3 device are not able to get detected”. Most of such case is related to the vbus timing… There are no other solution. ELPG is not the issue either…

5

Hello again,

  1. Yes i configured the device tree for the NX. If i remember correctly most parts are based on the Dev-Board Design. So i only have to adopt only small changes.

  2. You’re right. But we also used the NX previously with Jetson Linux 32.4.4 without this problem.

  3. For the VBus timing i asked my colleague who designed the pcb. He described that the vbus for the camera is switched statically by a internal hub. This means vbus is switched only during the first boot and should not be affected by later reboots.

In the test the NX is rebooted with “sudo reboot”.

We also can control the USB HUB with i2c. And if i disable it all other devices like keyboars will be disconnected. When i enable it again the keyboard will show up again. But the xhci are not affected by disabling the hub.

Currently i’m a little in hurry so i will investigate it further on monday.

6

As you said your board is similar to NV devkit, then are you able to reproduce this issue on devkit?

7

It took some time but now i can execute nearly the same test from our device on the devkit.
And indeed i run into an USB error but this error looks different and maybe have nothing to do with the ELPG problem.
This error happens about 1 percent of the time and leads also to the behaviour that superspeed devices are not usable anymore.
Even after unpluging the device and connecting to another port it is not accessible.

I’ve done the test with two configurations.
The first configuration uses the OS from our Device.
The second configuration uses the original sample OS by Nvidia when flashing via SDK manager.

In both cases i get following dmesg output in the error case:
[ 10.258853] usb usb2-port3: Cannot enable. Maybe the USB cable is bad?
Using our OS this error happens more frequently maybe due to different timing.

All in all i can not say that the “ELPG” error did not happen on the devkit because the “bad cable” error confuses my test script.

8

How to reproduce this on devkit?

9

You only need to reboot the devkit and sometimes this error occurs.
I’m using a systemd service that execute a python script to watch for the superspeed device. Than waiting a minute and reboot again. The rebooting will only stop if i stop the service manually or when the ss device is not available.

The systemd service looks like this:

[Unit]
Description=Reboot Test
[Service]
Type=simple
Environment="SYSTEMD_REBOOT_TEST=yes"
ExecStart=/home/user/reboot_test.py
[Install]
WantedBy=default.target

And the python script like following:

#!/usr/bin/env python3
import sys
import time
import subprocess
if __name__ == "__main__":
    ret = 1
    for _ in range(20):
        time.sleep(1)
        ret = subprocess.run("lsusb | grep \"Bus 002 .* XXXX:\"", shell=True, check=False).returncode
        if 0 == ret:
            break
    if not 0 == ret:
        sys.exit(ret)
    time.sleep(60)
    subprocess.run(['sudo', '/sbin/reboot'], check=True)
    sys.exit(0)

The XXXX have to be replaced to the vendor id of the ss device. It is also important that the user need no password to use sudo.
‘loginctl enable-linger’ will help also that nobody needs to login manually.

This is a stripped down version of the test. It should also be possible to do this with bash only. But the original python script have much more functionality.

10

Hi,

I mean how is the usb connection there for this issue?

Also, please be aware that resolving this issue on Devkit does not indicate this issue would be resolved on your board. As you also notice the error symptom on devkit is different from your custom board.
This may indicate these two issues are different.

11

Hello,

the U3V camera is directly connected by a cable like these to one of the devkit usb ports. Like already said, when the error occurs i can change the port but the camera will not recognized anymore until i reboot the devkit.

I tried this with 2 models from different companies to reduce the risk this error is triggered by the camera device.

I agree with you that this issues may be different. As next steps i will compare the compiled device tree’s of the devkit and our pcb. Maybe i missed something to adopt during the upgrade from 32.4.4 to 35.5.0.

12

So i have some news and i’m convinced the issue comes from the driver/kernel/devkit.
The missing superspeed devices are not limited to the U3V camera, also the hub is sometimes missing on the devkit.
So i removed any usb devices from the devkit usb ports and start the test again.
Only connected stuff to the devkit is the micro usb cable and a hdmi monitor.

Normaly i see following entry in dmesg where the hub is registering as usb3 device:

[ 6.763850] usb 2-3: new SuperSpeedPlus Gen 2 USB device number 2 using tegra-xusb
[ 6.764343] tegra_nvdisp_handle_pd_disable: Powergated Head1 pd
[ 6.765385] tegra_nvdisp_handle_pd_disable: Powergated Head0 pd
[ 6.776692] tegradc 15200000.display: unblank
[ 6.777958] tegra_nvdisp_handle_pd_enable: Unpowergated Head0 pd
[ 6.778146] tegra_nvdisp_handle_pd_enable: Unpowergated Head1 pd
[ 6.784385] tegradc 15200000.display: hdmi: tmds rate:148500K prod-setting:prod_c_hdmi_111m_223m
[ 6.800584] usb 2-3: New USB device found, idVendor=0bda, idProduct=0489, bcdDevice= 1.40
[ 6.800591] usb 2-3: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 6.800618] usb 2-3: Product: 4-Port USB 3.0 Hub
[ 6.800622] usb 2-3: Manufacturer: Generic

And lsusb returns following:

Bus 002 Device 002: ID 0bda:0489 Realtek Semiconductor Corp. 4-Port USB 3.0 Hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 13d3:3549 IMC Networks Bluetooth Radio
Bus 001 Device 002: ID 0bda:5489 Realtek Semiconductor Corp. 4-Port USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

In the case where the error happens no usb2-X are shown in dmesg log.
And lsusb returns as only usb3 device the linux hub:

Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 13d3:3549 IMC Networks Bluetooth Radio
Bus 001 Device 002: ID 0bda:5489 Realtek Semiconductor Corp. 4-Port USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

So now it should be easy for you to reproduce this issue.
Do a reboot loop and wait if the usb3 hub of the devkit is missing.
In my tests i need at maximum 4 hours to get the error state. Mostly it occurs after two hours.
You can use the systemd service and the python skript from above. Use as vendor ID “0bda”.
With the script the reboot stops automatically when the error occurs.

Now the important question. Can i solve this without rebooting the device?
Like reprobing USB or anything else. I tried a lot but without success.

Thanks in advance.

1 Like
14

Maybe try this to power on/off the usb hub.

We will check that devkit issue.

15

Thanks for your reply,

i tried your suggestion but without success.
Firstly i used the sources without looking into it. Calling the compiled executable resulted in following error message:

[error]:open device 0bda:5411 fail.

Maybe the hub in the devkit has changed so i adopt the sources to use the correct PID.
Now the hub resets correctly which i can see in the dmesg log:

[49481.774277] usb 1-2: USB disconnect, device number 2
[49482.345393] usb 1-2: new high-speed USB device number 4 using tegra-xusb
[49482.512148] usb 1-2: New USB device found, idVendor=0bda, idProduct=5489, bcdDevice= 1.40
[49482.512163] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[49482.512172] usb 1-2: Product: 4-Port USB 2.0 Hub
[49482.512179] usb 1-2: Manufacturer: Generic
[49482.516678] hub 1-2:1.0: USB hub found
[49482.517884] hub 1-2:1.0: 4 ports detected

Linux reports that the hub disconnects correctly. But sadly only the USB2 hub connects to linux again.
lsusb shows this also.

Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 13d3:3549 IMC Networks Bluetooth Radio
Bus 001 Device 004: ID 0bda:5489 Realtek Semiconductor Corp. 4-Port USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

This agrees with my observation when using our custom pcb. On the PCB we have this hub.
This hub supports a SMBus mode where i can control several things via i2c.
I disabled power of all ports so the camera device should have no effect on the test.
The disabled power is also present after reboot because the usb3 hub is powered seperately.

The hub also supports a USB2 mode only where USB3 is disabled. When switching to this mode dmesg reports following:

[ 234.561825] usb 2-3: USB disconnect, device number 2
[ 234.710464] tegra-xusb 3610000.xhci: entering ELPG done
[ 306.806180] tegra-xusb 3610000.xhci: Firmware timestamp: 2022-03-16 11:07:43 UTC, Version: 60.13 release
[ 307.069266] usb 2-3: new SuperSpeed Gen 1 USB device number 3 using tegra-xusb
[ 307.089961] usb 2-3: New USB device found, idVendor=0451, idProduct=8240, bcdDevice= 1.10
[ 307.089981] usb 2-3: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 307.094617] hub 2-3:1.0: USB hub found
[ 307.095126] hub 2-3:1.0: 4 ports detected

You see that the USB3 part of the hub is disconnecting from the system. After enabling the USB3 again it will come back.
I had the hope this can fix the issue. But when the Xavier NX is in the non functionall ss state nothing happens when switching USB modes.
This brings me to the assumption this issue is caused by the software.

17

Hello,

now there is some time ago. Do you have any news belonging this topic?
Can you confirm my observations.

18

No, this issue requires lots of iterations so it is hard to analyze. Still not fixed yet. But we are indeed looking into it.

19

Hi.

Could you help test this patch in UEFI?

diff --git a/Silicon/NVIDIA/Drivers/UsbPadCtlDxe/UsbPadCtlTegra194.c b/Silicon/NVIDIA/Drivers/UsbPadCtlDxe/UsbPadCtlTegra194.c
index 034ca8f..a4c20f8 100644
--- a/Silicon/NVIDIA/Drivers/UsbPadCtlDxe/UsbPadCtlTegra194.c
+++ b/Silicon/NVIDIA/Drivers/UsbPadCtlDxe/UsbPadCtlTegra194.c
@@ -2,7 +2,7 @@
 
   USB Pad Control Driver Platform Specific Definitions/Functions
 
-  Copyright (c) 2019-2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+  SPDX-FileCopyrightText: Copyright (c) 2019-2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 
   SPDX-License-Identifier: BSD-2-Clause-Patent
 
@@ -968,6 +968,7 @@
   UINT32                 i, RegData;
   PADCTL_PLAT_CONFIG     *PlatConfig;
   PORT_INFO              *Usb2Ports;
+  PORT_INFO              *Usb3Ports;
 
   if (NULL == This) {
     return;
@@ -976,6 +977,7 @@
   Private    = PADCTL_PRIVATE_DATA_FROM_THIS (This);
   PlatConfig = &(Private->PlatConfig);
   Usb2Ports  = PlatConfig->Usb2Ports;
+  Usb3Ports  = PlatConfig->Usb3Ports;
 
   /* Disable Over Current Handling and VBUS */
   DisableVbus (Private);
@@ -989,6 +991,7 @@
     /* Clear Each PAD's PD and PD_DR Bits */
     RegData  = PadCtlRead (Private, USB2_OTG_PADX_CTL_0 (i));
     RegData |= USB2_OTG_PD;
+    RegData |= USB2_OTG_PD_ZI;
     PadCtlWrite (Private, USB2_OTG_PADX_CTL_0 (i), RegData);
 
     RegData  = PadCtlRead (Private, USB2_OTG_PADX_CTL_1 (i));
@@ -1005,5 +1008,28 @@
   RegData |= USB2_PD_TRK;
   PadCtlWrite (Private, XUSB_PADCTL_USB2_BIAS_PAD_CTL1, RegData);
 
+  /* Power down usb3 part */
+  for (i = 0; i < PlatConfig->NumSsPhys; i++) {
+    if (Usb3Ports[i].PortEnabled == FALSE) {
+      continue;
+    }
+
+    RegData  = PadCtlRead (Private, XUSB_PADCTL_ELPG_PROGRAM_1_0);
+    RegData |= SSPX_ELPG_CLAMP_EN_EARLY (i);
+    PadCtlWrite (Private, XUSB_PADCTL_ELPG_PROGRAM_1_0, RegData);
+
+    gBS->Stall (200);
+
+    RegData  = PadCtlRead (Private, XUSB_PADCTL_ELPG_PROGRAM_1_0);
+    RegData |= SSPX_ELPG_CLAMP_EN (i);
+    PadCtlWrite (Private, XUSB_PADCTL_ELPG_PROGRAM_1_0, RegData);
+
+    gBS->Stall (350);
+
+    RegData  = PadCtlRead (Private, XUSB_PADCTL_ELPG_PROGRAM_1_0);
+    RegData |= SSPX_ELPG_VCORE_DOWN (i);
+    PadCtlWrite (Private, XUSB_PADCTL_ELPG_PROGRAM_1_0, RegData);
+  }
+
   return;
 }
diff --git a/Silicon/NVIDIA/Drivers/UsbPadCtlDxe/UsbPadCtlTegra234.c b/Silicon/NVIDIA/Drivers/UsbPadCtlDxe/UsbPadCtlTegra234.c
index c9fb059..c47036e 100644
--- a/Silicon/NVIDIA/Drivers/UsbPadCtlDxe/UsbPadCtlTegra234.c
+++ b/Silicon/NVIDIA/Drivers/UsbPadCtlDxe/UsbPadCtlTegra234.c
@@ -2,7 +2,7 @@
 
   USB Pad Control Driver Platform Specific Definitions/Functions
 
-  Copyright (c) 2019-2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+  SPDX-FileCopyrightText: Copyright (c) 2019-2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 
   SPDX-License-Identifier: BSD-2-Clause-Patent
 
@@ -1113,6 +1113,7 @@
   UINT32                 i, RegData;
   PADCTL_PLAT_CONFIG     *PlatConfig;
   PORT_INFO              *Usb2Ports;
+  PORT_INFO              *Usb3Ports;
 
   if (NULL == This) {
     return;
@@ -1121,6 +1122,7 @@
   Private    = PADCTL_PRIVATE_DATA_FROM_THIS (This);
   PlatConfig = &(Private->PlatConfig);
   Usb2Ports  = PlatConfig->Usb2Ports;
+  Usb3Ports  = PlatConfig->Usb3Ports;
 
   /* Disable Over Current Handling and VBUS */
   DisableVbus (Private);
@@ -1134,6 +1136,7 @@
     /* Clear Each PAD's PD and PD_DR Bits */
     RegData  = PadCtlRead (Private, USB2_OTG_PADX_CTL_0 (i));
     RegData |= USB2_OTG_PD;
+    RegData |= USB2_OTG_PD_ZI;
     PadCtlWrite (Private, USB2_OTG_PADX_CTL_0 (i), RegData);
 
     RegData  = PadCtlRead (Private, USB2_OTG_PADX_CTL_1 (i));
@@ -1150,5 +1153,28 @@
   RegData |= USB2_PD_TRK;
   PadCtlWrite (Private, XUSB_PADCTL_USB2_BIAS_PAD_CTL1, RegData);
 
+  /* Power down usb3 part */
+  for (i = 0; i < PlatConfig->NumSsPhys; i++) {
+    if (Usb3Ports[i].PortEnabled == FALSE) {
+      continue;
+    }
+
+    RegData  = PadCtlRead (Private, XUSB_PADCTL_ELPG_PROGRAM_1_0);
+    RegData |= SSPX_ELPG_CLAMP_EN_EARLY (i);
+    PadCtlWrite (Private, XUSB_PADCTL_ELPG_PROGRAM_1_0, RegData);
+
+    gBS->Stall (200);
+
+    RegData  = PadCtlRead (Private, XUSB_PADCTL_ELPG_PROGRAM_1_0);
+    RegData |= SSPX_ELPG_CLAMP_EN (i);
+    PadCtlWrite (Private, XUSB_PADCTL_ELPG_PROGRAM_1_0, RegData);
+
+    gBS->Stall (350);
+
+    RegData  = PadCtlRead (Private, XUSB_PADCTL_ELPG_PROGRAM_1_0);
+    RegData |= SSPX_ELPG_VCORE_DOWN (i);
+    PadCtlWrite (Private, XUSB_PADCTL_ELPG_PROGRAM_1_0, RegData);
+  }
+
   return;
 }
diff --git a/Silicon/NVIDIA/Drivers/XhciControllerDxe/XhciControllerDxe.c b/Silicon/NVIDIA/Drivers/XhciControllerDxe/XhciControllerDxe.c
index 74ca67e..c4a1110 100644
--- a/Silicon/NVIDIA/Drivers/XhciControllerDxe/XhciControllerDxe.c
+++ b/Silicon/NVIDIA/Drivers/XhciControllerDxe/XhciControllerDxe.c
@@ -2,7 +2,7 @@
 
   XHCI Controller Driver
 
-  Copyright (c) 2019-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+  SPDX-FileCopyrightText: Copyright (c) 2019-2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 
   SPDX-License-Identifier: BSD-2-Clause-Patent
 
@@ -117,6 +117,9 @@
     return;
   }
 
+  /* Do UsbPadCtlDxe DeInit */
+  Private->mUsbPadCtlProtocol->DeInitHw (Private->mUsbPadCtlProtocol);
+
   Status = gBS->HandleProtocol (Private->ControllerHandle, &gNVIDIAPowerGateNodeProtocolGuid, (VOID **)&PgProtocol);
   if (EFI_ERROR (Status)) {
     return;
diff --git a/Silicon/NVIDIA/Drivers/XusbControllerDxe/XusbControllerDxe.c b/Silicon/NVIDIA/Drivers/XusbControllerDxe/XusbControllerDxe.c
index 4f94c05..0555987 100644
--- a/Silicon/NVIDIA/Drivers/XusbControllerDxe/XusbControllerDxe.c
+++ b/Silicon/NVIDIA/Drivers/XusbControllerDxe/XusbControllerDxe.c
@@ -2,7 +2,7 @@
 
   XUDC Driver
 
-  Copyright (c) 2021-2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+  SPDX-FileCopyrightText: Copyright (c) 2021-2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 
   SPDX-License-Identifier: BSD-2-Clause-Patent
 
@@ -31,7 +31,6 @@
 } XUDC_CONTROLLER_PRIVATE_DATA;
 
 NVIDIA_COMPATIBILITY_MAPPING  gDeviceCompatibilityMap[] = {
-  { "nvidia,tegra194-xudc", &gNVIDIANonDiscoverableXudcDeviceGuid },
   { "nvidia,tegra234-xudc", &gNVIDIANonDiscoverableXudcDeviceGuid },
   { NULL,                   NULL                                  }
 };
1 Like
20

Hi WayneWWW,

now it runs for nearly 20 hours without forgetting the usb3 hub. I think the patch solves the issue.

Thanks a lot.

1 Like
21

Thanks for validation. We will merge this.