How to connect 4 USB Modem in USB3.1 on a jetson nano (host controller resources problem)

Autonomous Machines Jetson & Embedded Systems Jetson Nano
,
#1

Hello,

I’ve a Jetson nano on which I need to connect 4 Quectel EM12 modem.
One modem works perfectly. Two also works without any problem. But if I connect the third (and same when I connect 4) I have a “Not enough host controller resources for new device state” error …

Each modem have 5 usb interfaces, but I only need 2 (QMI and AT command).
I’ve tried to unlink driver to interface I don’t need, but the interface is still connected and took resources to the host controller.

I’ve nothing else connected on the 3.0 root hub.

Do you have any idea how can I connect these 4 modems ? By blocking unused interface or by bypassing the resources limitation (if possible), or other ?

Thanks for your help.

#2

Likely it won’t work, but for a more solid answer, post the output of both “lsusb -t” and “lsusb”.

Additionally, under the output of “lsusb”, note that each device has an “ID”. One example would be “0955:7020” (the “0955” is for NVIDIA, the “7020” is the ID for the Nano). Find a modem which works, and if possible, one which fails, and run this and post the output (but use the actual ID you see and not my example…I am naming the example just for context):
sudo lsusb -d 0955:7020 -vvv

Note that you can capture the output of any of those commands in a file (preferable to copy and paste) and attach it to the forum if you add logging like this:

sudo lsusb -d 0955:7020 -vvv 2>&1 | tee log_verbose.txt
lsusb -t 2>&1 | tee log_tree.txt
lsusb 2>&1 | tee log_all.txt

A problem which will probably not have a workaround is that all traffic eventually goes through a root HUB. It is a bit like an electrical extension cord rated for a certain load…if you plug in a lot of items which don’t cause much load, then it will handle the job, but if you plug several heavy appliances, then having extra sockets to plug causes a failure from exceeding specifications.

In the tree view note that lines end with something like “480M” or “5000M” or “10000M”. That is the speed it is running at in Mbit/s. “480M” corresponds to USB2, “5000M” could be called either “USB3” or “USB3.1 gen 1” (the standards merged). The “10000M” would be “USB3.1 gen 2”. Everything on a tree branch combines in resources used on the root HUB (and you will see “root_hub” for the class in the tree view). There is also overhead, so actual combined units will exceed ratings somewhat before it looks like it would.

In the USB2 days (and even now) most people have devices which barely use the available 480M bandwidth. Things like keyboards and mice are trivial. External hard drives would an example of something with significant bandwidth consumption.

Some USB3 devices are capable of throttling back to slower speeds if more bandwidth is not available. An example might be a camera being able to support a high resolution and frame rate might work if it is only using a lower resolution and/or frame rate. To see details of the device and its capabilities is why we look at the fully verbose “sudo lsusb -vvv”. Perhaps there is some port rearrangement possible, but it is not likely if all device use their full bandwidth

It is also possible to see the “not enough host controller resources” if there are too many devices even if the total bandwidth is not exceeded, but this is unlikely.

One more log would be useful. Monitor “dmesg --follow”, start with no modems installed, and add modems one at a time. Each one will show up in the log. When you get one which fails, you will need to copy and paste that log of what happens when you attach the device. Details in that might help.

Also, some devices are actually multiple devices using just a single wire. For example, a programmable keyboard which can emulate a mouse would likely have three devices show up: The keyboard device, the keyboard programming interface, and the mouse the keyboard emulates. So don’t be surprised if a modem has more than one device show up (the dmesg log will indicate what is seen upon plugin).

#3

Hello,

Thanks a lot for this complete answer @linuxdev .

This is the “lsusb” :

Bus 002 Device 005: ID 2c7c:0512 Quectel Wireless Solutions Co., Ltd.
Bus 002 Device 004: ID 2c7c:0512 Quectel Wireless Solutions Co., Ltd.
Bus 002 Device 007: ID 2c7c:0512 Quectel Wireless Solutions Co., Ltd.
Bus 002 Device 006: ID 2c7c:0512 Quectel Wireless Solutions Co., Ltd. 4-Port USB 3.1 Hub
Bus 002 Device 003: ID 0424:5804 Microchip Technology, Inc. (formerly SMSC)
Bus 002 Device 002: ID 0bda:0411 Realtek Semiconductor Corp. 4-Port USB 3.1 Hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 8087:0a2b Intel Corp.
Bus 001 Device 004: ID 1e0e:9001 Qualcomm / Option
Bus 001 Device 006: ID 0424:2840 Microchip Technology, Inc. (formerly SMSC)
Bus 001 Device 005: ID 0424:2804 Microchip Technology, Inc. (formerly SMSC) 4-Port USB 2.1 Hub
Bus 001 Device 002: ID 0bda:5411 Realtek Semiconductor Corp. 4-Port USB 2.1 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

And with the “-t” parameter :

/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 5000M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
        |__ Port 2: Dev 3, If 0, Class=Hub, Driver=hub/6p, 5000M
            |__ Port 3: Dev 4, If 3, Class=Vendor Specific Class, Driver=, 5000M
            |__ Port 3: Dev 4, If 1, Class=Vendor Specific Class, Driver=, 5000M
            |__ Port 3: Dev 4, If 4, Class=Vendor Specific Class, Driver=qmi_wwan, 5000M
            |__ Port 3: Dev 4, If 2, Class=Vendor Specific Class, Driver=option, 5000M
            |__ Port 3: Dev 4, If 0, Class=Vendor Specific Class, Driver=, 5000M
            |__ Port 4: Dev 5, If 0, Class=Vendor Specific Class, Driver=, 5000M
            |__ Port 4: Dev 5, If 1, Class=Vendor Specific Class, Driver=, 5000M
            |__ Port 4: Dev 5, If 2, Class=Vendor Specific Class, Driver=option, 5000M
            |__ Port 4: Dev 5, If 3, Class=Vendor Specific Class, Driver=, 5000M
            |__ Port 4: Dev 5, If 4, Class=Vendor Specific Class, Driver=qmi_wwan, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/5p, 480M
    |__ Port 2: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 2: Dev 5, If 0, Class=Hub, Driver=hub/5p, 480M
            |__ Port 5: Dev 6, If 0, Class=Vendor Specific Class, Driver=, 480M
        |__ Port 3: Dev 4, If 3, Class=Vendor Specific Class, Driver=, 480M
        |__ Port 3: Dev 4, If 1, Class=Vendor Specific Class, Driver=option, 480M
        |__ Port 3: Dev 4, If 4, Class=Vendor Specific Class, Driver=, 480M
        |__ Port 3: Dev 4, If 2, Class=Vendor Specific Class, Driver=option, 480M
        |__ Port 3: Dev 4, If 0, Class=Vendor Specific Class, Driver=, 480M
        |__ Port 3: Dev 4, If 5, Class=Vendor Specific Class, Driver=qmi_wwan, 480M
    |__ Port 3: Dev 3, If 0, Class=Wireless, Driver=, 12M
    |__ Port 3: Dev 3, If 1, Class=Wireless, Driver=, 12M

I’ve also attached the log file. But as the 4 modems have the same vendor_id and device_id, the 4 modems are in the log.

And this is the dmesg :

[  195.721540] usb 2-1.2.3: new SuperSpeed USB device number 4 using tegra-xusb
[  195.750586] usb 2-1.2.3: New USB device found, idVendor=2c7c, idProduct=0512
[  195.757691] usb 2-1.2.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  195.765199] usb 2-1.2.3: Product: EM12-G
[  195.769146] usb 2-1.2.3: Manufacturer: Quectel
[  195.773633] usb 2-1.2.3: SerialNumber: 0123456789ABCDEF
[  195.862728] option 2-1.2.3:1.2: GSM modem (1-port) converter detected
[  195.869459] usb 2-1.2.3: GSM modem (1-port) converter now attached to ttyUSB2
[  195.895931] qmi_wwan 2-1.2.3:1.4: cdc-wdm1: USB WDM device
[  195.902232] qmi_wwan 2-1.2.3:1.4 wwan0: register 'qmi_wwan' at usb-70090000.xusb-1.2.3, WWAN/QMI device, 02:a9:07:ba:d9:34
[  195.920336] qmi_wwan 2-1.2.3:1.4 wwan-aux4: renamed from wwan0
[  196.141485] usb 2-1.2.4: new SuperSpeed USB device number 5 using tegra-xusb
[  196.166623] usb 2-1.2.4: New USB device found, idVendor=2c7c, idProduct=0512
[  196.173729] usb 2-1.2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  196.181241] usb 2-1.2.4: Product: EM12-G
[  196.185183] usb 2-1.2.4: Manufacturer: Quectel
[  196.189672] usb 2-1.2.4: SerialNumber: 0123456789ABCDEF
[  196.278634] option 2-1.2.4:1.2: GSM modem (1-port) converter detected
[  196.285362] usb 2-1.2.4: GSM modem (1-port) converter now attached to ttyUSB3
[  196.311807] qmi_wwan 2-1.2.4:1.4: cdc-wdm2: USB WDM device
[  196.317943] qmi_wwan 2-1.2.4:1.4 wwan0: register 'qmi_wwan' at usb-70090000.xusb-1.2.4, WWAN/QMI device, 02:a9:07:ba:d9:34
[  196.335962] qmi_wwan 2-1.2.4:1.4 wwan-aux5: renamed from wwan0
[  196.557461] usb 2-1.2.1: new SuperSpeed USB device number 6 using tegra-xusb
[  196.582699] usb 2-1.2.1: New USB device found, idVendor=2c7c, idProduct=0512
[  196.590327] usb 2-1.2.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  196.597831] usb 2-1.2.1: Product: EM12-G
[  196.601771] usb 2-1.2.1: Manufacturer: Quectel
[  196.606258] usb 2-1.2.1: SerialNumber: 0123456789ABCDEF
[  196.693362] usb 2-1.2.1: Not enough host controller resources for new device state.
[  196.701123] usb 2-1.2.1: can't set config #1, error -12
[  196.929519] usb 2-1.2.2: new SuperSpeed USB device number 7 using tegra-xusb
[  196.958545] usb 2-1.2.2: New USB device found, idVendor=2c7c, idProduct=0512
[  196.965625] usb 2-1.2.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  196.973136] usb 2-1.2.2: Product: EM12-G
[  196.977079] usb 2-1.2.2: Manufacturer: Quectel
[  196.981539] usb 2-1.2.2: SerialNumber: 0123456789ABCDEF
[  197.068216] usb 2-1.2.2: Not enough host controller resources for new device state.
[  197.075939] usb 2-1.2.2: can't set config #1, error -12

As you can see, the device 2-1.2.3 and 2-1.2.4 connect successfully, but 2-1.2.1 and 2-1.2.2 have not enough resources.

These modems have 5 (or 6 interfaces) but have 22 endpoints. I’m only using 2 interfaces (so 6 endpoints), so what I’m trying to do now is add a filter in the xhci kernel driver to discard endpoints I don’t use (based on vendor id, device id and endpoint address).
But if there is another solution, more clean solution, it would be better than edit a kernel driver ^^.

Thanks a lot.

log_verbose.txt (38.6 KB)

#4

Hello,
I’ve made my patch on the kernel and now I’m able to use my 4 modems on the USB3 host controller.

--- a/drivers/usb/host/xhci.c   2022-03-11 10:28:56.600696446 +0100
+++ b/drivers/usb/host/xhci.c   2022-03-14 09:47:09.919813358 +0100
@@ -1742,6 +1742,35 @@
        struct xhci_virt_device *virt_dev;
        int ret = 0;

+       //EM12G
+       if(udev->descriptor.idVendor == 0x2c7c && udev->descriptor.idProduct == 0x0512 &&
+               ep->desc.bEndpointAddress != 0x88 &&
+               ep->desc.bEndpointAddress != 0x8e &&
+               ep->desc.bEndpointAddress != 0x0f &&
+               ep->desc.bEndpointAddress != 0x85 &&
+               ep->desc.bEndpointAddress != 0x84 &&
+               ep->desc.bEndpointAddress != 0x03)
+       {
+               dev_warn(&udev->dev, "Discard endpoint 0x%x : 0x%x - 0x%x.\n", udev->descriptor.idVendor, udev->descriptor.idProduct, ep->desc.bEndpointAddress);
+               return 0;
+       }
+
+       //SIMCOM
+       if(udev->descriptor.idVendor == 0x1e0e && udev->descriptor.idProduct == 0x9001 &&
+               ep->desc.bEndpointAddress != 0x8b &&
+               ep->desc.bEndpointAddress != 0x8a &&
+               ep->desc.bEndpointAddress != 0x06 &&
+               ep->desc.bEndpointAddress != 0x83 &&
+               ep->desc.bEndpointAddress != 0x82 &&
+               ep->desc.bEndpointAddress != 0x02 &&
+               ep->desc.bEndpointAddress != 0x85 &&
+               ep->desc.bEndpointAddress != 0x84 &&
+               ep->desc.bEndpointAddress != 0x03)
+       {
+               dev_warn(&udev->dev, "Discard endpoint 0x%x : 0x%x - 0x%x.\n", udev->descriptor.idVendor, udev->descriptor.idProduct, ep->desc.bEndpointAddress);
+               return 0;
+       }
+
        ret = xhci_check_args(hcd, udev, ep, 1, true, __func__);
        if (ret <= 0) {
                /* So we won't queue a reset ep command for a root hub */

It works well, but it is not a pretty solution

#5

The above says all Quectel are on the same bus (same root HUB), bus 002.

The above says there is a USB HUB feeding another USB HUB, and this is what has all devices for that root HUB. Sometimes a HUB is built into a device, but can you say if there is a physically external HUB connected to another HUB, with devices in one end, and the opposite end cable going to the Jetson? If so, what would the reason be for the extra HUB? It does show that all devices are truly running in USB3 mode (USB3.1 gen. 1), so we know that signal quality is not an issue, but it may still be that during operation devices are consuming more bandwidth than the cable can provide.

I’d like to also mention that each device is an “endpoint”. So is each HUB, so when a root HUB does this:
root_hub->hub->device
…then this uses two end points even though you have only one “device” (the HUB is likely considered an endpoint so far as resources being consumed). Also, an actual endpoint might be more than one endpoint depending on its internal layout.

The other root HUB is only USB2, so you couldn’t use that unless the device you want to use it with is ok to run at slower USB2 speeds.

Within the dmesg log I see it is creating ttyUSB2 and ttyUSB3, and not starting with ttyUSB0. Would it be possible to plug in all devices and get a “dmesg” log of the entire boot? What I see there is useful, it really does seem like you are hitting a device count limit and not a signal issue.

My next question is about device count. In terms of the USB3 HUB, I have already asked if one HUB feeds another. Keep in mind that in part I am wondering if the unused ports on a HUB might be counted as a device (I’m not positive) because if this is the case, then by eliminating a HUB one would suddenly gain more device resources (the amount of unused ports), but I am speculating.

Are any other devices on this USB3 HUB something you can remove, at least temporarily for testing? I’d like to reduce total USB device count.

#6

Hello. Thanks for your help.

I’m not using any hub, hubs are in the device :

  • Bus 02.Port 1: Dev 1 is the root hub in the Jetson USB controller
  • Bus 02.Port 1: Dev 2 is the USB3.1 Hub on the Jetson carrier board
  • Bus 02.Port 1: Dev 3 is the USB3.1 Hub on the modem carrier board (can handle up to 4 MiniPCIe modems)

So I can not remove any hub :(.

USB3 is mandatory because my modems will made a 5G network survey (speedtest, signal quality, etc) so 480M will not be enough.

I think it start from ttyUSB2 because I removed /unlink some interfaces from their drivers. As you can see :

/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 5000M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
        |__ Port 2: Dev 3, If 0, Class=Hub, Driver=hub/6p, 5000M
            |__ Port 3: Dev 4, If 3, Class=Vendor Specific Class, Driver=, 5000M
            |__ Port 3: Dev 4, If 1, Class=Vendor Specific Class, Driver=, 5000M
            |__ Port 3: Dev 4, If 4, Class=Vendor Specific Class, Driver=qmi_wwan, 5000M
            |__ Port 3: Dev 4, If 2, Class=Vendor Specific Class, Driver=option, 5000M
            |__ Port 3: Dev 4, If 0, Class=Vendor Specific Class, Driver=, 5000M

If 0, 1 and 3 have no driver.
All my devices are pluggued so I guess it’s the reason.

I think the problem here is the endpoint count (not the interface but endpoint).
Because the Quectel modem seems to use 22 endpoints (but I think Hub endpoints used by the modem are also count). With my kernel patch, it now only use 6 endpoints (Hub endpoints are not counted).

If I believe the xHCI driver of the jetson kernel, new endpoints are refused if the endpoints count are greater than 32 (0x01 < 5).

#7

It may indeed be just the number of endpoints. The driver itself is a custom driver, and not any sort of standard class, so your patch may end up being the only way you have to change things. Despite patching not being something desirable just to run the board, will it do the job for you?

Question for NVIDIA: What is the official maximum number of USB endpoints for the Nano?

#8

Yes it does the job pretty well.
I got no error since I use the patched kernel (about 5 days) and it saved my life for this project (it was a “No go” issue)

#9

Hi,
For end point limit, please check
USB3 xhci driver device count limitation - #2 by DaneLLL

#10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.