USB Devices not detected at PCIe Card, using Ubuntu 16.04

Autonomous Machines Jetson & Embedded Systems Jetson TX1
1

Hi all,

I am using a TX1 with freshly installed L4T using Jetpack 2.3.1. TX1 runs with 64bit Ubuntu 16.04. When I am using a PCIE4XE111 (Gen 2) card. The card is identified, because I can see the four Fresco Controller:

ubuntu@tegra-ubuntu:~$ lspci
00:01.0 PCI bridge: NVIDIA Corporation Device 0fae (rev a1)
01:00.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
02:01.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
02:05.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
02:07.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
02:09.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
03:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev 10)
04:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev 10)
05:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev 10)
06:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev 10)

When I connect a USB stick or camera to the PCIe card, they aren’t detected. With Ubuntu 14.04 on my TX1 I didn’t had that kind of problem.

I also tried a different PCIe card (Gen 3), also with negative results. Do you have any suggestions what I can do to use a PCIe card?

Best
ManuKlause

2

It might be that the driver for that card was there by default in the older version, but not enabled by default in the newer kernel. There were many changes when the older R23.2 32-bit user space migrated to 64-bit…kernel config would have changed. I’d start by finding out which driver that card uses, and comparing it to the content of “/proc/config.gz”.

3

Hi!

I have the same problem with 2 different PCIe cards too. I have collected the kernel configs, but dont see anything suspicious at first sight.
But I am curious, Would the USB controllers be available (via lsusb or lspci) if the driver of the card was not loaded?

4

I tried to upload the two kernel configs, but they were to large to insert as code blocks. :(

5

About config file posting: Perhaps you could rename the configs with a “.txt” or “.rtf” extension first, and then attach them instead of pasting as a code block. Or, maybe use the “zip” program to compress if direct “.txt” file attach fails (I’d use bzip2 instead, but I don’t know if the forum spam filters allow that file extension to be attached…it’s a question of what the spam filters allow as a file attachment type…file name suffix matters for that purpose). If you own a post which already exists you can hover your mouse over the upper right corner’s “quote” icon and you’ll also see a paper clip icon…the paper clip icon lets you attach files to your preexisting post.

The PCIe has its own driver; if that driver works and enumerates a PCIe device, then it is possible for the device to continue. Enumeration is the process of taking notes on what is plugged in, and presenting a description of the device to the next stage. lspci describes what was enumerated.

If the device on PCIe is a USB HUB, the PCIe enumeration will result in USB drivers then getting a chance to claim ownership of the USB portion of the hardware (PCIe drivers told USB drivers about this device during enumeration). If a driver knows it can handle the HUB, then it will take ownership of the HUB and the HUB will work. lsusb shows which HUBS are seen, and what devices were enumerated for the next stage to work with.

If a device is plugged into USB by this HUB, then an event will be generated during enumeration; this will in turn offer any device drivers a chance to take ownership of whatever device is behind the USB pipe. If and only if a driver answers and binds to the device behind the USB pipe will the device work. lsusb can show an end device which has no driver…USB does not care if enumeration found a driver or not.

You have a chain of drivers and events to announce hardware, and if any fails, the end device fails. It is a chain of drivers, enumeratons, and new drivers being appended as the next link in the chain; this keeps going until reaching the endpoint of the chain (the final device). The chain can be left dangling without the end device having a driver, yet the previous stages are working 100%.

The kernel config is to know that drivers are present and able to take ownership of their respective devices when enumeration announces something present. Some devices are only cold-plug, and must be present upon power up to enumerate; other devices are hot-plug, and can enumerate as added (plus they have a graceful removal). If USB end devices have a driver, then the USB enumeration should have resulted in the driver loading and taking ownership of the end device, along with all of the side effects (e.g., creating a device special file or being available in some configuration). Specific end device enumeration behavior depends on the specific driver.

One cause of a failure is lack of a driver (see kernel config…a successful enumeration does not matter if there is no driver). Another cause of failure is insufficient signal quality (perhaps cables too long, power delivery insufficient quality, so on). The last category of failure would be failure to enumerate because of power up order (power on sequence matters if not hot plug; hot plug prior to a driver being available would also fail enumeration, so even hot plug has prerequisites, though it would be rare for this to occur).

6

I am attaching the config files of the 32bit and 64bit kernels to this comment.
config32.txt (115 KB)
config64.txt (116 KB)

7

I looked at some of the config and did not see anything on 32-bit or 64-bit specific to that card (this doesn’t mean such a thing doesn’t exist, it just means I didnt find it). I also looked on the internet and did not find anything. Finally, I searched through kernel source code on the R23.2 kernel (you said it worked on 32-bit, this would be the kernel) and found it is part of:

usb/host/xhci-pci.c

So it seems xhci handles this, and you have xhci exactly matching on both kernel configs, so xhci config isn’t the problem unless the particular support somehow got split off into a new config on the next kernel (this is highly unlikely between those particular kernels).

The reason I found the FL1100 in the kernel source is because it has “quirks” which had to be adjusted for. The quirk is adjusted for depending on revision of the board. Here’s an excerpt from the kernel:

"has broken MSI implementation\n"

I see no difference in the code between R23.2 and R24.2.1, so we know both kernel config and quirk differences can be eliminated. What remains are changes to the rest of the XHCI code or the environment being operated in as the issue (code could remain unchanged and still differ in behavior because of the environment or data).

I think for this to be debugged someone would have to have the specific card…there isn’t a lot which could be done to know why the switch from 32-bit user space to 64-bit user space allows the card to be recognized but not function. Unless you have a strong need for this specific card you might want to consider getting a different card with a different known chipset.

8

Thank you for your great effort, I really appreciate it!

I have considered using a different card, but I have experienced the same issue with two other cards as well. Both of them have Renesas USB Host Controllers. One of them has the same PLX PCIe bridge IC as above, the other one has Pericom PCIe bridge IC.

So it seems to be a general problem with PCIe USB cards, and not just with that specific card.

9

I have also tried a Gen 3 PCIe card. The PCI bridge and USB controller (a ASMedia Technology Ind. ASM1142 USb 3.1 Controller) are recognized via “lspci” but ain’t working. Thanks a lot for your answer.

10

Because lspci works you can eliminate the PCIe bridge as being an issue in this case. The code in question would be entirely for handling the Renesas USB chipset. One thing I did notice while looking at Renesas chipset issues is that quirks are tied to specific revisions; this may mean other revisions behave differently under different circumstances…perhaps there are other unknown quirks in need of adjustment for this chipset if other revisions are used.

PCIe v3 is not supported on a JTX1. Such cards should back off and run at v2 speeds though, they are backwards compatible. Having lspci work means the PCIe is functioning, so once more, this is unlikely a PCI issue (“sudo lspci -vvv” can show if errors are occurring). If the USB chipset part of such a card is also new (PCIe v3 is new relative to the JTX1 and its kernel), then I would expect odds are in favor of no driver for that chipset (or at least not one being enabled by default…perhaps it would work if a kernel module is added).

On any PCIe USB card I would expect the PCIe part of it to work regardless of being older v2 or newer v3 PCIe standards. I would consider finding out what USB chipset is used on any prospective USB PCIe card and researching what driver is used for that chipset, then checking if this kernel can be configured for that driver prior to purchasing it (some working drivers may have been introduced in a newer kernel, so unless you backport those drivers would not help anyway). The Renesas chipset could probably be made to work, but it seems to be hit and miss as to how much effort and expertise is needed for that.

11

Thank you for your help so far.

Finally, this patch solved my problem:
[url]https://devtalk.nvidia.com/default/topic/967515/jetson-tx1/l4t-r24-2-pcie-iommu-not-working-when-switches-bridges-are-present/post/4982441/#4982441[/url]

12

@blabla019
Thank you for sharing your solution for the problem.
Could you please let me know what should I do with the link?
I have a same issue…

Thank you.