Jetson TK1 - USB3 PCIe card not working

Hello,

I’m trying to connect two USB 3.0 point grey cameras on my Jetson TK1 kit via this PCIe USB 3.0 card I found on different topics listed as compatible with the TK1 (running on L4T 21.5 freshly flashed).

Ref : Amazon.com
Topic example : https://devtalk.nvidia.com/default/topic/903602/jetson-tk1/usb3-via-pcie-card/2

Though I still can’t make any device recognized by the TK1 via this card.
The card is listed in lspci and the usb ports too in lsusb but nothing happen when I connect a device to a USB 3.0 port.

Is there any tweak or fix to make it functional ?

Thanks,
Enzo.

Research which driver supports this card on any other architecture. Then compare it to the running system’s configuration:

zcat /proc/config.gz | less

It is possible that you simply need to compile the correct module. Embedded systems do not build in everything made you would have to do that yourself. You can get help here, but you’ll need to know which driver you want.

Hi and thanks for your help,

I’ve already managed to find the driver for this card, it was listed on a CD and I managed to find it on the web.
http://drivers-download.com/en/index.php
research code : DL-0406101
I did followed the instruction in the readme files and it was only a matter of copying files to the right place. Do I have to make further commands because TK1 is an embedded system ?

In addition, here are lspci, lsusb -t and a part of zcat where I found something related to Renesas. All commands were asked with the card plugged (listed in lspci as Renesas) and a camera device plugged to the USB 3 female of the card. The keyboard is plugged directly to the USB 3.0 port of the TK1.

ubuntu@tegra-ubuntu:~$ lsusb
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 007: ID 05ac:0220 Apple, Inc. Aluminum Keyboard (ANSI)
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
ubuntu@tegra-ubuntu:~$ lsusb# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
CONFIG_USB_XHCI_HCD=y
# CONFIG_USB_XHCI_HCD_DEBUGGING is not set
CONFIG_TEGRA_XUSB_PLATFORM=y
CONFIG_USB_EHCI_HCD=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
CONFIG_USB_EHCI_PCI=y
CONFIG_USB_EHCI_TEGRA=y
# CONFIG_USB_EHCI_HCD_PLATFORM is not set
# CONFIG_USB_OXU210HP_HCD is not set
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_ISP1760_HCD is not set
# CONFIG_USB_ISP1362_HCD is not set
# CONFIG_USB_OHCI_HCD is not set
# CONFIG_USB_UHCI_HCD is not set
# CONFIG_USB_SL811_HCD is not set
# CONFIG_USB_R8A66597_HCD is not set
# CONFIG_USB_EHCI_ONOFF_FEATURE is not set
# CONFIG_USB_MUSB_HDRC is not set
# CONFIG_USB_RENESAS_USBHS is not set
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 007: ID 05ac:0220 Apple, Inc. Aluminum Keyboard (ANSI)
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
ubuntu@tegra-ubuntu:~$ lsusb -t
/:  Bus 05.Port 1: Dev 1, Class=root_hub, Driver=tegra-ehci/1p, 480M
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/2p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=tegra-xhci/6p, 480M
    |__ Port 3: Dev 7, If 0, Class=Human Interface Device, Driver=usbhid, 12M
    |__ Port 3: Dev 7, If 1, Class=Human Interface Device, Driver=usbhid, 12M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
CONFIG_USB_XHCI_HCD=y
# CONFIG_USB_XHCI_HCD_DEBUGGING is not set
CONFIG_TEGRA_XUSB_PLATFORM=y
CONFIG_USB_EHCI_HCD=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
CONFIG_USB_EHCI_PCI=y
CONFIG_USB_EHCI_TEGRA=y
# CONFIG_USB_EHCI_HCD_PLATFORM is not set
# CONFIG_USB_OXU210HP_HCD is not set
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_ISP1760_HCD is not set
# CONFIG_USB_ISP1362_HCD is not set
# CONFIG_USB_OHCI_HCD is not set
# CONFIG_USB_UHCI_HCD is not set
# CONFIG_USB_SL811_HCD is not set
# CONFIG_USB_R8A66597_HCD is not set
# CONFIG_USB_EHCI_ONOFF_FEATURE is not set
# CONFIG_USB_MUSB_HDRC is not set
# CONFIG_USB_RENESAS_USBHS is not set

Thanks,
Enzo.

Copying pre-built binary drivers won’t work unless they were compiled against this architecture. In most cases the manufacturers only provide such a binary image for a desktop PC (x86_64), and not for the 64-bit ARM (arm64/aarch64). If you placed such a driver in your modules directory (“/lib/modules/$(uname -r)/”), then you will see it loaded with “lsmod”. I doubt it loaded.

Finding a driver name for something already in the kernel is your desired outcome. The driver name could be searched for by its symbol in “/proc/config.gz”. Then you would also be able to know if it was already there or not. Probably not, but then you’d take this exact current configuration and use a config editor to add the correct driver and build the module or a new kernel with that driver integrated (depending on what format you compiled it as).

Do you know a name for the driver from the existing Linux kernel? If not, then does the driver you found have a source code format which can be compiled against the current kernel source?

Hello,

So I managed to find that the pre build driver on the Kernel was named “renesas_usbhs” and I did a clean .config install with the help of this video :

I do have the drivers installed in “/usr/src/kernel/drivers/usb/”
but the chipset is still not working after reboot and no drivers is loaded in lsmod.

Do I have to make the .config install in “/lib/modules/$(uname -r)/kernel/drivers/usb” ?

Thanks,
Enzo.

The first thing is to remember the symbol for that driver feature, as listed in the existing “/proc/config.gz”, is:

# CONFIG_USB_RENESAS_USBHS is not set

You will want to copy your running system’s “/proc/config.gz” somewhere, and then make a copy of that for actually working with…don’t worry about the copy yet, but do save this file with some safe name for future use. I named my copy “config_3.10.40-gb271e8f.gz” because “uname -r” is “3.10.40-gb271e8f”. You will want to remember this because it implies this kernel finds its modules in “/lib/modules/3.10.40-gb271e8f/”. The “3.10.40” is from the kernel source. Whoever compiled it had the CONFIG_LOCALVERSION option set to “-gb271e8f”.

You could cross compile as per the docs, but I’m only going to give information as if you are compiling directly on the JTK1. The tool you are going to use is actually a make target in the kernel source, but you’ll need to make sure you have this package installed:

sudo apt-get install libncurses5-dev

You top level directory is apparently “/usr/src/kernel/”. When I write “$SRC” or “${SRC}” pretend it is that directory location. You could, for just one terminal:

export SRC="/usr/src/kernel/"

Then:

cd $SRC
make mrproper
cp /where/ever/it/is/config_3.10.40-gb271e8f.gz ${SRC}
gunzip config_3.10.40-gb271e8f.gz
mv config_3.10.40-gb271e8f.gz .config
# ...you could hand edit CONFIG_LOCALVERSION, but showing from "make nconfig"...there
# are many cases where hand editing will bite back, but CONFIG_LOCALVERSION is safe to edit.
# There are many variations of this tool, the $SRC/README mentions them.
make nconfig
# Look at the help menu...one is for searching for symbols. Search for LOCALVERSION
# (or CONFIG_LOCALVERSION). Navigate to this, then set it to "<b>-gb271e8f</b>"
# Now search for "CONFIG_USB_RENESAS_USBHS". Use the "y" key to integrate this into the kernel,
# or instead use the "m" key to build as a module and build a module instead. Not all code
# can be built as a module, I suspect this driver can. So I'm assuming "m".
#
# You don't necessarily have to build the whole kernel, but it is a sanity check...it's a waste
# of time to build only a module and put it in place when something else is wrong. To build the
# kernel using 4 CPU cores:
make -j4 Image
# If you skip this previous step you instead need this:
make modules_prepare
# To now build modules:
make -j4 modules
# Your driver should now exist in its directory.

If you read the official docs it will mention things you can set to have the compile output go somewhere else and leave your original directory pristine. If you export the given variables, then this builds the Image and intermediate files in “$TEGRA_KERNEL_OUT”:

make -j4 O=$TEGRA_KERNEL_OUT Image

…this would place the modules in “$TEGRA_MODULES_OUT”:

make modules_install INSTALL_MOD_PATH=$TEGRA_MODULES_OUT

One trick of building with output in a different directory is that you need your “.config” in that different directory and the original source pristine (“make mrproper” cleans source completely).

So alternate commands run as a non-root user:

mkdir ~/build
mkdir ~/modules
export TEGRA_KERNEL_OUT="~/build"
export TEGRA_MODULES_OUT="~/modules"
export SRC="/usr/src/kernel/"
cp /where/ever/it/is/config_3.10.40-gb271e8f.gz $TEGRA_KERNEL_OUT
<b>cd $SRC</b>
sudo make mrproper
make O=$TEGRA_KERNEL_OUT nconfig
# ...do your config stuff...
make -j4 O=$TEGRA_KERNEL_OUT Image
make -j4 O=$TEGRA_KERNEL_OUT modules
make O=$TEGRA_KERNEL_OUT modules_install INSTALL_MOD_PATH=$TEGRA_MODULES_OUT
cd $TEGRA_MODULES_OUT
find . -type f -name "whatever_it_is.ko"
# Put this in the correct place.

The available documentation is just a variation on this for cross compile. This adds for example variables to say it is for a different architecture while building from a host PC.

Can you please share the output of ‘sudo lspci -vv’ ? (Though you already mentioned that you see usb ports of this connected card getting listed in lsusb command, I just want to confirm that there is no issue as such with the PCIe device driver for this card)
Also, I see a 4-pin power port (white color). Is it supplied with power from SMPS or an appropriate source?

@linuxdev

I did a clean build of the kernel and modules as you described and so the drivers are installed at the right place but afterwards I still can read in /proc/config.gz :

CONFIG_USB_RENESAS_USBHS is not set

@vidyas
This is the output of lsusb -vv.

00:00.0 PCI bridge: NVIDIA Corporation TegraK1 PCIe x4 Bridge (rev a1) (prog-if 00 [Normal decode])
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
    Memory behind bridge: 32200000-322fffff
    Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
    BridgeCtl: Parity+ SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
        PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
    Capabilities: <access denied>
    Kernel driver in use: pcieport

01:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) (prog-if 30 [XHCI])
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin A routed to IRQ 130
    Region 0: Memory at 32200000 (64-bit, non-prefetchable) 
    Capabilities: <access denied>
    Kernel driver in use: xhci_hcd

02:00.0 PCI bridge: NVIDIA Corporation TegraK1 PCIe x1 Bridge (rev a1) (prog-if 00 [Normal decode])
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Bus: primary=02, secondary=03, subordinate=03, sec-latency=0
    I/O behind bridge: 00010000-00010fff
    Memory behind bridge: 32100000-321fffff
    Prefetchable memory behind bridge: 0000000012100000-00000000121fffff
    Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
    BridgeCtl: Parity+ SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
        PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
    Capabilities: <access denied>
    Kernel driver in use: pcieport

03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
    Subsystem: Realtek Semiconductor Co., Ltd. Device 0123
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin A routed to IRQ 642
    Region 0: I/O ports at 10000 
    Region 2: Memory at 32100000 (64-bit, non-prefetchable) 
    Region 4: Memory at 12100000 (64-bit, prefetchable) 
    Capabilities: <access denied>
    Kernel driver in use: r8169

The chipset does have a 4-pin molex power adaptator on it but the documentation mentions it as a power bonus, not a requirement. I’ll try to aliment it anyway in order to test. Is it possible to connect it directly to the Jetson TK1 with this kind of cable ?

Thanks,
Enzo.

If you built a module at a later date, then the running kernel image will not know about this in “/proc/config.gz”…but it won’t matter. If you have your original set of modules for the rest of the system in “/lib/modules/$(uname -r)/”, then adding the new module there and running “sudo depmod -A” should make the system aware of the module. Once this is done, then “modprobe” can be used to actually insert the module by name. Be sure to check the actual module name, I’m just using an example and am not sure how your module file is really named/spelled, but here is an example (whatever your “filename.ko” is substitute for the “filename”):

sudo modprobe filename
# Now see if the module is visible:
lsmod

Do you get any errors or output from modprobe? Where did you place the actual “.ko” file? What is your output from “uname -r”?

Note that you need to use “sudo” when you run lspci if you are to see everything. Some inspection of what you have follows.

The PCIe bus is working and sees the card:

01:00.0 USB controller: Renesas Technology Corp.

…the card is in slot “01:00.0”. Thus you could limit “lspci” to just this card and show all for this card via:

# Summary detail:
sudo lspci -s 01:00.0
# Medium detail:
sudo lspci -s 01:00.0 -v
# More details:
sudo lspci -s 01:00.0 -vv
# <b>Maximum</b> detail:
sudo lspci -s 01:00.0 -vvv

This implies that perhaps drivers are in place:

Kernel driver in use: xhci_hcd

…however, you might want to run “sudo lspci -s 01:00.0 -vvv” and post the result since the shorter version won’t show things like errors.

You probably do need that power cable connected (the card might not use PCIe power or might still require both power points connected), though perhaps this might only matter once a USB3 device is plugged in (USB3 allows devices to consume more power than USB2, but this doesn’t mean the device will need more power).

The TK1’s SATA power connector (the “Molex” connector near the mini-PCIe slot) would work for this for any USB2 device, I don’t know if it has enough power available for USB3 (chances are it is ok for a single camera…probably even for two USB3 cameras). There may be some USB3 devices drawing more power, but perhaps trying it out first on lower powered devices would be a good test (use USB2 or lower power USB3 devices, e.g., a USB2 camera would consume less power than a USB3 camera…I doubt a USB3 camera would draw much more though and should be ok…USB3 hard drives would be a problem). Then run “lsusb -t” again and see if the PCIe HUB shows up.

FYI, the standards are such that if the card is bus powered the maximum allowed power delivery is less than if it is externally powered. Should it turn out that you cross that boundary and consume more power than the bus powered standard allows, then the device will essentially disconnect and not show up until external power is added (software won’t enumerate the camera and the drivers won’t know it is there).

Hello,

I did manage to connect an external power supply to the molex adaptator on the chipset and with a clean driver install, everything is now perfectly fine. I can run two USB 3.0 cameras with no bandwidth issue at all !

I’d like to thank you again for your help :)

Enzo.