How to Install TSI721 PCIe to SRIO bridge driver from source on Jetson TX2, R28

I have the TSI721 (PCIe-2 to SRIO-2 bridge) made by IDT connected to my TX2:

https://www.idt.com/products/interface-connectivity/pci-express-solutions/pci-express-serial-rapidio-bridges/tsi721-rapidio-bridge

Syslog shows the TX2 can detect this device (SER errors accordingly if I hot swap it out), and lshw says it is unclaimed, so it seems like all it needs is a driver. I managed to find the source code for the driver in the latest torvalds/linux repo, provided by the manufacturers themselves for free:

Rather than buying the driver from the reseller, I was investigating if I could build it from the above source and dynamically load it to see if it works, although I’ve just begun researching how to do this. What is the recommended way to add this driver to my TX2 given the above source code? I have seen other inquiries about this bridge device in the past, but I am new to working with drivers:

https://devtalk.nvidia.com/default/topic/1001535/jetson-tx1/how-to-enable-pcie-/

I read the NVIDIA package documentation about BSP, but I couldnt find detailed guidance about how to add a driver in general. I also tried installing Linux extras, since this device driver is contained in that package, but it resulted in a system error and I had to reflash the TX2. Any guidance would be appreciated! Thank you very much.

FYI, most of the kernel build is no different on a Jetson than any other Ubuntu build. What differs most of the time is the install procedure, and the fact that it isn’t a PC architecture. Cross-compile is more common as well, but cross-compile doesn’t care about any particular driver, it is agnostic.

There is official documentation, but here are some notes on how to build a kernel which matches your running system…this is the first step since you would only be adding this driver to an otherwise duplicate config:
https://devtalk.nvidia.com/default/topic/1038175/jetson-tx2/tx2i-wifi-support/post/5274619/#5274619

I don’t know anything about this particular driver. It could have architecture-specific code, I don’t know. Once you’re at the stage where you can create a duplicate kernel you’ll be ready to find out. I have no idea if this driver is designed for a 4.x series kernel, I have no idea if it has something depending on a desktop PC architecture, so on.

As for steps of building someone else’s driver you will find this is 100% the same for a Jetson or any other Linux-capable hardware, so any information you might find on building this driver out of tree should be valid (this is basically an edit to your procedure once you can build Image and modules). You can always ask when you get to that point if needed.

Btw, if you don’t have a serial console cable already, get one now. You’ll be saving yourself a lot of time since you are testing kernel configs which might fail. See:
http://www.jetsonhacks.com/2017/03/24/serial-console-nvidia-jetson-tx2/

Thanks a lot for the fast response and information! I read your notes on building the kernel and theres a lot of good starting points for me to learn. I also went a bit through your post history to learn up some more and I have to ask, where did you get all of this experience? Any tips/advice?

FYI, I reached out to the bridge chip manufacturer and apparently they had some relationship with NVIDIA in the past and got this up and running on a TK1, actually a cluster of TK1s for some HPC/gaming application I believe. Im not sure of the differences between the TK1 and TX2, but it seems a bit more feasible now given this information. Im not sure why the vendor I purchased my board from wrote their own driver when the bridge chip manufacturer has one available, but maybe Ill find out after I try to get this one running. In any case, you may be interested in their doc for your own future reference/questions. Actually they even referenced some of your posts: https://pastebin.com/MzKDHZqz. If youre interested in the pdf I can share it somehow.

My first Linux system had kernel 1.0.9. It wasn’t long after I installed the 1.1.59 kernel to get experimental support of modules. Linux just turned into an obsession since it never crashed and Windows always did on the same hardware. It could also run the same software I used on the university’s IBM R/S 6000 if I figured out how to compile it (some software was for legitimate use, other software was for games). I already had hardware experience before that, the software just extended it.

Early on I discovered that I could look at “/usr/bin/”, “/usr/sbin/”, “/bin/”, “/sbin/”, so on, and then use the man page to see what those files were. One man page led to another (that pesky “SEE ALSO” meant I didn’t get much sleep)…

The major change in hardware between a TK1 and TX2 is that a TK1 is 32-bit (ARMv7/armhf), and that a TX2 is 64-bit (ARMv8-a/aarch64/arm64). The major difference in software is that the TK1 stopped development while the kernel was in the 3.x series, the TX2 is a 4.x series kernel. Beyond this and performance there isn’t much difference so far as software is concerned (both have a GPU even if at a different generation). In terms of versus a PC, other than architecture, it is how it boots which differs…much of what goes on after boot is identical between PC and Jetson.

Please use the below patch to enable RAPIDIO and driver for TSI-721 in kernel

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 20354a2946a1..91facd9c8eef 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -307,6 +307,16 @@ source "drivers/pci/Kconfig"
 source "drivers/pci/pcie/Kconfig"
 source "drivers/pci/hotplug/Kconfig"

+config RAPIDIO
+        tristate "RapidIO support"
+        depends on PCI
+        default n
+        help
+          If you say Y here, the kernel will include drivers and
+          infrastructure code to support RapidIO interconnect devices.
+
+source "drivers/rapidio/Kconfig"
+
 endmenu

 menu "Kernel Features"
diff --git a/arch/arm64/configs/tegra18_defconfig b/arch/arm64/configs/tegra18_defconfig
index a4974a172142..393f898f86e7 100644
--- a/arch/arm64/configs/tegra18_defconfig
+++ b/arch/arm64/configs/tegra18_defconfig
@@ -50,6 +50,8 @@ CONFIG_PCI=y
 CONFIG_PCI_TEGRA=m
 CONFIG_PCIEPORTBUS=y
 # CONFIG_PCIEASPM is not set
+CONFIG_RAPIDIO=y
+CONFIG_RAPIDIO_TSI721=y
 # CONFIG_ARM64_ERRATUM_843419 is not set
 CONFIG_SCHED_MC=y
 CONFIG_PREEMPT=y

@vidyas thanks a lot! Saved me a lot of work porting over the TK1 patches.

Hi vidyas, linuxdev,

After receiving the patch above, I followed the following guide:

https://www.jetsonhacks.com/2017/03/25/build-kernel-and-modules-nvidia-jetson-tx2/

to flash the TX2 with my patched kernel that I compiled natively. First, on the TX2, I scp’d my patch file over. Then, I downloaded the latest kernel using the getKernelSources.sh script from that guide. After the kernel was downloaded into /usr/src/, I went in to the tld of the kernel and patched it successfully (with some fuzz):

patch –p1 < /home/nvidia/jetsontx2_tsi721.patch

I executed the remaining scripts in that tutorial (setting kernel config, building the kernel, copying image over to /boot). However, something is not working. I suspect either I did the config incorrectly or missed a step, however I tried to keep the configs at default.

Doesnt return any devices at all:

sudo lspci -vvv

Errors pcilib: Cannot open on all buses:

sudo lspci -M

Tailing the syslogs doesnt show anything when I hotswap the tsi271 also, whereas before it did from AER driver IIRC.

My patch file looks like this:

---
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 20354a2946a1..91facd9c8eef 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -307,6 +307,16 @@ source "drivers/pci/Kconfig"

 source "drivers/pci/pcie/Kconfig"
 source "drivers/pci/hotplug/Kconfig"

+config RAPIDIO
+        tristate "RapidIO support"
+        depends on PCI
+        default n
+        help
+          If you say Y here, the kernel will include drivers and
+          infrastructure code to support RapidIO interconnect devices.
+
+source "drivers/rapidio/Kconfig"
+
 endmenu

 menu "Kernel Features"
---
diff --git a/arch/arm64/configs/tegra18_defconfig b/arch/arm64/configs/tegra18_defconfig
index a4974a172142..393f898f86e7 100644
--- a/arch/arm64/configs/tegra18_defconfig
+++ b/arch/arm64/configs/tegra18_defconfig
@@ -50,6 +50,8 @@ CONFIG_PCI=y
 CONFIG_PCI_TEGRA=m
 CONFIG_PCIEPORTBUS=y
 # CONFIG_PCIEASPM is not set
+CONFIG_RAPIDIO=y
+CONFIG_RAPIDIO_TSI721=y
 # CONFIG_ARM64_ERRATUM_843419 is not set
 CONFIG_SCHED_MC=y
 CONFIG_PREEMPT=y

--

If possible attach the dmesg of booting with your device in the slot and freshly booted after the patch.

Was your initial kernel .config from the running Jetson? If this was from a defconfig I’d be suspicious that it isn’t a good test (you’d have to save “/proc/config.gz” somewhere while using the original kernel).

Hi linuxdev thanks for the quick response. The config was copied from the running Jetson’s config just before my native compile. In the script I ran, config is copied into the kernel tld using:

zcat /proc/config.gz > .config

See: https://github.com/jetsonhacks/buildJetsonTX2Kernel/blob/master/scripts/getKernelSourcesNoGUI.sh

Is there something special about the config if its copied while the Jetson is running? After this is copied, the script will

make xconfig

to customize the kernel options via xconfig GUI, where I tried keeping everything default since I dont know much about all of the various settings. FWIW, when I make the kernel itself, it does pick up the patch and did and prompt me to enable various SRIO settings, including the tsi271 driver, timeouts, etc.

Anywho, here is the dmesg output: https://pastebin.com/LsFCx4DN

Ill have a look at dmesg and see what I can find. Thanks!

Heres the PCI-specific messages from dmesg:

[    0.142220] node /plugin-manager/fragment-500-pcie-config match with board >=3310-1000-500
[    0.142937] node /plugin-manager/fragment-500-e3325-pcie match with board >=3310-1000-500
[    0.259655] iommu: Adding device 10003000.pcie-controller to group 50
[    0.433687] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    7.600405] tegra-pcie 10003000.pcie-controller: 4x1, 1x1 configuration
[    7.610805] tegra-pcie 10003000.pcie-controller: PCIE: Enable power rails
[    7.620751] tegra-pcie 10003000.pcie-controller: probing port 0, using 4 lanes
[    7.622466] tegra-pcie 10003000.pcie-controller: probing port 2, using 1 lanes
[    8.044176] tegra-pcie 10003000.pcie-controller: link 0 down, retrying
[    8.458158] tegra-pcie 10003000.pcie-controller: link 0 down, retrying
[    8.876168] tegra-pcie 10003000.pcie-controller: link 0 down, retrying
[    8.878219] tegra-pcie 10003000.pcie-controller: link 0 down, ignoring
[    9.364162] tegra-pcie 10003000.pcie-controller: link 2 down, retrying
[    9.767629] tegra-pcie 10003000.pcie-controller: link 2 down, retrying
[   10.180196] tegra-pcie 10003000.pcie-controller: link 2 down, retrying
[   10.182200] tegra-pcie 10003000.pcie-controller: link 2 down, ignoring
[   10.182216] tegra-pcie 10003000.pcie-controller: PCIE: no end points detected
[   10.182676] tegra-pcie 10003000.pcie-controller: PCIE: Disable power rails

I had enabled hot-swapping as you can see.

Is this working correctly for you?

The TX2 seems to be working properly, just I am not able to verify vs lspci that the PCIE-SRIO bridge I have works with the driver I patched L4T with. lspci doesnt show anything at all whereas before I could see all PCI devices on the TX2. I tried using different kernel configurations yesterday but no luck. Ill keep at it today and dig more. I’m wondering if something happened with the dtb file.

Apparently that guide is just to build the kernel natively and copy it over to /boot, and so I guess I have to run the flash script provided by NVIDIA to actually flash with the new kernel. Further, I think I might have gotten lspci to return output previously because I had attempted to install the linux kernel extras package because it included a driver for tsi721, but that errored out half way through installing and I had forgotten about that.

Device trees must be flashed on more recent releases, but kernels are just a copy of the Image file (I suggest don’t overwrite the original and add an entry to extlinux.conf…then use a serial console to pick the entry you want at boot time). Modules can just be put in the proper subdirectory of “/lib/modules/$(uname -r)/”.

Here are some kernel build hints:
https://devtalk.nvidia.com/default/topic/1038175/jetson-tx2/tx2i-wifi-support/post/5274619/#5274619

Here are some device tree compiler (dtc) hints:
https://devtalk.nvidia.com/default/topic/1037131/jetson-tx1/does-anybody-know-how-to-enable-and-program-an-spi-protocol-on-the-tx1-/post/5269528/#5269528

For flashing DTB on R28.1 (and R28.2? Can’t remember) there was a patch to keep it from also flashing rootfs (the script is human readable so it should be obvious after examining it if you need to patch):
https://devtalk.nvidia.com/default/topic/1036286/jetson-tx1/flashing-just-dtb-on-28-2-and-tx1/post/5264465/#5264465

An example of DTB flash is to extract the existing DTB (see the dtc hints), and then reflash it something like this:

sudo ./flash.sh -r -d ./your_custom_tree.dtb jetson-tx2 mmcblk0p1 2>&1 | tee log_dtb_flash.txt

Hi linuxdev,

Thanks for the guidance. I downloaded the kernel source, cross compilation tools, applied my patch file, customized the config fie via make nconfig to add the tsi271 as modules, built the kernel and modules the nice way. The modules for the tsi271 were in the temporary modules directory I created as you said, so I copied them over to my TX2 and rebooted and tried lspci -vvv while the bridge adapter was plugged in but saw nothing. Then I copied over the Image file that was built, since I wasnt sure if this was necessary also, and still no output from lspci -vvv. Then I went onto the TX2 and manually loaded in the modules via insmod with no error, however still the bridge device doesnt show up in lspci. However, lsmod does show the modules were loaded. Do I have to flash the DTB as well? Im not sure under which scenarios the DTB must be flashed, or how to make the new appropriate DTB. Thanks.

PS: there is a typo in your kernel building post. You forgot the dollar sign for the environment variable for the line “make -j4 O=TEGRA_KERNEL_OUT modules”

If you copied the Image file verify that your “uname -r” has not changed. Then verify that this directory exists with your entire kernel module tree:

/lib/modules/$(uname -r)/

FYI, thanks for the typo note…I fixed this to now have the ‘$’.

Hi linuxdev, thanks for the response. Yep there is just one directory

4.4.38-tegra

I copied the entire subdirectory for rapidio drivers over to there from my host after cross-compiling the kernel via your instructions and just now added them to /etc/modules so theyd load during boot.

After flashing the new Image, it says *** Update [6] is not supported. ***:

###############################################################################
# L4T BSP Information:
# R28 (release), REVISION: 2.1, GCID: 11272647, BOARD: t186ref, EABI: aarch64, 
# DATE: Thu May 17 07:29:06 UTC 2018
###############################################################################
# Target Board Information:
# Name: jetson-tx2, Board Family: t186ref, SoC: Tegra 186, 
# OpMode: production, Boot Authentication: , 
###############################################################################
./tegraflash.py --chip 0x18 --applet "/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/mb1_recovery_prod.bin" --cmd "dump eeprom boardinfo cvm.bin" --skipuid 
Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands
 
[   0.0027 ] Generating RCM messages
[   0.0047 ] tegrarcm_v2 --listrcm rcm_list.xml --chip 0x18 --download rcm /home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/mb1_recovery_prod.bin 0 0
[   0.0064 ] RCM 0 is saved as rcm_0.rcm
[   0.0078 ] RCM 1 is saved as rcm_1.rcm
[   0.0079 ] List of rcm files are saved in rcm_list.xml
[   0.0079 ] 
[   0.0079 ] Signing RCM messages
[   0.0101 ] tegrasign_v2 --key None --list rcm_list.xml --pubkeyhash pub_key.key
[   0.0119 ] Assuming zero filled SBK key
[   0.0198 ] 
[   0.0199 ] Copying signature to RCM mesages
[   0.0218 ] tegrarcm_v2 --chip 0x18 --updatesig rcm_list_signed.xml
[   0.0241 ] 
[   0.0242 ] Boot Rom communication
[   0.0258 ] tegrarcm_v2 --chip 0x18 --rcm rcm_list_signed.xml --skipuid
[   0.0273 ] RCM version 0X180001
[   0.1264 ] Boot Rom communication completed
[   1.1326 ] 
[   1.1346 ] tegrarcm_v2 --isapplet
[   1.1365 ] Applet version 01.00.0000
[   1.2359 ] 
[   1.2379 ] Retrieving EEPROM data
[   1.2382 ] tegrarcm_v2 --oem platformdetails eeprom cvm /home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/cvm.bin
[   1.2400 ] Applet version 01.00.0000
[   1.3999 ] Saved platform info in /home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/cvm.bin
[   1.4772 ] 
Board ID(3310) version(B02) 
copying bctfile(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/t186ref/BCT/P3310_A00_8GB_Samsung_8GB_lpddr4_204Mhz_A02_l4t.cfg)... done.
copying misc_config(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/t186ref/BCT/tegra186-mb1-bct-misc-si-l4t.cfg)... done.
copying pinmux_config(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/t186ref/BCT/tegra186-mb1-bct-pinmux-quill-p3310-1000-c03.cfg)... done.
copying pmic_config(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/t186ref/BCT/tegra186-mb1-bct-pmic-quill-p3310-1000-c04.cfg)... done.
copying pmc_config(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/t186ref/BCT/tegra186-mb1-bct-pad-quill-p3310-1000-c03.cfg)... done.
copying prod_config(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/t186ref/BCT/tegra186-mb1-bct-prod-quill-p3310-1000-c03.cfg)... done.
copying scr_config(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/t186ref/BCT/minimal_scr.cfg)... done.
copying scr_cold_boot_config(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/t186ref/BCT/mobile_scr.cfg)... done.
copying bootrom_config(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/t186ref/BCT/tegra186-mb1-bct-bootrom-quill-p3310-1000-c03.cfg)... done.
copying dev_params(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/t186ref/BCT/emmc.cfg)... done.
Existing bootloader(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/nvtboot_cpu.bin) reused.
	populating kernel to rootfs... done.
	populating initrd to rootfs... done.
	populating extlinux.conf.emmc to rootfs... done.
	populating /home/chris/tx2/64_TX2/Linux_for_Tegra/kernel/dtb/tegra186-quill-p3310-1000-c03-00-base.dtb to rootfs... done.
done.
Making Boot image... done.
Existing sosfile(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/mb1_recovery_prod.bin) reused.
copying tegraboot(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/t186ref/nvtboot.bin)... done.
Existing mb2blfile(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/nvtboot_recovery.bin) reused.
Existing mtspreboot(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/preboot_d15_prod_cr.bin) reused.
Existing mts(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/mce_mts_d15_prod_cr.bin) reused.
Existing mb1file(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/mb1_prod.bin) reused.
Existing bpffile(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/bpmp.bin) reused.
copying bpfdtbfile(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/t186ref/tegra186-a02-bpmp-quill-p3310-1000-c04-00-te770d-ucm2.dtb)... done.
Existing scefile(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/camera-rtcpu-sce.bin) reused.
Existing spefile(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/spe.bin) reused.
copying wb0boot(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/t186ref/warmboot.bin)... done.
Existing tosfile(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/tos.img) reused.
Existing eksfile(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/eks.img) reused.
copying dtbfile(/home/chris/tx2/64_TX2/Linux_for_Tegra/kernel/dtb/tegra186-quill-p3310-1000-c03-00-base.dtb)... done.
Existing tbcfile(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/cboot.bin) reused.
copying tbcdtbfile(/home/chris/tx2/64_TX2/Linux_for_Tegra/kernel/dtb/tegra186-quill-p3310-1000-c03-00-base.dtb)... done.
copying cfgfile(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/t186ref/cfg/flash_l4t_t186.xml) to flash.xml... done.
Existing flasher(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/nvtboot_recovery_cpu.bin) reused.
Existing flashapp(/home/chris/tx2/64_TX2/Linux_for_Tegra/bootloader/tegraflash.py) reused.
*** Update [6] is not supported. ***

What do you mean by flashing the image? If you updated the kernel, then only the Image file gets copied (no flash tool). If you added a module, then only the module gets copied (no flash tool). The flash tool itself would tend to be for bootloader or device tree content. I just want to verify what was actually copied.

Sorry I was a bit confused about what steps I needed to carry out, since theres a lot of information that doesnt apply to my use case. From my understanding, I applied the patch provided earlier which modifies the kernel configuration to enable the driver for the device I want. After this patch is applied, I compile the kernel. Now that I have the compiled kernel Image file, all thats needed is to copy this file over to the TX2 and the PCIE device will just work? What about the dtb?

What I did above was try to flash the TX2 after making my changes, running ./apply_binaries, and then ./flash script. But now, it seems that may not be necessary.