Syslog shows the TX2 can detect this device (SER errors accordingly if I hot swap it out), and lshw says it is unclaimed, so it seems like all it needs is a driver. I managed to find the source code for the driver in the latest torvalds/linux repo, provided by the manufacturers themselves for free:
Rather than buying the driver from the reseller, I was investigating if I could build it from the above source and dynamically load it to see if it works, although I’ve just begun researching how to do this. What is the recommended way to add this driver to my TX2 given the above source code? I have seen other inquiries about this bridge device in the past, but I am new to working with drivers:
I read the NVIDIA package documentation about BSP, but I couldnt find detailed guidance about how to add a driver in general. I also tried installing Linux extras, since this device driver is contained in that package, but it resulted in a system error and I had to reflash the TX2. Any guidance would be appreciated! Thank you very much.
FYI, most of the kernel build is no different on a Jetson than any other Ubuntu build. What differs most of the time is the install procedure, and the fact that it isn’t a PC architecture. Cross-compile is more common as well, but cross-compile doesn’t care about any particular driver, it is agnostic.
I don’t know anything about this particular driver. It could have architecture-specific code, I don’t know. Once you’re at the stage where you can create a duplicate kernel you’ll be ready to find out. I have no idea if this driver is designed for a 4.x series kernel, I have no idea if it has something depending on a desktop PC architecture, so on.
As for steps of building someone else’s driver you will find this is 100% the same for a Jetson or any other Linux-capable hardware, so any information you might find on building this driver out of tree should be valid (this is basically an edit to your procedure once you can build Image and modules). You can always ask when you get to that point if needed.
Thanks a lot for the fast response and information! I read your notes on building the kernel and theres a lot of good starting points for me to learn. I also went a bit through your post history to learn up some more and I have to ask, where did you get all of this experience? Any tips/advice?
FYI, I reached out to the bridge chip manufacturer and apparently they had some relationship with NVIDIA in the past and got this up and running on a TK1, actually a cluster of TK1s for some HPC/gaming application I believe. Im not sure of the differences between the TK1 and TX2, but it seems a bit more feasible now given this information. Im not sure why the vendor I purchased my board from wrote their own driver when the bridge chip manufacturer has one available, but maybe Ill find out after I try to get this one running. In any case, you may be interested in their doc for your own future reference/questions. Actually they even referenced some of your posts: RapidIO Support for NVIDIA Jetson TK1To enable RapidIO support to NVIDIA Jetso - Pastebin.com. If youre interested in the pdf I can share it somehow.
My first Linux system had kernel 1.0.9. It wasn’t long after I installed the 1.1.59 kernel to get experimental support of modules. Linux just turned into an obsession since it never crashed and Windows always did on the same hardware. It could also run the same software I used on the university’s IBM R/S 6000 if I figured out how to compile it (some software was for legitimate use, other software was for games). I already had hardware experience before that, the software just extended it.
Early on I discovered that I could look at “/usr/bin/”, “/usr/sbin/”, “/bin/”, “/sbin/”, so on, and then use the man page to see what those files were. One man page led to another (that pesky “SEE ALSO” meant I didn’t get much sleep)…
The major change in hardware between a TK1 and TX2 is that a TK1 is 32-bit (ARMv7/armhf), and that a TX2 is 64-bit (ARMv8-a/aarch64/arm64). The major difference in software is that the TK1 stopped development while the kernel was in the 3.x series, the TX2 is a 4.x series kernel. Beyond this and performance there isn’t much difference so far as software is concerned (both have a GPU even if at a different generation). In terms of versus a PC, other than architecture, it is how it boots which differs…much of what goes on after boot is identical between PC and Jetson.
Please use the below patch to enable RAPIDIO and driver for TSI-721 in kernel
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 20354a2946a1..91facd9c8eef 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -307,6 +307,16 @@ source "drivers/pci/Kconfig"
source "drivers/pci/pcie/Kconfig"
source "drivers/pci/hotplug/Kconfig"
+config RAPIDIO
+ tristate "RapidIO support"
+ depends on PCI
+ default n
+ help
+ If you say Y here, the kernel will include drivers and
+ infrastructure code to support RapidIO interconnect devices.
+
+source "drivers/rapidio/Kconfig"
+
endmenu
menu "Kernel Features"
diff --git a/arch/arm64/configs/tegra18_defconfig b/arch/arm64/configs/tegra18_defconfig
index a4974a172142..393f898f86e7 100644
--- a/arch/arm64/configs/tegra18_defconfig
+++ b/arch/arm64/configs/tegra18_defconfig
@@ -50,6 +50,8 @@ CONFIG_PCI=y
CONFIG_PCI_TEGRA=m
CONFIG_PCIEPORTBUS=y
# CONFIG_PCIEASPM is not set
+CONFIG_RAPIDIO=y
+CONFIG_RAPIDIO_TSI721=y
# CONFIG_ARM64_ERRATUM_843419 is not set
CONFIG_SCHED_MC=y
CONFIG_PREEMPT=y
to flash the TX2 with my patched kernel that I compiled natively. First, on the TX2, I scp’d my patch file over. Then, I downloaded the latest kernel using the getKernelSources.sh script from that guide. After the kernel was downloaded into /usr/src/, I went in to the tld of the kernel and patched it successfully (with some fuzz):
patch –p1 < /home/nvidia/jetsontx2_tsi721.patch
I executed the remaining scripts in that tutorial (setting kernel config, building the kernel, copying image over to /boot). However, something is not working. I suspect either I did the config incorrectly or missed a step, however I tried to keep the configs at default.
Doesnt return any devices at all:
sudo lspci -vvv
Errors pcilib: Cannot open on all buses:
sudo lspci -M
Tailing the syslogs doesnt show anything when I hotswap the tsi271 also, whereas before it did from AER driver IIRC.
My patch file looks like this:
---
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 20354a2946a1..91facd9c8eef 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -307,6 +307,16 @@ source "drivers/pci/Kconfig"
source "drivers/pci/pcie/Kconfig"
source "drivers/pci/hotplug/Kconfig"
+config RAPIDIO
+ tristate "RapidIO support"
+ depends on PCI
+ default n
+ help
+ If you say Y here, the kernel will include drivers and
+ infrastructure code to support RapidIO interconnect devices.
+
+source "drivers/rapidio/Kconfig"
+
endmenu
menu "Kernel Features"
---
diff --git a/arch/arm64/configs/tegra18_defconfig b/arch/arm64/configs/tegra18_defconfig
index a4974a172142..393f898f86e7 100644
--- a/arch/arm64/configs/tegra18_defconfig
+++ b/arch/arm64/configs/tegra18_defconfig
@@ -50,6 +50,8 @@ CONFIG_PCI=y
CONFIG_PCI_TEGRA=m
CONFIG_PCIEPORTBUS=y
# CONFIG_PCIEASPM is not set
+CONFIG_RAPIDIO=y
+CONFIG_RAPIDIO_TSI721=y
# CONFIG_ARM64_ERRATUM_843419 is not set
CONFIG_SCHED_MC=y
CONFIG_PREEMPT=y
--
If possible attach the dmesg of booting with your device in the slot and freshly booted after the patch.
Was your initial kernel .config from the running Jetson? If this was from a defconfig I’d be suspicious that it isn’t a good test (you’d have to save “/proc/config.gz” somewhere while using the original kernel).
Hi linuxdev thanks for the quick response. The config was copied from the running Jetson’s config just before my native compile. In the script I ran, config is copied into the kernel tld using:
Is there something special about the config if its copied while the Jetson is running? After this is copied, the script will
make xconfig
to customize the kernel options via xconfig GUI, where I tried keeping everything default since I dont know much about all of the various settings. FWIW, when I make the kernel itself, it does pick up the patch and did and prompt me to enable various SRIO settings, including the tsi271 driver, timeouts, etc.
The TX2 seems to be working properly, just I am not able to verify vs lspci that the PCIE-SRIO bridge I have works with the driver I patched L4T with. lspci doesnt show anything at all whereas before I could see all PCI devices on the TX2. I tried using different kernel configurations yesterday but no luck. Ill keep at it today and dig more. I’m wondering if something happened with the dtb file.
Apparently that guide is just to build the kernel natively and copy it over to /boot, and so I guess I have to run the flash script provided by NVIDIA to actually flash with the new kernel. Further, I think I might have gotten lspci to return output previously because I had attempted to install the linux kernel extras package because it included a driver for tsi721, but that errored out half way through installing and I had forgotten about that.
Device trees must be flashed on more recent releases, but kernels are just a copy of the Image file (I suggest don’t overwrite the original and add an entry to extlinux.conf…then use a serial console to pick the entry you want at boot time). Modules can just be put in the proper subdirectory of “/lib/modules/$(uname -r)/”.
Thanks for the guidance. I downloaded the kernel source, cross compilation tools, applied my patch file, customized the config fie via make nconfig to add the tsi271 as modules, built the kernel and modules the nice way. The modules for the tsi271 were in the temporary modules directory I created as you said, so I copied them over to my TX2 and rebooted and tried lspci -vvv while the bridge adapter was plugged in but saw nothing. Then I copied over the Image file that was built, since I wasnt sure if this was necessary also, and still no output from lspci -vvv. Then I went onto the TX2 and manually loaded in the modules via insmod with no error, however still the bridge device doesnt show up in lspci. However, lsmod does show the modules were loaded. Do I have to flash the DTB as well? Im not sure under which scenarios the DTB must be flashed, or how to make the new appropriate DTB. Thanks.
PS: there is a typo in your kernel building post. You forgot the dollar sign for the environment variable for the line “make -j4 O=TEGRA_KERNEL_OUT modules”
Hi linuxdev, thanks for the response. Yep there is just one directory
4.4.38-tegra
I copied the entire subdirectory for rapidio drivers over to there from my host after cross-compiling the kernel via your instructions and just now added them to /etc/modules so theyd load during boot.
What do you mean by flashing the image? If you updated the kernel, then only the Image file gets copied (no flash tool). If you added a module, then only the module gets copied (no flash tool). The flash tool itself would tend to be for bootloader or device tree content. I just want to verify what was actually copied.
Sorry I was a bit confused about what steps I needed to carry out, since theres a lot of information that doesnt apply to my use case. From my understanding, I applied the patch provided earlier which modifies the kernel configuration to enable the driver for the device I want. After this patch is applied, I compile the kernel. Now that I have the compiled kernel Image file, all thats needed is to copy this file over to the TX2 and the PCIE device will just work? What about the dtb?
What I did above was try to flash the TX2 after making my changes, running ./apply_binaries, and then ./flash script. But now, it seems that may not be necessary.