How to Install TSI721 PCIe to SRIO bridge driver from source on Jetson TX2, R28

For patch do you mean “enable RAPIDIO and driver for TSI-721 in kernel”? If so, then no flash is required. If set up correctly you just compile the kernel and copy the Image file and/or module file.

Making the changes and then running the steps you listed would have reflashed the Jetson back to its unmodified state (any previous changes would have been wiped out).

FYI, “/boot/” has the file “Image”. This is named in “/boot/extlinux/extlinux.conf”. I’d suggest adding an alternate name, e.g., “Image-custom”. Then edit extlinux.conf to have a second entry where “Image” is replaced by “Image-custom”. You can select the second entry at boot time via serial console, or else change the “default” at the top to that entry.

Any dtb change does need to use the flash tool, but not with the flash command you ran…that command replaces everything, and not just dtb.

See:
https://devtalk.nvidia.com/default/topic/1036286/jetson-tx1/flashing-just-dtb-on-28-2-and-tx1/post/5264465/#5264465
Then (if the dtb file has been put in the correct place):

sudo ./flash.sh -r -k kernel-dtb jetson-tx2 mmcblk0p1

Or just naming the dtb (sample dtb name is the one from the driver package…you’d basically be using one which is the same other than your edits):

sudo ./flash.sh -r -d ./kernel/dtb/tegra186-quill-p3310-1000-c03-00-base.dtb jetson-tx2 mmcblk0p1

Hi linuxdev,

Yes that is correct. I had compiled the kernel and copied over the image previously, but lspci was not showing the device when I plugged it in, so I started thinking I was missing some steps and hence started trying flashing. Actually, I flashed in the way you described, however I was not sure if flashing was necessary because I dont know if applying the simple kernel patch to enable tsi27 and compiling produces an updated device tree blob that needs to be flashed onto the device. Im still learning this stuff so thats a bit beyond my current knowledge.

EDIT: After just compiling and copying a new Image over again, I confirmed I could see that the new /proc/config.gz does have the new settings to enable the device, but still lspci isnt showing anything and also modules.builtin does not show that the driver was added to the kernel. In the build directory under lib/modules/4.4.38-tegra, I see at least directories were created for rapidio (my PCI device), but just contain kconfig and make files. In the actual kernel/drivers directory, there is no directory for rapidio, but possibly because I didnt configure the kernel to make the driver a loadable module. Im not sure. If there is another way to quickly confirm that this device is detected by the OS and will likely work once I start using it, Im open to that. Thats all I really need to confirm before I start development. Thanks!

Sorry, this is long. The most important questions are in bold/italic.

Does the “/proc/config.gz” show the feature is added as integrated (“=y”), or does it show as a module (“=m”)? If “=y”, then the Image file contains everything and you can be sure the feature is there. If “=m”, then you need to be sure that all modules can be found somewhere in “/lib/modules/$(uname -r)/”. Example:

find /lib/modules/$(uname -r) -name 'whatever_it_is.*'

Once that is verified, any device tree change becomes the next topic and is independent of kernel build and kernel install. Normal kernel feature changes do not touch the device tree, this would be a separate config and build step. It happens that device tree and device tree compiler ships with the Linux kernel source, and this is useful due to the relationship between device tree and drivers, but the actual build of kernel and device tree is more or less independent and working on one does not usually change the other.

Think of device tree as a way of abstracting the registers needed to route or glue the hardware to a generic driver interface…unless the driver interface changes (and many drivers can use the same interface) or the method of wiring/routing the hardware changes you won’t need to worry about device tree. A driver’s API basically is told how to find hardware by a device tree, and the firmware loaded to the hardware matches the driver’s API (not all hardware has firmware…if no firmware, then there is a single hardware concept which can’t be changed…regulated hardware, e.g., WiFi, tends to have firmware, and other hardware tends to not have firmware). Finding the hardware (device tree), and having the hardware match the API (firmware matching driver version) is the complete circle of requirements.

You can expect custom carrier boards to need device tree changes…such boards alter how the wiring is routed even when two boards sitting next to each other implement the same features using the same drivers.

Custom sensors most likely also need a device tree change. Sensors have registers and configuration to change which is only known in an abstract way to drivers…the device tree translates the specific register changes needed to follow the abstract concept the driver knows about and the way the wiring is routed.

On a lower level, a device tree is just an abstracted list of name to value mappings a driver uses in order to initialize and find hardware. A name and value are mapped together like an associative array which causes hardware registers to change. If those changes occur as the driver expects, then the driver can find and use the hardware.

Note that lspci shows if PCIe sees your card. Once PCIe sees the card and tells the driver what is there its job is done. If no driver takes ownership, then PCIe is functioning 100% correct and the driver can be blamed. Having lspci show the card means PCIe worked.

If a driver is loaded as a module (“=m”), then “lsmod” would show the driver. If the driver is integrated (“=y”), then the driver is always available. How did you build the driver (module versus integrated)? As a module? If external to the kernel source tree this is what I’d expect.

Even if a module is in place it may be the system has to either reboot or be told to look for the module. This is what “sudo depmod -a” is for (and run right after installing the module to “/lib/modules/$(uname -r)/”).

Regardless of depmod it should be possible to force a module to load as a test with the “sudo insmod ”. Or a more gentle “sudo modprobe ”. “insmod” does not search for dependencies, whereas “modprobe” does. “insmod” will fail if a dependency is not met, but “modprobe” will attempt to work with dependencies. Individual modules without dependencies won’t care.

Look at your driver source. If you see “.c” or “.h” files, then this is purely kernel source and no device tree involved (config files, e.g., Kconfig and Makefile, are always there and don’t say much). If you see “.dts” or “.dtsi” files, then this is device tree code. What kind of source code is the driver you are working with? Answering will require knowing if this is purely kernel code, purely device tree, or some mix. In the case of kernel code, answering will also require knowing if it is module or integrated.

Thanks for the detailed explanation and patience of steel.
To answer your questions first:

  1. I did both (Y vs M in config), + copying compiled modules from kernel/drivers/rapidio to my TX2 and using insmod to load them (successfully), and also making them load at boot time.
  2. I didnt touch nor manually add any source code. I thought the driver simply needed enabling, and is already part of the kernel: https://github.com/torvalds/linux/blob/master/drivers/rapidio/devices/tsi721.c. At least, thats the impression I got from earlier when someone from NVIDIA commented in our thread saying I just need to apply their patch to enable the driver.

Detailed explanation:
So I tried both methods actually, with the guidance of your post you linked me to re:wifi and also followed another tutorial in case I was missing something Compiling Tegra Source Code | Jetson Tegra X1 and X2 | RidgeRun. To be honest, I cant remember how many times and methods I used to compile this kernel, but heres the time when I followed your guidance:

$ ./source_sync.sh #made sure to checkout tegra-l4t-r28.2.1 branch for all repos
$ export CROSS_COMPILE=/opt/l4t-gcc-toolchain-64-bit-28-2.1/install/bin/aarch64-unknown-linux-gnu-
$ export ARCH=arm64
$ export SRC=~/NVIDIA/64_TX2/Linux_for_Tegra/sources/kernel/kernel-4.4
$ export TEGRA_KERNEL_OUT=$HOME/kernel-out/kernel
$ cd $SRC
$ patch -p1 < /home/me/Documents/jetsontx2_tsi721.patch
# https://github.com/chrislgarry/Jetson_TX2-SRIO_Patch/blob/master/jetson_tx2_srio_tsi271.patch
$ make ARCH=arm64 O=$TEGRA_KERNEL_OUT tegra18_defconfig
# verified the generated config includes rapidio support, trying both kernel and module methods
$ make ARCH=arm64 O=$TEGRA_KERNEL_OUT -j4
$ scp arch/arm64/boot/Image $TX2
# Copied the image over to the TX2 nvidia and rebooted

For the module method, additionally I compiled the modules and copied them over to /lib/modules/ and used insmod to load them and hot plugged the device. Additionally, I made the modules load at boot and left the device in the PCI slot to see if it made a difference (the time when I showed you what dmesg was saying upon boot for PCI). Also in the previous instance I mentioned flashing, where I flashed the image and also copied the generated dts contents over to dtb directory in Linux_for_Tegra and ran the flash script with the appropriate arguments to flash the dtb, I believe

$ sudo ./flash.sh -d kernel/dtb/tegra186-quill-p3310-1000-c03-00-base.dtb -K kernel/Image jetson-tx2 mmcblk0p1
$ sudo ./flash.sh -k kernel-dtb jetson-tx2 mmcblk1p11

Assuming “$TX” is “root@:/boot” the Image would go in the correct place. Otherwise you might have an Image file somewhere else and still be using the original.

FYI, if you have an integrated feature (“=y”), then you won’t be able to insmod (nor lsmod). Not all drivers can be made as a module, but if “insmod” works on the driver, then it should be ok.

Of the two lines for dtb flash both could be valid, but only one would be used. I’d recommend the first line and using a dtb file you know is guaranteed to have your edited device tree. On the other hand, I don’t know if this driver requires a new device tree or not. If it does, and if PCIe sees the device but fails to load the driver, then this might be part of the issue. I don’t know what this particular driver requires, and I don’t know what the patch does.

At this point, what does your “sudo lspci” show? Note that if something shows up in lspci it will show a slot number on the left. An example is “01:00.0”. Using that as an example, here is how you would get the verbose lspci:

sudo lspci -s 01:00.0 -vvv

If lspci shows up, post the content of the verbose lspci. If nothing shows up for that device, then it changes how to debug it.

I verified that $TX2 was the correct directory. Since I attempted both ways, as a loadable module and as integrated, I verified that the module could be loaded using insmod. I believe I also used just both of those lines to flash, but at different times/attempts. I also used dtc to view the dtb file contents from a fresh kernel and compare with one after my patch compilation, and I didnt see anything special added for rapidio nor the tsi721 using grep. At this point, nothing shows up for sudo lspci -vvv.

Since this device was previously tested with the TX1, I can tell you that there were about 25 different patches needed to get it working which the manufacturer provided me with, as opposed to the one patch that the nvidia representative posted in our thread before. Initially, I was porting all 25 of those patches over to the TX2, and some of what I saw had already made it into the official kernel release. I stopped porting after the rep told me I just needed that one simple patch. Perhaps I should resume porting those over.

A kernel build and install doesn’t change the device tree. This would be an entirely different step. Device tree does ship with kernel source, but the kernel itself and the modules never change the tree.

On the other hand, if any of your devices require device tree or firmware, then this could cause a failure to detect and also cause the driver to be unable to work with the device.

I’m a bit tired and probably leaving out some details, but is this using the regular PCIe slot and not the m.2?

Does the manufacturer say anything about firmware or device tree? If not, then it may just be the kernel module needing attention.

Hi linuxdev,

Thats correct, I used the normal slot. The manufacturer doesnt mention anything about device tree or firmware. I did a sanity check today by plugging in the device to my host PC (Intel, Ubuntu 16.04), and running lspci after booting up, the device was listed.

For this particular carrier, do you have other PCIe cards you can plug in just to see if “lspci” works to see the card? Even without a driver lspci should at least see the card exists. It would be nice to see if perhaps other cards also fail to show up, or if it is unique to that card.

I have two of the same card. Neither were detected. Unfortunately I dont have any other devices from that carrier. Im trying to get my hands on something else, like a graphics card, just to see if anything at all is detected. I also have a second tx2. I can test that one out as well with this card.

Unfortunately, two cards of the same design won’t answer the question of whether the signal is related to design (it isn’t the card functionality itself we are testing so much as it is how it interacts with the Jetson).

I wouldn’t use a graphics card since they use a lot of power, but you might get something like a cheap Realtek network card (some are in the $10-20 range). Or a PCIe USB2 HUB would possibly even be useful for other testing (I wouldn’t go USB3 because of power requirements…USB2 simplifies what is being tested versus USB3).

FYI, I have a PCIe Realtek network card which works on both the TX1 and TX2. These are nice because the chipset is supported without adding any kernel modules.

I was able to get my other TX2 to detect the device after a clean flash. It seems the PCIe slot on my original board was damaged somehow, possibly from attempting a hot swap, which I see now is advised against in the documentation.

Just something interesting I noticed:

If I apply the patch and perform the cross complication on my host machine, and copy the Image file over to the working TX2, it then does not show anything for sudo lspci -vvv, just as the old TX2. Reflashing to the original NVIDIA provided sources resolves this issue. After discovering that, I thought maybe thats what was going on with my original TX2 so I did a quick flash, but lspci still didnt show the device. Once I get another PCI device to test with I can confirm for sure whether or not that TX2’s PCI bus is dead.

Some alternate slot types, other than the full-sized PCIe slot, might offer hot swap. PCMCIA is an example of a variation which has the hardware hot swap capability (there would be a hot plug software layer on top of that…technically hardware swap without software is “warm swap”). Never ever add or remove a regular PCIe slot card with power attached (even if the unit is off there can be standby power delivery to the slot which is not intended for hot swap).