There is mellanox connectX-6 connected to Xavier AGX and I am trying to install MFT drivers and package on Jetpack 4.6 on Xavier AGX. When I try to do mst start, I am getting below error. The Installation works great for Jetpack 5.0.2.
How can I enable MFT kernel modules for Jetpack 4.6 ? I don’t see the same options available for Jetpack 5.0.2.
xavier@xavier:/usr/local/etc/mft-4.24.0-72-arm64-deb$ sudo mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI modulemodprobe: FATAL: Module mst_pci not found in directory /lib/modules/4.9.253
- Failure: 1
Loading MST PCI configuration modulemodprobe: FATAL: Module mst_pciconf not found in directory /lib/modules/4.9.253
- Failure: 1
Create devices
mst_pci driver not found
Unloading MST PCI module (unused)modprobe: FATAL: Module mst_pci not found.
- Failure: 1
Unloading MST PCI configuration module (unused)modprobe: FATAL: Module mst_pciconf not found.
- Failure: 1
This looks like you installed a different kernel. Perhaps that module is installed by default in the 5.x kernel, don’t know. I can tell you though that if the kernel wants modules here: /lib/modules/4.9.253/
…then it implies CONFIG_LOCALVERSION was not set during the kernel build. This is not necessarily wrong, but it is highly suspicious.
If you are building anything related to the kernel for some specific L4T release (see “head -n 1 /etc/nv_tegra_release”), then you’d get the source from the URL for that specific L4T release: https://developer.nvidia.com/linux-tegra
Then you’d set up a matching configuration to the running system. If this is the default kernel, then the build target “tegra_defconfig” will do this. You would still have to set up “CONFIG_LOCALVERSION”. Then you could enable that driver in the form of a module. If that driver is not available as a module, it tends to mean you need to rebuild the entire kernel and all modules with a new (changed from “-tegra” to something else, e.g., “-modified”) “CONFIG_LOCALVERSION”.
I am guessing (I don’t know for sure) that you can find the symbol needed via: zcat /proc/config.gz | grep -i 'mellanox'
What do you see when you run that command?
Incidentally, if the feature is “=y”, then you cannot load it as a module as it is already permanently installed. If software demanded to load it as a module when it is integrated (“=y”), then the software might fail because it is naive about the format of the driver. Also, sometimes one kernel symbol (the symbol is what build configuration refers to for a driver) can require another (an example is that certain network drivers require a checksum driver). If the Mellanox driver is present, then it is still possible there is some other driver (symbol) which Mellanox needs (a chain of dependencies).
If CONFIG_LOCALVERSION is now “-tegra”, and “uname -r” replies as “4.9.253-tegra”, then it means the module would be in a subdirectory of: /lib/modules/4.9.253-tegra/kernel
Thus you must install it there, and the rest of the kernel config should match the existing integrated kernel (usually the “/boot/Image”).
Note that in the kernel source, after building a module, that module is in some subdirectory. A contrived example: “drivers/net/something.ko”. The subdirectory of “/lib/modules/4.9.253-tegra/kernel” would then be: drivers/net/something.ko
Sometimes you’ll use a hyphen in the name instead of an underscore as well, so perhaps it wants mst-pci as the name if it is already there.
Hi @linuxdev , Thanks for commenting and for your suggestions. The kernel module mst-pci.ko is not available for 4.9.253-tegra as per menuconfig options. The main concern is how I can get these modules.
You need to find the kernel symbol which provides mst-pci.ko. That is if it is already in the kernel source. In that case you simply download the kernel, configure, build, and then copy the module file to the right place.
If this is part of the kernel, but not available in the format of a module, then it is the same, but install is more detailed and involves more.
If the driver is not part of the kernel source, but is available in a source format such that it can work with a 4.x series kernel (newer installs of Xavier can use a 5.x kernel, but L4T R32.x uses a 4.x kernel…see “head -n 1 /etc/nv_tegra_release” to get your L4T version), then it is similar to the above steps, but involves also adding that content to the kernel source first.
If the module is available only in binary format, then it must be compatible with your kernel, and must be compiled for 64-bit ARM (arm64/aarch64). Compatibility for slightly off versions sometimes exist, but there are certain “integrated” features (meaning no module format) which have to be available and compatible for it to work.
I don’t have the hardware, so it wouldn’t do any good for me to work on it anyway. However, if you can get more information, or get to the point where you find a useful URL to ask about, we can still help here. So mostly it is now a case of searching for a driver in the existing source, or what is available if it is not already in the 4.x series of kernel (if it is in the 5.x series kernel, then you could upgrade the entire operating system; you’re currently using L4T R32.x/Ubuntu 18.04, but it is possible to flash and move to L4T R35.x/Ubuntu 20.04).
Hi,
Since Jetpack 5 releases support Xavier series, could you use Jetpack 5(K5.10)? Certain configs may be missing in previous Jetapck 4(K4.9). The latest release is Jetpack 5.1.1.
Hi @DaneLLL , I have already mentioned that Jetpack 5.X works good with ConnectX-6, and we don’t see any issue there. The customer actually wants to downgrade to Jetpack 4.6 for their driver support, and I am really stuck in getting mst_pci kernel module.
Is it possible to add the required modules & configs through external source?
On the 5.x kernel which works, what driver symbol is providing that? You should be able to run something like this and get a possible list: zcat /proc/config.gz | grep -i mst
Any software which builds against any kernel source or configuration other than the running Jetson is bound to fail due to customizations needed in Jetson kernels. One would have to dig out the individual steps and perform those manually against the correct source/configuration of the running Jetson. I don’t know all of what that script does, but in the logs I see:
dpkg-buildpackage: info: source package kernel-mft
dpkg-buildpackage: info: source version 4.24.0-72
On the other hand, I wonder if kernel-mft used in that script is available only in a non-stock kernel. Basically, what is needed, is to find either the configuration of the existing kernel which is configured for the script, or else the patch from that kernel which could apply that to the Jetson’s kernel (either way you’d start with the existing kernel configuration, and then append that configuration on to it; this would enable you to build the kernel with the needed support). Sadly, I have no idea if that is reasonable or not (it depends on what kernel-mft is).