Installing OFED with NVMe-oF on a System with OS Installed on NVMe

[user157267]

Hi OFED Driver Team,

I am installing OFED on a server with a ConnectX-7 adapter and aiming to enable NVMe-oF. However, I am encountering an issue because the server’s operating system is also installed on an NVMe SSD. After successfully installing OFED, I encountered the following message:

Note: In order to load the new nvme-rdma and nvmet-rdma modules, the nvme module must be reloaded.

$ sudo modprobe -r nvme
modprobe: FATAL: Module nvme is in use.

The problem appears to be that modprobe -r nvme fails due to the NVMe SSD being used as the system drive.

My question is: How can I safely reload the nvme module and load nvme-rdma and nvmet-rdma when the OS is also installed on an NVMe SSD?

I would appreciate any guidance on how to properly reload these modules without disrupting the OS on the NVMe SSD. Thank you for your assistance!

---

Environment Details:

$ uname -r
6.8.0-41-generic

$ lsb_release -a
No LSB modules are available.
Distributor ID:    Ubuntu
Description:    Ubuntu 24.04 LTS
Release:    24.04
Codename:    noble

$ ibstat
CA 'mlx5_0'
    CA type: MT4129
    Number of ports: 1
    Firmware version: 28.41.1000
    Hardware version: 0
    Node GUID: ----
    System image GUID: ----
    Port 1:
        State: Down
        Physical state: Disabled
        Rate: 400
        Base lid: 0
        LMC: 0
        SM lid: 0
        Capability mask: 0x00010000
        Port GUID: ----
        Link layer: Ethernet

OFED Installation Command Used:

wget "https://content.mellanox.com/ofed/MLNX_OFED-24.04-0.6.6.0/MLNX_OFED_LINUX-24.04-0.6.6.0-ubuntu24.04-x86_64.iso"
sudo mlnxofedinstall -vvv --with-nvmf --add-kernel-support --basic --force --force-fw-update

End of mlnxofedinstall Log:

Device (c1:00.0):
    c1:00.0 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
    Link Width: x16
    PCI Link Speed: Unknown

Installation passed successfully
To load the new driver, run:
/etc/init.d/openibd restart
Note: In order to load the new nvme-rdma and nvmet-rdma modules, the nvme module must be reloaded.

dmesg Errors Encountered when Loading Modules (nvme-fabrics, nvme-rdma, nvmet, nvmet-rdma):

[ 1711.360216] nvme_fabrics: disagrees about version of symbol __nvme_submit_sync_cmd
[ 1711.360222] nvme_fabrics: Unknown symbol __nvme_submit_sync_cmd (err -22)

[ 1724.339810] nvmet: disagrees about version of symbol nvme_command_effects
[ 1724.339816] nvmet: Unknown symbol nvme_command_effects (err -22)
[ 1724.339857] nvmet: disagrees about version of symbol nvme_passthru_end
[ 1724.339859] nvmet: Unknown symbol nvme_passthru_end (err -22)
[ 1724.340038] nvmet: disagrees about version of symbol nvme_find_get_ns
[ 1724.340040] nvmet: Unknown symbol nvme_find_get_ns (err -22)
[ 1724.340063] nvmet: Unknown symbol nvme_find_noiob_from_bdev (err -2)
[ 1724.340141] nvmet: disagrees about version of symbol nvme_passthru_start
[ 1724.340144] nvmet: Unknown symbol nvme_passthru_start (err -22)
[ 1724.340219] nvmet: Unknown symbol nvme_find_pdev_from_bdev (err -2)
[ 1724.340254] nvmet: disagrees about version of symbol nvme_ctrl_from_file
[ 1724.340257] nvmet: Unknown symbol nvme_ctrl_from_file (err -22)
[ 1724.340299] nvmet: disagrees about version of symbol nvme_put_ns
[ 1724.340301] nvmet: Unknown symbol nvme_put_ns (err -22)
[ 1724.340345] nvmet: disagrees about version of symbol nvme_get_features
[ 1724.340347] nvmet: Unknown symbol nvme_get_features (err -22)

[user157267]

I also checked similar thread posts, but I think they can do sudo modprove -r nvme:

Issue observed while inserting 'nvme_rdma' module after OFED driver installation.

Hello Ankit, Many thanks for posting your issue on the Mellanox Community. Based on the information you provided, we were not able to reproduce the issue in our lab. We were able to successfully install the latest MLNX_OFED 4.7 GA with NVMEoF, load the modules and connect to the NVMEoF target. Make sure, that after the driver is successfully is installed, you reload the nvme modules. And also before you install the driver, install the Python DEB ( apt-get install python). Note: MLNX_OFED 4.6…

Unknown symbol nvme_find_pdev_from_bdev

Hi all, After installing MLNX_OFED_LINUX-4.4-1 on Ubuntu 18.04 (kernel 4.15.0-24) as “$ mlnxofedinstall --force --without-dkms --with-nvmf” I’m trying to use RDMA tools, but modprobe on nvme_rdma fails with “nvme_rdma: Unknown symbol nvme_delete_wq (err 0)” modprobe on nvmet_rdma fails with “nvmet: Unknown symbol nvme_find_pdev_from_bdev (err 0)” What am I doing wrong, please? I see two kernel modules loaded: nvme and nvme_core. This is Mellanox MCX516A-CCAT ConnectX-5 EN Network Int…

OFED NVMET and NVMET-RDMA on Ubuntu symbol errors

Hello, I tried installing ofed on unbuntu 20.04. I had issues inserting nvmet and nvmet-rdma. I know ubuntu 20.04 was only somewhat supported so i tried 19.10 but i am having the same issues. basically i have tried the following install methods, they all install fine without error, but they all result in the same two errors: ERROR: could not insert ‘nvmet_rdma’: Unknown symbol in module, or unknown parameter (see dmesg) modprobe: ERROR: could not insert ‘nvmet’: Unknown symbol in module, or u…

Unknown symbol nvme_find_pdev_from_bdev

I did not run the installer properly, I used this installed command: ./mlnxofedinstall --force --without-dkms --with-nvmf When it should be: ./mlnxofedinstall --force --without-dkms --with-nvmf --add-kernel-support Complete installation steps that worked for me: wget http://content.mellanox.com/ofed/MLNX_OFED-4.4-1.0.0.0/MLNX_OFED_LINUX-4.4-1.0.0.0-ubuntu18.04-x86_64.tgz tar xvzf MLNX_OFED_LINUX-4.4-1.0.0.0-ubuntu18.04-x86_64.tgz cd MLNX_OFED_LINUX-4.4-1.0.0.0-ubuntu18.04-x86_64/ …

Issue observed while inserting 'nvme_rdma' module after OFED driver installation.

Hello Ankit, Many thanks for posting your issue on the Mellanox Community. Based on the information you provided, we were not able to reproduce the issue in our lab. We were able to successfully install the latest MLNX_OFED 4.7 GA with NVMEoF, load the modules and connect to the NVMEoF target. Make sure, that after the driver is successfully is installed, you reload the nvme modules. And also before you install the driver, install the Python DEB ( apt-get install python). Note: MLNX_OFED 4.6…

OEL 7.7 + OFED 4.7

Hello I have a problem with OEL 7.7 with kernel 4.14.35-1902.3.2.el7uek.x86_64 and OFED version MLNX_OFED_LINUX-4.7-1.0.0.1 (OFED-4.7-1.0.0) when I try to load modules : modprobe nvmet modprobe nvmet-rdma modprobe nvme-rdma modprobe nvme-fabrics I get those errors : [ 220.107697] nvme_core: Unknown symbol __dtrace_probe___perf_nvme_complete_rq (err 0) [ 220.107854] nvme_core: Unknown symbol __dtrace_probe___perf_nvme_setup_cmd (err 0) [ 220.108236] nvme_core: Unknown symbol __dtrace_prob…

[gilh]

Hi

1. You can reboot the kernel, and all the drivers will load
2. Don’t try to remove the nvme (modprobe -r nvme) do only modprobe nvme-rdma