Issue observed while inserting 'nvme_rdma' module after OFED driver installation.

Used MLNX OFED driver (downloaded from Mellanox website) for my Ubuntu-18.04.3 OS. The installation happens fine but however, I am not able to insert the “nvme-rdma” module. Hence require your help here to rectify.

The setup details are:-

  1. OS :- Ubuntu 18.04.3

  2. Kernel :- 4.15.0-65-generic

  3. OFED packages :- MLNX_OFED_LINUX-4.6-1.0.1.1-ubuntu18.04-x86_64.tgz, MLNX_OFED_LINUX-4.7-1.0.0.1-ubuntu18.04-x86_64.tgz (Tried seperately)

  4. Type of Card :- Mellanox CX-3 and Mellanox CX-5 (Tried seperately)

The errors observed are :-

  1. OFED command used :- ./mlnxofedinstall --kernel-sources /lib/modules/4.15.0-65-generic/build --with-nvmf --add-kernel-support

Reboot the server and restart the openibd service

  1. Command for module insertion :- modprobe nvme-rdma

  2. Output observed :- modprobe: ERROR: could not insert ‘nvme_rdma’: Unknown symbol in module, or unknown parameter (see dmesg)

  3. dmesg output :-

[ 63.987417] nvme_rdma: Unknown symbol nvme_cleanup_cmd (err 0)

[ 63.987590] nvme_rdma: Unknown symbol nvme_delete_wq (err 0)

Please help.

Thanks in advance.

inserting nvme module also give error.

modprobe nvme

[ 3508.577314] nvme: Unknown symbol nvme_reset_ctrl_sync (err 0)

[ 3508.577715] nvme: Unknown symbol nvme_cleanup_cmd (err 0)

[ 3508.577813] nvme: Unknown symbol disk_to_nvme_ns (err 0)

Hello Ankit,

Many thanks for posting your issue on the Mellanox Community.

Based on the information you provided, we were not able to reproduce the issue in our lab. We were able to successfully install the latest MLNX_OFED 4.7 GA with NVMEoF, load the modules and connect to the NVMEoF target.

Make sure, that after the driver is successfully is installed, you reload the nvme modules. And also before you install the driver, install the Python DEB ( apt-get install python).

Note: MLNX_OFED 4.6 does not support Ubuntu 18.04.3 LTS, you need to use the latest GA version which is version 4.7-1.0.0.1

We used the following driver installation options:

./mlnxofedinstall --force-dkms --with-nvmf -vvv

modprobe -rv nvme

/etc/init.d/openibd force-restart

modprobe -v nvme

modprobe -v nvmet

modprobe -v nvmet-rdma

modprobe -v nvme-rdma

Client:

./mlnxofedinstall --force-dkms --with-nvmf -vvv

modprobe -rv nvme

/etc/init.d/openibd force-restart

modprobe -v nvme

modprobe -v nvme-rdma

nvme discover -t rdma -a 192.168.10.21 -s 4420

nvme connect -t rdma -n nvme-subsystem-name -a 192.168.10.21 -s 4420

nvme list-subsys

nvme-subsys0 - NQN=nvme-subsystem-name

\

± nvme0 rdma traddr=192.168.10.21 trsvcid=4420

nvme list

Node SN Model Namespace Usage Format FW Rev


/dev/nvme0n1 50e63ee721969762 Linux 10 400.09 GB / 400.09 GB 512 B + 0 B 4.15.0-6

Dmesg output from target:

[ 4617.496355] nvme nvme0: pci function 0000:03:00.0

[ 4831.046648] nvmet: adding nsid 10 to subsystem nvme-subsystem-name

[ 4924.511922] nvmet_rdma: enabling port 1 (192.168.10.21:4420)

OS: Ubuntu 18.04.3 LTS

Kernel version: 4.15.0-65-generic

Driver version: MLNX_OFED_LINUX-4.7-1.0.0.1

Many thanks,

~Mellanox Technical Support

Hi @Ankit Jain​,

Did you manage to solve the case?

I have the same issue with Oracle Linux 7.7 with MLNX_OFED 4.7 as client:

modprobe nvme-rdma

modprobe: ERROR: could not insert ‘nvme_rdma’: Unknown symbol in module, or unknown parameter (see dmesg)

dmesg|grep nvme

[358215.772584] nvme_core: Unknown symbol __dtrace_probe___perf_nvme_complete_rq (err 0)

[358215.772741] nvme_core: Unknown symbol __dtrace_probe___perf_nvme_setup_cmd (err 0)

[358215.773130] nvme_core: Unknown symbol __dtrace_probe___perf_block_bio_remap (err 0)

[358215.773467] nvme_core: Unknown symbol __dtrace_probe___perf_nvme_async_event (err 0)

Target CentOS 8 works fine.

Thanks for your help,

Bart