Deep buried in MLNX_OFED 4 release notes is a laconic remark that support for NFS over RDMA has been removed. No rationale is provided, and seemingly no one knows why this useful feature was omitted. Fortunately, the release notes turn out to be inaccurate on this point, and the NFS/RDMA support is in fact included in MLNX_OFED, but is not installed by default.
The package that must be installed in addition to the standard MLNX_OFED install is mlnx-nfsrdma-dkms. The procedure below is for Ubuntu 16.04 with kernel 4.13. As of this writing this is the HWE kernel; Ubuntu ISOs are currently distributed with kernel 4.10, which is updated to 4.13 by full upgrade. I tested kernel 4.4 and found that it is not compatible with mlnx-nfsrdma-dkms, so if you have an install with kernel 4.4, you need to replace it with the HWE kernel.
I assume that a similar solution would work with other versions of Ubuntu and with other Linux distros, but I did not test that. If someone is interested and would like to try, please don’t hesitate to post your results.
I tested MLNX_OFED 4.2-1.2.0.0 (newest) and 4.2-1.0.0.0 (previous), with the same success, by installing it from the tarball. This needs to be done on all machines involved with NFS over RDMA, i.e. NFS server(s) and clients.
Tarball used here is MLNX_OFED_LINUX-4.2-1.2.0.0-ubuntu16.04-x86_64.tgz http://www.mellanox.com/page/mlnx_ofed_eula?mtag=linux_sw_drivers&mrequest=downloads&mtype=ofed&mver=MLNX_OFED-4.2-1.2.0.0&mname=MLNX_OFED_LINUX-4.2-1.2.0.0-ubuntu16.04-x86_64.tgz
First download MLNX_OFED tarball and install it the standard way (here tarball is unpacked to the home directory):
tar xzf ~/MLNX_OFED_LINUX-4.2-1.2.0.0-ubuntu16.04-x86_64.tgz cd ~/MLNX_OFED_LINUX-4.2-1.2.0.0-ubuntu16.04-x86_64/ ./mlnxofedinstall
The missing package is in source directory of the above tarball. Unpack, build and install it:
tar xzf ~/MLNX_OFED_LINUX-4.2-1.2.0.0-ubuntu16.04-x86_64/src/MLNX_OFED_SRC-4.2-1.2.0.0.tgz cd ~/MLNX_OFED_SRC-4.2-1.2.0.0/ ./install.pl --kernel-only --build-only cd ~/MLNX_OFED_SRC-4.2-1.2.0.0/DEBS/ubuntu16.04/x86_64/ dpkg -i mlnx-nfsrdma-dkms_3.4-OFED.4.2.1.2.0.1.gf8de107_all.deb
Next, reboot the machine.
To test, export a share on the server, load rpcrdma module and configure rdma port.
Example (customize to your needs):
exportfs -o rw,fsid=1,async,no_subtree_check 10.0.0.0/24:/mnt/share/ modprobe rpcrdma echo rdma 20049 > /proc/fs/nfsd/portlist
On the client, load rpcrdma module and mount the share. Example:
modprobe rpcrdma mount -o rdma,port=20049 10.2.0.1:/mnt/share /mnt/mountpoint
Enjoy!