Replacement for NFS over RDMA(RoCE)

I am attempt to build a new NAS that will be dedicated for our HPC. The current NAS uses NFS to share a number of ZFS filesystems over 10G Ethernet. I would like to have the same ZFS filesystems shared via NFS over the Infiniband network that is already in place to hopefully speed things up. During my research I found out about NFS over RDMA. I than found out that as of OFED version 4.0- driver support was dropped, see page 14 of the release notes. Since the new NAS is going to use Debian 9 I would have to either build the 3.x OFED drivers myself or use the distro drivers. Since this is all Mellanox hardware I would rather not use the distro drivers nor do I want to use old versions if I can avoid it.

Hence this question. Is there a good replacement for NFS over RDMA? I would like to avoid using an iSCSI solution as that would mean a large amount of work getting something new set up. I can try using NFS with IPoIB only but testing shows there are some odd slowdowns when the fabric is in use for computational runs.

The article HowTo Configure NFS over RDMA (RoCE) was what first had me hopeful but it uses OFED 2.4 which will not work out.

I did find this discussion: How to use NFS over RDMA with MLNX_OFED, but the driver has been removed from the latest 4.x versions.


As of MLNX_OFED v4.7, the NFSoRDMA driver is no longer installed by default. In order to install it over a supported kernel, add the “ --with-nfsrdma ” installation option to the “mlnxofedinstall” script.

Debian 9 is not a supported OS for NFSoRDMA . Here the the supported operating systems as listed here

  • SLES12 SP4
  • SLES12 SP5
  • SLES15 SP1
  • Ubuntu 18.04.3
  • RedHat 7.5
  • RedHat 7.6
  • RedHat 7.7
  • RedHat 7.8
  • RedHat 8.0
  • RedHat 8.1

In order to use NFSoRDMA you must use one of these operating systems.

In terms of other solutions you can find more information on our storage solutions here

If you have any questions about specific storage solutions feel free to open a case with us about it as you have a valid support contract.

Thanks and regards,

~Mellanox Technical Support