NFSoRDMA: svcrdma: Error -12 posting RDMA_READ

1

Hi

I configured the NFSoRDMA according to this document. HowTo Configure NFS over RDMA (RoCE) https://community.mellanox.com/s/article/howto-configure-nfs-over-rdma--roce-x

This is the same problem, but it does not help me. ConnectX-4 LX: NFSoRDMA failures in MOFED-3.3

Here’s my question:

After the configuration, start fio testing,The read test will succeed and the write test will fail !

INFO: NFS SERVER

http://upload-images.jianshu.io/upload_images/1693068-bfa75cf582c071fd.png?imageMogr2/auto-orient/strip|imageView2/2/w/1240

INFO: NFS CLIENT:

http://upload-images.jianshu.io/upload_images/1693068-0550a9ace71ee071.png?imageMogr2/auto-orient/strip|imageView2/2/w/1240

Any suggestions on it ?

Regards

guoqingwang

2

Recently,

I tried to upgrade the kernel to 3.16.46 ( MLNX_OFED_LINUX-3.4-1.0.0.0 ) , but it did not work.

I tried to upgrade the driver to MLNX_OFED_LINUX-4.1-1.0.2.0 (kernel: 3.16.46), but The kernel module (xprtrdma) does not seem to match the driver version.

alkx Infrastructure & Networking - NVIDIA Developer Forums Can you help me and give me some tips ? Thank you

3

Hi guoqingwang,

Thank you for reaching out to the Mellanox Support Community and for your patience in this matter.

Unfortunately, as from Mellanox OFED version 3.4-x and above NFSoRDMA is not supported anymore. This was never mentioned in the Release Notes for version 3.4-1.0.0.0 but it is mentioned in later released version.

If you want to use NFSoRDMA, you need to downgrade the driver to Mellanox OFED version 3.3-1.0.4.0 in which we do still support NFSoRDMA and is available for Ubuntu 14.04 or switch to the OS-vendor supplied driver (INBOX Driver) for the Mellanox NIC.

Hopefully this will clarify and resolves your issue.

Thanks.

Cheers,

~Martijn

4

Hi Martijn :

thank you very much for your reply

After downgraded the driver to Mellanox OFED version 3.3-1.0.4.0 from your advice, I have recompiled the mlnx-ofed-kernel module with nfsrdma support .

But still get same error “svcrdma: Error -12 posting RDMA_READ” while executing “fio --rw=randwrite“.

I wonder if there is a bug or some wrong configuration .

Hope to get any of your suggestions .

Thanks.

Regards

guoqingwang

PS:

HowTo Compile MLNX_OFED Drivers (mlnx-ofa_kernel example) https://community.mellanox.com/s/article/howto-compile-mlnx-ofed-drivers--160---mlnx-ofa-kernel-example-x

options taken from /etc/infiniband/info + –with-nfsrdma-mod

mlnx-ofed-kernel-3.3# ./configure --with-core-mod --with-user_mad-mod --with-user_access-mod --with-addr_trans-mod --with-mlx4-mod --with-mlx4_en-mod --with-mlx5-mod --with-ipoib-mod --with-sdp-mod --with-e_ipoib-mod –with-nfsrdma-mod

mlnx-ofed-kernel-3.3# make

copy xprtrdma.ko to nfs client and svcrdma.ko to nfs server

mlnx-ofed-kernel-3.3# cp net/sunrpc/xprtrdma/xprtrdma.ko /lib/modules/3.16.46-031646-generic/updates/dkms/

mlnx-ofed-kernel-3.3# depmod -a && modprobe xprtrdma