I’m having trouble with NFS over RoCE on Ubuntu 16.04 using the latest OFED (MLNX_OFED_LINUX-3.3-1.0.4.0-ubuntu16.04-x86_64.tgz)
Works with Inbox drivers (mostly) but not no much with latest OFED
I managed to get NFS working with RoCE by following the docs on this site using the Inbox drivers for Ubuntu 16.04. I was having some little issues and I know the Ubuntu stuff is out of date so I wanted to install the latest OFED/mlx4 drivers, etc… as per recommendations on this site. So I did that. All went as planned. IP functionality is all there and RDMA tools/tests all work. The newest mlx4 driver is confirmed loaded and everything seems to work great. Except one thing.
Now I have a problem. The svcrdma and xprtrdma modules won’t load. Thus no RDMA support for NFS. I get the following errors. I have a feeling this can be resolved somehow - like by recompiling kernel modules and such but that is over my head at the moment. Or maybe I just messed something up (crossing fingers)? Can anyone help?
NFS server:
modprobe svcrdma
modprobe: ERROR: could not insert ‘rpcrdma’: Invalid argument
dmesg errors:
[105699.696980] rpcrdma**: Unknown symbol rdma_event_msg (err 0)**
[105699.697056] rpcrdma**: disagrees about version of symbol ib_create_cq**
[105699.697059] rpcrdma**: Unknown symbol ib_create_cq (err -22)**
[105699.697069] rpcrdma**: disagrees about version of symbol rdma_resolve_addr**
[105699.697071] rpcrdma**: Unknown symbol rdma_resolve_addr (err -22)**
[105699.697183] rpcrdma**: Unknown symbol ib_event_msg (err 0)**
[105699.697213] rpcrdma**: disagrees about version of symbol ib_dereg_mr**
[105699.697215] rpcrdma**: Unknown symbol ib_dereg_mr (err -22)**
[105699.697224] rpcrdma**: disagrees about version of symbol ib_query_qp**
[105699.697226] rpcrdma**: Unknown symbol ib_query_qp (err -22)**
[105699.697236] rpcrdma**: disagrees about version of symbol rdma_disconnect**
[105699.697238] rpcrdma**: Unknown symbol rdma_disconnect (err -22)**
[105699.697245] rpcrdma**: disagrees about version of symbol ib_alloc_fmr**
[105699.697247] rpcrdma**: Unknown symbol ib_alloc_fmr (err -22)**
[105699.697294] rpcrdma**: disagrees about version of symbol ib_dealloc_fmr**
[105699.697295] rpcrdma**: Unknown symbol ib_dealloc_fmr (err -22)**
[105699.697301] rpcrdma**: disagrees about version of symbol rdma_resolve_route**
[105699.697303] rpcrdma**: Unknown symbol rdma_resolve_route (err -22)**
[105699.697398] rpcrdma**: disagrees about version of symbol rdma_bind_addr**
[105699.697400] rpcrdma**: Unknown symbol rdma_bind_addr (err -22)**
[105699.697441] rpcrdma**: disagrees about version of symbol rdma_create_qp**
[105699.697443] rpcrdma**: Unknown symbol rdma_create_qp (err -22)**
[105699.697479] rpcrdma**: Unknown symbol ib_map_mr_sg (err 0)**
[105699.697487] rpcrdma**: disagrees about version of symbol ib_destroy_cq**
[105699.697489] rpcrdma**: Unknown symbol ib_destroy_cq (err -22)**
[105699.697494] rpcrdma**: disagrees about version of symbol rdma_create_id**
[105699.697496] rpcrdma**: Unknown symbol rdma_create_id (err -22)**
[105699.697582] rpcrdma**: disagrees about version of symbol rdma_listen**
[105699.697584] rpcrdma**: Unknown symbol rdma_listen (err -22)**
[105699.697587] rpcrdma**: disagrees about version of symbol rdma_destroy_qp**
[105699.697589] rpcrdma**: Unknown symbol rdma_destroy_qp (err -22)**
[105699.697597] rpcrdma**: disagrees about version of symbol ib_query_device**
[105699.697599] rpcrdma**: Unknown symbol ib_query_device (err -22)**
[105699.697606] rpcrdma**: disagrees about version of symbol ib_get_dma_mr**
[105699.697607] rpcrdma**: Unknown symbol ib_get_dma_mr (err -22)**
[105699.697617] rpcrdma**: disagrees about version of symbol ib_alloc_pd**
[105699.697618] rpcrdma**: Unknown symbol ib_alloc_pd (err -22)**
[105699.697673] rpcrdma**: Unknown symbol ib_alloc_mr (err 0)**
[105699.697734] rpcrdma**: disagrees about version of symbol rdma_connect**
[105699.697736] rpcrdma**: Unknown symbol rdma_connect (err -22)**
[105699.697769] rpcrdma**: Unknown symbol ib_wc_status_msg (err 0)**
[105699.697842] rpcrdma**: disagrees about version of symbol rdma_destroy_id**
[105699.697844] rpcrdma**: Unknown symbol rdma_destroy_id (err -22)**
[105699.697872] rpcrdma**: disagrees about version of symbol rdma_accept**
[105699.697874] rpcrdma**: Unknown symbol rdma_accept (err -22)**
[105699.697882] rpcrdma**: disagrees about version of symbol ib_destroy_qp**
[105699.697883] rpcrdma**: Unknown symbol ib_destroy_qp (err -22)**
[105699.697964] rpcrdma**: disagrees about version of symbol ib_dealloc_pd**
[105699.697965] rpcrdma**: Unknown symbol ib_dealloc_pd (err -22)**