CIFS Error - Alma Linux 8.7

After we install cifs-utils package on to the system and There is no cifs support advertised in /proc/filesystems. Doing a modprobe cifs works, but modinfo cifs shows that it’s the Mellanox dummy cifs kernel module, and trying to mount shares gives the same error as above. Looking deeper, I found that the cifs module is getting overridden by the Mellanox drivers.

[root@almahpc ~]# modprobe cifs
[root@almahpc ~]# modinfo cifs
filename: /lib/modules/4.18.0-425.3.1.el8.x86_64/extra/mlnx-ofa_kernel/fs/cifs/cifs.ko
version: 2.31
license: Dual BSD/GPL
description: cifs dummy kernel module
author: Mohammad Kabat
rhelversion: 8.7
srcversion: 01E451882B55F354B7F130B
depends: mlx_compat
name: cifs
vermagic: 4.18.0-425.3.1.el8.x86_64 SMP mod_unload modversions
[root@ almahpc ~]# grep -c cifs /proc/filesystems
0
[root@almahpc ~]# grep cifs /proc/filesystems
[root@almahpc ~]#

I removed the MLNX driver, and I see now the modinfo cifs is giving the correct version.

[root@almahpc MLNX_OFED_LINUX-23.04-0.5.3.3-rhel8.7-x86_64]# ./uninstall.sh

This program will uninstall all MLNX_OFED_LINUX-23.04-0.5.3.3 packages on your machine.

Do you want to continue?[y/N]:y

rpm -e --allmatches --nodeps mstflint sharp mlnx-ethtool mlnx-ofa_kernel-modules iser infiniband-diags ibutils2 ucx-rdmacm ibarr mlnx-tools knem-modules rdma-core librdmacm-utils libibverbs mlnx-ofa_kernel-source opensm-static srp_daemon ucx ucx-ib hcoll rshim xpmem isert librdmacm libibverbs-utils opensm-libs opensm-devel dpcp ucx-cma ucx-xpmem mlnx-iproute2 mpitests_openmpi mlnx-ofa_kernel kernel-mft xpmem-modules libxpmem libibumad ibsim opensm ibdump ucx-knem mlnxofed-docs knem ibacm perftest ucx-devel openmpi mlnx-ofa_kernel-devel srp rdma-core-devel infiniband-diags rdma-core rshim mlnx-fw-updater mft kernel-mft ofed-scripts rdma-core-devel mlnx-ofa_kernel-23.04-OFED.23.04.0.5.3.1.rhel8u7.x86_64 sharp-3.3.0.MLNX20230417.ec919ce9-1.2304053.x86_64 hcoll-4.8.3221-1.2304053.x86_64 opensm-5.15.0.MLNX20230417.d84ecf64-0.1.2304053.x86_64 opensm-devel-5.15.0.MLNX20230417.d84ecf64-0.1.2304053.x86_64 opensm-static-5.15.0.MLNX20230417.d84ecf64-0.1.2304053.x86_64
Uninstall finished successfully

[root@almahpc MLNX_OFED_LINUX-23.04-0.5.3.3-rhel8.7-x86_64]# modinfo cifs
filename: /lib/modules/4.18.0-425.3.1.el8.x86_64/kernel/fs/cifs/cifs.ko.xz
softdep: gcm
softdep: ccm
softdep: aead2
softdep: sha512
softdep: sha256
softdep: cmac
softdep: aes
softdep: nls
softdep: md5
softdep: md4
softdep: hmac
softdep: ecb
softdep: pre: des
version: 2.29
description: VFS to access SMB3 servers e.g. Samba, Macs, Azure and Windows (and also older servers complying with the SNIA CIFS Specification)
license: GPL
author: Steve French
alias: smb3
alias: fs-smb3
alias: fs-cifs
rhelversion: 8.7
srcversion: 368290631B75A5A1CA09E2C
depends: ib_core,rdma_cm,libarc4,dns_resolver
intree: Y
name: cifs
vermagic: 4.18.0-425.3.1.el8.x86_64 SMP mod_unload modversions
sig_id: PKCS#7
signer: AlmaLinux kernel signing key
sig_key: 34:62:E5:6E:49:5A:36:CC:92:3A:DC:C9:8B:40:D9:2C:16:AB:77:F6
sig_hashalgo: sha256
signature: 5E:4D:97:42:7B:80:B8:41:60:84:FC:AB:CE:57:76:22:B6:B3:7F:CA:
AF:73:72:E1:2B:74:08:2B:8B:0A:AF:A3:F6:02:36:3C:DC:3D:B6:46:
A9:22:17:00:26:02:C2:39:32:5A:D0:95:65:70:C3:21:9D:58:3A:AC:
EF:F8:0F:41:05:96:62:BD:AA:BA:86:E3:BB:A1:2B:FE:E5:F0:EF:EA:
5C:8E:7F:4B:B9:96:55:FD:12:00:F3:F4:43:7C:5F:06:53:4D:CE:24:
4F:B4:DA:FE:51:DD:2A:10:A5:66:5B:5A:FD:40:8D:33:CD:AE:69:F1:
9A:2A:F4:86:6D:DB:26:18:83:A4:33:68:57:55:9C:7E:A3:E4:7B:A8:
B9:1D:D3:74:78:1D:E4:FE:9F:E8:33:8A:51:C0:79:54:15:20:AC:95:
C6:54:CD:8F:71:5E:FA:01:FE:1B:08:76:76:0D:C4:AF:35:A8:96:D6:
C6:11:6C:E8:4D:62:B9:B5:D7:B3:BA:C5:8D:D7:8F:90:F8:5D:E1:62:
E0:85:22:6F:73:49:FE:74:D5:8C:48:8D:70:C2:98:4E:AB:8C:AB:FC:
76:55:2D:E6:08:23:99:A9:D4:60:07:1A:08:8F:5B:56:74:F6:1F:4C:
80:BA:48:BC:BE:9D:8C:A1:79:5E:92:34:CA:BA:E1:B3:93:12:23:33:
BB:12:9B:9E:D4:B1:AD:0C:10:04:2D:92:4C:B6:D6:1A:C6:1A:D9:0C:
21:F0:BA:2D:43:C5:F3:2E:B0:10:8D:6C:51:1A:81:A1:52:46:F9:8D:
88:48:57:63:BC:25:21:2A:06:76:F0:CE:F1:98:DF:04:6E:4E:DF:28:
8C:4B:90:6C:2F:52:CE:4A:6A:A7:12:66:4E:90:D9:29:F2:11:44:D7:
B3:49:C8:79:E3:DD:DC:47:00:04:44:6D:73:65:EC:40:95:7B:38:0A:
65:0C:88:C5:E7:B2:95:53:F1:83:91:D3:AA:1C:A5:80:09:6B:03:42:
5C:B7:5D:C6
parm: smbd_logging_class:Logging class for SMBD transport 0x0 to 0x100 (uint)
parm: smbd_logging_level:Logging level for SMBD transport, 0 (default): error, 1: info (uint)
parm: CIFSMaxBufSize:Network buffer size (not including header) for CIFS requests. Default: 16384 Range: 8192 to 130048 (uint)
parm: cifs_min_rcv:Network buffers in pool. Default: 4 Range: 1 to 64 (uint)
parm: cifs_min_small:Small network buffers in pool. Default: 30 Range: 2 to 256 (uint)
parm: cifs_max_pending:Simultaneous requests to server for CIFS/SMB1 dialect (N/A for SMB3) Default: 32767 Range: 2 to 32767. (uint)
parm: enable_oplocks:Enable or disable oplocks. Default: y/Y/1 (bool)
parm: enable_gcm_256:Enable requesting strongest (256 bit) GCM encryption. Default: n/N/0 (bool)
parm: require_gcm_256:Require strongest (256 bit) GCM encryption. Default: n/N/0 (bool)
parm: disable_legacy_dialects:To improve security it may be helpful to restrict the ability to override the default dialects (SMB2.1, SMB3 and SMB3.02) on mount with old dialects (CIFS/SMB1 and SMB2) since vers=1.0 (CIFS/SMB1) and vers=2.0 are weaker and less secure. Default: n/N/0 (bool)
[root@almahpc MLNX_OFED_LINUX-23.04-0.5.3.3-rhel8.7-x86_64]#

This looks like a known issue with the Mellanox driver and still no solution is available.
https://docs.nvidia.com/networking/display/MLNXOFEDv23041130/Known+Issues

If you remove the MLNX driver, you can mount CIFS as usual.
I found some information in the following link to fix it
https://mellanox.my.site.com/mellanoxcommunity/s/question/0D51T00008sHiFlSAK/mount-error-cifs-filesystem-not-supported-by-the-system

Could you please provide the procedure to recompile the CIFS module manually after installation of MLNX_OFED.?

Hello and thank you for your question.

When MLNX_OFED is installed, it replaces the INBOX IB driver implementation, rendering the original CIF kernel module dependent on it non-functional.

Consequently, we provide a solution by substituting it with a dummy module, effectively removing CIF support.

With the release of MLNX_OFED 5.5, a minor enhancement was introduced to prevent error displays during CIF loading. However, it’s important to note that CIFS functionality will still not be available.

To ensure compatibility with CIF and our driver, you have two options:

  1. Utilize the inbox driver that comes pre-installed with your operating system distribution, eliminating the need for MLNX_OFED.
  2. Alternatively, you can manually recompile the CIF module after installing MLNX_OFED.

As CIFS is not part of the NVIDIA product portfolio, for further assistance and guidance on re-compiling the CIFS module after MLNX_OFED installation, we recommend reaching out to the CIFS vendor directly. They will be better equipped to provide the necessary support and instructions for this specific scenario.

Thanks,
Ilan.

Hello @ipavis ,

What if we need to mount a SMB share on a machine which has MLNX driver installed?

We are not using IB network for mounting SMB, its via standard ethernet.

Can you please provide the steps of recompile the CIFS module after installing MLNX_OFED ?

@Vinil-Vadakkepurakkal Did you ever find a solution to this? The Mellanox/NVIDIA support has been absolutely useless. They broke a perfectly good cifs module from OS and now refusing to fix it or provide any help about how to recompile it with MOFED symbols. If you managed to make it work that would be really helpful for us.

@kernel_panic - Unfortunately, there wasn’t a clear solution to this problem. However, removing the Mellanox driver allowed CIFS to start working again. With the Mellanox driver installed, I’m unable to use CIFS shares. in HPC with Mellanox driver we can make the cluster working !!

@ipavis I tried compiling/installing the cifs-utils code from source but this, while it compiled cleanly, did not work, as there were likely other dependencies and simply installing cifs from the compile is not enough. I noticed the original kernel cifs module depends on some ib and rdma modules. So instead of fixing this, it appears Nvidia just decided to disable it ? I think the only solution would be to download MOFED source and replace the cifs-utils source there and compile. I’m not sure why this wasn’t fixed or addressed correctly. Can you please explain ? Mounting a share drive should not be broken by installing MOFED, as it is a critical part of our automation to use shared drives via cifs-utils.