I opened the following case with Ubuntu, but figured I might as well go to the ‘source’ as well:
We are running Ubuntu 18.04 LTS and using the 5.4 kernel series in a bunch of servers. Trying to compile the kernel fails unless the “–no-peermem” option is used.
However, if we try to register the kernel DKMS it does not work because, it seems, the DKMS system does not know about the “–no-peermem” flag and so always try to compile everything.
This is true regardless of whether we use the “NVIDIA-Linux-x86_64-[foo].run” shell installer (which asks yes/no for DKMS), or if we use the “nvidia-driver-470” packages (which pulls in various dependencies). In the latter case it is added:
# dkms status -k `uname -r`
iser, 4.7: added
kernel-mft-dkms, 4.13.0, 5.4.0-97-generic, x86_64: installed
knem, 1.1.3.90mlnx1: added
mlnx-ofed-kernel, 4.7: added
nvidia, 470.103.01: added
rshim, 1.8, 5.4.0-97-generic, x86_64: installed
srp, 4.7: added
However, doing a build gets:
# dkms build nvidia/470.103.01
Kernel preparation unnecessary for this kernel. Skipping...
applying patch disable_fstack-clash-protection_fcf-protection.patch...patching file Kbuild
Hunk #1 succeeded at 82 (offset 11 lines).
Building module:
cleaning build area...
unset ARCH; [ ! -h /usr/bin/cc ] && export CC=/usr/bin/gcc; env NV_VERBOSE=1 'make' -j16 NV_EXCLUDE_BUILD_MODULES='' KERNEL_UNAME=5.4.0-97-generic IGNORE_XEN_PRESENCE=1 IGNORE_CC_MISMATCH=1 SYSSRC=/lib/modules/5.4.0-97-generic/build LD=/usr/bin/ld.bfd modules.........(bad exit status: 2)
ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/nvidia-dkms-470.0.crash'
Error! Bad return status for module build on kernel: 5.4.0-97-generic (x86_64)
Consult /var/lib/dkms/nvidia/470.103.01/build/make.log for more information.
We do not have Infiniband (or RoCE), but do have Mellanox cards: they support VPI, but are completely in ‘Ethernet mode’.
Anyway to tell the kernel module in DKMS to not compile the “peermem” module?
As mentioned in the above Ubuntu bug, I found this thread:
Supposedly “peermem” does not work with an (unpatched?) 5.4 Linux kernel, but supposedly does for 5.11. (Or perhaps it is only in reference to autoloading peermem?)
Thanks for any info.