When the nvidia-peermem driver is compiled without Mellanox ib_peer_mem symbols available, the resulting module silently returns EINVAL when you try to load it. The offending code in the driver is here:
This can happen if you install (e.g.) the DKMS cuda drivers from NVIDIA.
This is an extremely unfriendly failure mode as it provides zero information about what the problem is. Please add a call to peer_err() with a clear error message before the “return -EINVAL” at the end of nv_mem_client_init(). Thank you!