Unable to create send CQ of size 5080 on mlx4_0: Cannot allocate memory

Infrastructure & Networking Switches and Gateways Ethernet Switches
1

I recently installed Intel’s oneapi and want to run some benchmarks on an HP enclosure with a Mellanox SX1018HP Enet Switch .

When I run this command I get the errors below.

[hussaif1@lustwzb31 pt2pt]$ mpirun -v -np 1 -ppn 2 -genv I_MPI_DEBUG=5 -genv I_MPI_HYDRA_DEBUG=1 -hosts b31,b32 ./osu_latency

lustwzb31:rank0.osu_latency: Unable to create send CQ of size 5080 on mlx4_0: Cannot allocate memory

lustwzb31:rank0.osu_latency: Unable to initialize verbs

lustwzb31:rank0: PSM3 can’t open nic unit: 0 (err=23)

Abort(1615503) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:

MPIR_Init_thread(138)…:

MPID_Init(1183)…:

MPIDI_OFI_mpi_init_hook(1916):

create_endpoint(2560)…: OFI endpoint open failed (ofi_init.c:2560:create_endpoint:Invalid argument)

2

Hello Faraz,

Thank you for posting your inquiry on the NVIDIA Networking Community.

Based on the information provided, unfortunately we do not support Intel MPI. NVIDIA delivers HPC-X for which all components are optimized with our products.

It is best to move this discussion to the Intel MPI One API community → Intel® oneAPI HPC Toolkit - Intel Communities

Thank you and regards,

~NVIDIA Networking Technical Support

3

Hi Faraz,

I recently upgraded my oneapi to 2021.04 and got the same error. It turned out I was using incorrect libfabric provider. Once I set it right, the code ran fine. Hope this helps.

4

Does Mellanox support PSM3?