libi40iw-i40iw_upost_send: post work request failed, invalid opcode: 0x3

[markus.bentele1]

hello

running an mpi test on ib Fails.

host is hpe xl230 redhatel 7.5 mlx5 . ibstat ibping is successful. Limits memlock are unlimited

any ideas?

thanks a lot

Markus Bentele

starccm+ -licpath 1716@flexlmhpc mailto:1716@flexlmhpc -mpi platform -fabricverbose -mppflags “-v -prot -e MPI_IC_ORDER=ibv -affopt=v” -on fnx601:4 -server -rsh “ssh -x”

Parsing application description…

Identifying hosts…

Spawning processes…

Process Map:

Process on host: fnx601

rank 0

rank 1

rank 2

rank 3

Process Map End.

Starting STAR-CCM+ parallel server

Process layout for world 0 is as follows:

mpirun: proc 3259

daemon proc 3262 on host 10.96.75.63

rank 0: proc 3887

rank 1: proc 3876

rank 2: proc 3893

rank 3: proc 3892

libi40iw-i40iw_upost_send: post work request failed, invalid opcode: 0x3

libi40iw-i40iw_upost_send: post work request failed, invalid opcode: 0x3

libi40iw-i40iw_upost_send: post work request failed, invalid opcode: 0x3

starccm+: Rank 0:1: MPI_Init: ibv_post_send() failed

starccm+: Rank 0:0: MPI_Init: ibv_post_send() failed

starccm+: Rank 0:1: MPI_Init: ibv_ring_sync() failed

starccm+: Rank 0:0: MPI_Init: ibv_ring_sync() failed

starccm+: Rank 0:1: MPI_Init: Internal Error: Processes cannot connect to rdma device

starccm+: Rank 0:0: MPI_Init: Internal Error: Processes cannot connect to rdma device

[yairi]

Hi Markus,

Thank you for posting your question on the Mellanox Community.

Based on the information provided, the issue occurs on the Intel Ethernet card and not on the Mellanox ConnectX-4/5 (mlx5) card.

For this issue, you need to contact Intel as they are the maintainer of the card and i40iw verbs.

Thanks and regards,

~Mellanox Technical Support