I’m trying to setup a partition on an sx6036 switch between two linux hosts. I’ve defined a partition on the switch having Pkey 0x0001with “Full/ALL” Membership. I’ve added the PKey to the linux hosts with echo 0x0001 > /sys/class/net/ib1/create_child which created a virtual nic named ib1.8001 on both. I’ve assigned 172.100.1.x/24 to each ib1.8001 but unable to ping between them. Both linux hosts report P_Key 0x8001 is not found and ib1.8001: link is not ready. I presume the link is down since the Pkey isn’t found, but why is the Pkey not found? Thanks.
Hello Dee,
Thank you for posting your inquiry on the Mellanox Community.
Based on your information provided, we want to re-share the following information.
There are special requirements for the names of IPoIB P_Key interfaces.
All IPoIB P_Keys range from 0x0000 to 0x7fff, and the high bit, 0x8000, denotes that membership in a P_Key is full membership instead of partial membership.
The Linux kernel’s IPoIB driver only supports full membership in P_Key subnets, so for any subnet that Linux can connect to, the high bit of the P_Key number will always be set. That means that if a Linux computer joins P_Key 0x0001, its actual P_Key number once joined will be 0x8001, denoting that we are full members of P_Key 0x0001.
For this reason, when creating a P_Key definition on a switch or in an opensm partitions.conf file on a node which is running OpenSM, it is required to specify a P_Key value without 0x8000, but when defining the P_Key IPoIB interfaces on the Linux clients, add the 0x8000 value to the base P_Key value.
In this situation, you need to create the child interface by the following syntax:
# echo 0x8001 > /sys/class/net/ib1/create_child
Thanks and regards,
~Mellanox Technical Support