Multiple RoCE devices with the same node guid

I’m working with Soft RoCE on debian. Specifically we have 3 network interfaces on debian machine: eth0, eth0.1001 (vlan), eth0.1002 (vlan). All 3 have the same mac address. I’ve created 3 rxe devices: rxe0, rxe1, rxe2 corresponding to the above interfaces. All 3 rxe devices received the same node_guid since as far as I understand node_guid is derived from the interface mac address. When I try to resolve (using rdma_resolve_addr function) a link-local interface ipv6 address (through one of the above interfaces) the code in librdmacm library always selects the first rxe device out of 3 available devices (no matter to which interface it is attached) because rxe device search is done by node_guid and since all 3 rxe devices have the same node_guid the first one is always matched. My question is how to resolve such situation ? I’d like librdmacm to select rxe device attached to the given link-local interface. Should I make mac addresses of the 3 network devices different ? Any other options ?

BTW, it looks like that similar operation in linux kernel does succeed to match rxe device attached to the interface because it matches the device by both gid and ndev (network device/interface).



Hi Gregory,

Can you please clarify - are you working with Mellanox adapter cards?