ConnectX-5 RoCEv2 Reliable connection Ack vlan issue

[jpaul2]

Trying to support RoCEv2 in our product. We have a particular network configuration where we segregate traffic using internal vlans. Incoming traffic comes on a particular internal vlan (let’s say vlan-10), and we expect outgoing traffic to go over an external vlan (vlan-id different from that of internal vlan. let’s say vlan-500).
gid table has the correct mapping…, and IP’s in the packet is also right. i.e.,
CLIENT-FACING-CLUSTER-IP → vlan-500
INTERNAL-IP → vlan-10

For Reliable connection Ack’s, looks like firmware is not using gid table to set vlan-id in the Ack. Looks to be a bug. Can anyone confirm? Any workaround?

For reference, gid table is set in kernel (mlx5_core_roce_gid_set) using this structure.
struct mlx5_ifc_roce_addr_layout_bits {
u8 source_l3_address[16][0x8];

  u8         reserved_at_80[0x3];
  u8         vlan_valid[0x1];
  u8         vlan_id[0xc];
  u8         source_mac_47_32[0x10];

  u8         source_mac_31_0[0x20];

  u8         reserved_at_c0[0x14];
  u8         roce_l3_type[0x4];
  u8         roce_version[0x8];

  u8         reserved_at_e0[0x20];

};

For eg., for above vlan-id’s, incoming packet has
src-ip: CLIENT-IP
dest-ip: CLIENT-FACING-CLUSTER-IP
vlan-id: 10

My expectation is that RC Ack be built as:
src-ip: CLIENT-FACING-CLUSTER-IP
dest-ip: CLIENT-IP
vlan-id: 500

[xiaofengl]

Why think The RC ACK on internal VLAN not correct? Are you using VF QinQ?

If ACK on external VLAN, then how the APP ack? It run internal VLAN, right?

If you do have requirement ACK on external VLAN, then try use Verbs API strip cvaln.

   ibv_modify_wq() modifys a WQ wq.  The argument wq_attr is an
   ibv_wq_attr struct, as defined in <infiniband/verbs.h>.

   struct ibv_wq_attr {
           uint32_t                attr_mask;     /* Use enum ibv_wq_attr_mask */
           enum ibv_wq_state       wq_state;      /* Move to this state */
           enum ibv_wq_state       curr_wq_state; /* Assume this is the current state */
           uint32_t                flags;         /* Flags values to modify, use enum ibv_wq_flags */
           uint32_t                flags_mask;    /* Which flags to modify, use enum ibv_wq_flags */
   };

   The function ibv_modify_wq() will modify the WQ based on the
   given wq_attr->attr_mask

   enum ibv_wq_flags {
           IBV_WQ_FLAGS_CVLAN_STRIPPING            = 1 << 0, /* CVLAN

[jpaul2]

We have particular network design due to a switch and a bunch of internal vlans, hence the data comes in on a particular vlan, and goes out in a different vlan as explained in example above.

Will check/try the cvlan option here and get back.

Thanks.

[jpaul2]

@xiaofengl : We use SRQ, and I dont see a way of running above verb on it. Any other suggestions?