vNIC Limitations of BX5020 Gateways

I was recently engaging with Mellanox technical support about the vNIC limitations on BX5020 gateways in various modes of operation, and was told:

The “vnics-mode 4k” limitation allows up to 4,000 total vNics on the gateway.

For this to work correctly, the “single-gw-mode” must also be enabled in the group. With these settings configured, there should be no problems increasing your vNic count above 1024.

If you weren’t including Ethernet ports in the port channel, then you could actually set the “vnics-mode” to “unlimited”.

But, in your configuration 4K will be the limitation.

This response gave rise to some follow-up questions that I haven’t been able to get Mellanox to answer, so I thought I’d ask the community in case anyone knows or has experience with this.

Our current configuration is that we have a 2x10GbE LAG (port-channel) bridged to an InfiniBand fabric with around 950 hosts on it. The reason for my initial question is that we are adding to the fabric and will soon be exceeding 1,000 nodes (and 1,000 vNICs) and I wanted to make sure we wouldn’t run into a 1k VNIC limit. It sounds like we will not, but now I am wondering the following:

  1. What is meant by “not including Ethernet ports in the port channel” in order to be able to set unlimitied-VNICs mode? What other ports could you include in a port channel on a BX5020? Doesn’t the BX5020 consist of only Ethernet and InfiniBand ports (which don’t really have a concept of port channels, as the subnet manager handles the parallel routes)? This response makes me wonder if I have some sort of fundamental misunderstanding of the BX5020 as a system.

  2. Other than allowing one to increase the vNIC limit to 4k from 1k, what are the effects of turning on single-gw-mode? What does single-gw-mode do? I have not been able to find an answer to this in the documentation.

Thanks in advance for any answers or suggestions,

Rusty D

e-port = LAG member external port. These are the A2/3/4/5

which connect to the ETH fabric

Single GW mode = Means you can take an

entire (or a part of - meaning part of the eports) a group (A, B, C, D) and

configure it to run as a single GW. All the configured eports of that group will

be associated with the port-channel (LAG) configured for that GW.

If no eports are used - the BXM will assume that all the eports are used - and will

not set require any vNIC limitation to the group (aka unlimited).

A group of vNICs in multiple hosts together with a single Gateway form a virtual

Hub (vHub). A gateway = proxy for the IO traffic of all the vNICs in the

vHub, and is associated with a single Gateway port (i.e., a physical Ethernet


What is meant by "not including Ethernet ports in the port

channel" in order to be able to set unlimitied-VNICs mode? Please refer to the

above. It means not to include eports in a LAG.

Single Gateway mode enables you to save lookup table (AKA context table) space. The new lookup will

be executed by the pair [MAC,VLAN] enabling the vNIC entry to be written only

once instead of per eport (in case you setup the LAG with an specific eports),

hence increasing the number of vNICs configured.

Attaching an eport to a LAG triggers a dump of all vNICs to the eport’s context table. With an

“unlimited vNIC” LAG gateway, the number of vNICs might exceed the hardware vNIC

capacity - thus a user cannot use the unlimited flag in this mode (i.e. non-single GW mode)

If a LAG gateway is configured without any eports, it is unnecessary to limit the number of vNICs since no vNIC will be written to the hardware (as there are no per eport context tables - because no eports are

explicitly attached to the LAG)

So - In the Single Mode, the gateway

vNIC count limit it set to unlimited hence not requiring any changes to the

group gateway lookup mode such as in case of 1K/4K modes.

The limitation is 5k vnics for the entire box (when working in global mode of 12 external ports

and 4 internal ports)