What is the use of Mellanox MT27800 Family ConnectX-5 virtual ports?

Infrastructure & Networking Software And Drivers
, , , ,
1

I have 100 Gb/s EDR two port card (ConnectX-5) in CentOS system.

I have installed “InfiniBand Support” group package and have the opensm running in fabric.

Driver creates four interfaces ib[0-3] where ib0 and ib2 are in connected state. It corresponds with physical port 1 connected to the switch. My question is, what is the use of two virtual interfaces on one physical? Currently this puzzles me, since when I try to connect all flash OceanStor array via IB (SRP), it show me two targets, probably since there are two virtual ports connected to the same target. Also on the array I see two initiators from this system.

2

Thanks for the clear and comprehensible answer.

Kreps

3

Hi Kreps,

Please provide the following output from the host:

#[lspci -d 15b3:]​

#[ip addr]​

#ibdev2netdev​

#[ofed_info -s]​

#ibv_devinfo​

To generate the below outputs, make sure you have MFT installed on the host. Also, these commands are executed as “root”

To download if not already installed → https://www.mellanox.com/page/management_tools

For compilation and installation —> https://docs.mellanox.com/display/MFTV4120/Compilation+and+Installation

#[mst start]​

#[mst status -v]​

Thanks,

Namrata.

4

Hi Namrata,

please find the required output in the attached file.

Thanks Kreps

5

Hi Kreps,

I will address the following question by you “My question is, what is the use of two virtual interfaces on one physical?”

Based on the information on the PSID (MT_0000000023) provided , this is a Socket Direct Card and based on the Product Brief of this card, “ConnectX-5 Socket Direct provides 100Gb/s port speed even to servers without x16 PCIe slots by splitting the 16-lane PCIe bus into two 8-lane buses, one of which is accessible through a PCIe x8 edge connector and the other through a parallel x8 Auxiliary PCIe Connection Card, connected by a dedicated harness.”

Thus, due to lane splitting each port is seen as 2 network interfaces thereby providing better performance.

For information from the Product Brief ----> http://www.mellanox.com/related-docs/prod_adapter_cards/PB_ConnectX-5_VPI_Card_SocketDirect.pdf

Thanks,

Namrata.

6

One more question.

Does it mean, that to fully utilize the 100Gb/s speed, I have to bond these two network interfaces ports together, since each is connected to one PCIe x8 bus?

Thanks Kreps