Is there anyway to force the link down on a port on a ConnectX-4 EN card?

Infrastructure & Networking Adapters and Cables
1

I have never encountered a network card before where I couldn’t do this. I tried the admin_status but that does not bring the actual port down and the other end still sees a link, which is rather inconvenient when trying to manage link aggregation and such. There must be some way to do this. I have thought of trying a hack by forcing the port to 56Gb and just assuming it won’t ever be connected to something that supports that speed (so as long as it isn’t a Mellanox switch or another identical card it should kill the link I would hope, but that’s not a great solution).

2

Hi,

You can use the mlxconfig tool (part of MFT package):

https://www.mellanox.com/page/management_tools

Please run the following :

  • mst start
  • mst mstatus -v ( to obtain the device name)
  • This port functionality can be disabled by executing this mlxconfig command:
    • sudo mlxconfig -d mlx5_0 set KEEP_ETH_LINK_UP_P1=0

or

  • sudo mlxconfig -d mlx5_0 set KEEP_ETH_LINK_UP_P2=0

Note 1 : mlx5_0 is the device name

Note 2 : KEEP_ETH_LINK_UP_P[1/2] is for the two ports (or just _P1 on a single port card).

Thanks,

Samer

3

I am looking for something to add to the driver as an ioctl since it is just to be done as part of setting the link down. mlxconfig does not seem at all appropriate unless whatever that does I can copy and do directly from the driver instead but I highly doubt that. I just want setting the link down from software to actually set the link down.

4

Hi Lennart,

There are 2 answers for such question:

  1. For logical link there are some methods to bring down the interface such as:

ifdown eth0

ip link set dev eth0 down

ifconfig eth0 down

  1. For the physical link , in addition to the mlxconfig you can use the mlxlink utility:

https://docs.mellanox.com/display/MFTv4110/mlxlink+Utility

By using the -a flag , -a |–port_state <port_state>Configure Port State [UP(up)/DN(down)/TG(toggle)]

i.e:

mlxlink -d /dev/mst/mt4119_pciconf0 -a DN

Thanks,

Samer

5

I just realized that what I am in fact trying to do is power off the transceiver to make sure the other end of the link sees it as down. Right now doing ifconfig down does not make the link appear down to the switch at the other end.

Is there an equivalent for mlxlink in mstflint that works with the linux kernel driver? I have installed the mlx stuff once before on a test setup and I don’t ever want to go through that again.

Our software uses a custom ioctl to shut down ports in some cases to do with link aggregation, so I need something I can add as an ioctl in the network driver that will turn off the port. I can’t be calling external tools or using extra drivers. So far this has worked fine for bnx2, e1000, e1000e, igb, egbxe and i40e in various products over the years, but no luck so far with mlx5_core for our newest one we are working on. I tried changing the advertised port speeds, forcing the port speed, etc, but that does nothing, probably because the transceiver is still powered and happily connecting to the other side at its correct speed.

So I guess really the question is if there is a way from software to control power to the transceiver in a specific port.

6

After rereading this and trying it, that actually works exactly as we wanted. Perfect. Now I just have to update the tool for manufacturing to start setting that setting on new systems. And I am happy to discover mlxconfig and mlxlink work just fine without mst drivers loaded which makes life much simpler. Using -d af:00.0 worked perfectly with the standard linux kernel driver.