ConnectX-6 and Intel X710-TM4 does not link up using 10GbaseKR

[meiguodong]

Hello, Support,

We are using a PCB board(Board A) that uses ConnectX-6 chip , and another PCB(Board B) that uses Intel X710-TM4 chip. We intend to link them up using the 10G Base KR. The 10G Base KR lines(Tx/Rx differential lines) are inter-connected through a backplane(i.e. direct connection in PCB).

Unfortunately the 10G Base KR link does not link up.

We have done some basic trouble-shooting below:
1). Board A, the 10G Base KR can loop-back test, no issues.
2). Board B, if we use Board B to Board B, the 10GBaseKR can link to each other.
3). Between A and B, it does not link up.

I would like to check is there any settings in the ConnectX6 to get the link up?

Below is the configuration info using the mlxlink utility:

root@tegra-ubuntu:~# mlxlink -d /dev/mst/mt4123_pciconf0.1

Operational Info

State : Polling
Physical state : ETH_AN_FSM_ENABLE
Speed : N/A
Width : N/A
FEC : N/A
Loopback Mode : No Loopback
Auto Negotiation : ON

Supported Info

Enabled Link Speed (Ext.) : 0x00000012 (10G,1G)
Supported Cable Speed (Ext.) : 0x00000000 ()

Troubleshooting Info

Status Opcode : 2
Group Opcode : PHY FW
Recommendation : Negotiation failure

Tool Information

Firmware Version : 20.42.0139
amBER Version : 2.13
MFT Version : mft 4.24.0-72

[user155616]

Hi MEIGUODONG,

Please kindly share the output of below command:
mlxconfig -d /dev/mst/mt4123_pciconf0.1 q |grep LINK_TYPE

Thanks

[meiguodong]

Hello user155616,

Thank you for your support.

I will get you the info once it is available.

Just one more question, I noted the shown link speed is 1G and 10G, Do I need to specifically specify the ConnectX-6 chip to be 10G?

Supported Info

Enabled Link Speed (Ext.) : 0x00000012 (10G,1G)
Supported Cable Speed (Ext.) : 0x00000000 ()

I noted these commands in the mlxlink support page:
mlxlink -d -p <port_number> --speeds 25G,50G,100G

Is there an option to specify “–speeds 10G” ?

Do I need to use below commands to configure the port state?
mlxlink -d -p <port_number> --port_state UP

Do I need to specify --rx_rate and --tx_rate?

Thank you for your support.
Mei Guodong

[user155616]

Hi Mei Guodong,

The Auto-negotiation is enabled by default, so you’re not recommended to change the speed and rate.
Could you please share the serial number of CX6 adapter?

Thanks

[user155616]

Hi MEIGUODONG,

Please kindly share the output of below command:
mlxconfig -d /dev/mst/mt4123_pciconf0.1 q |grep LINK_TYPE

Thanks

[meiguodong]

Hello Mellanox Support,

Sorry it took some time to reply you.

mlxconfig -d /dev/mst/mt4123_pciconf0.1 q |grep LINK_TYPE
this commands returns empty results.

Instead I run this command:
xavier@tegra-ubuntu:~$ sudo mlxconfig -d /dev/mst/mt4123_pciconf0 q |grep LINK
PHY_COUNT_LINK_UP_DELAY DELAY_NONE(0)
KEEP_ETH_LINK_UP_P1 True(1)
KEEP_IB_LINK_UP_P1 False(0)
KEEP_LINK_UP_ON_BOOT_P1 False(0)
KEEP_LINK_UP_ON_STANDBY_P1 False(0)
AUTO_POWER_SAVE_LINK_DOWN_P1 False(0)
KEEP_ETH_LINK_UP_P2 True(1)
KEEP_IB_LINK_UP_P2 False(0)
KEEP_LINK_UP_ON_BOOT_P2 False(0)
KEEP_LINK_UP_ON_STANDBY_P2 False(0)
AUTO_POWER_SAVE_LINK_DOWN_P2 False(0)
UNKNOWN_UPLINK_MAC_FLOOD_P1 False(0)
UNKNOWN_UPLINK_MAC_FLOOD_P2 False(0)

Does it give you any useful info?

Please let me re-phrase my requirements:
1). We have a PCB(board-A) board that uses Mellanox ConnectX-6 chip, it supports 10GBaseKR and 1GBaseKX.
2). We have another PCB(board-B), it uses an Intel X710-TM4 chip. The chip supports 1GBaseKX.
3). the two eth ports are connected through backplane connection.
4). Now we will need to link them up using 1GbaseKX mode.

Basic question:
In order to link the above in 1GBaseKX, do I need to specifically specify to use 1GbaseKX? How to?

I was using this command:
sudo mlxlink -d /dev/mst/mt4123_pciconf0.1 -s 1G
sudo mlxlink -d /dev/mst/mt4123_pciconf0.1 -a tg

It can link up in loop-back mode, but does not link up with the Intel x710 chip.

Does the Connect-X6 chip auto-negotiate at 1GBaseKX mode?

Thank you.
Mei Guodong

[user155616]

Hi Mei Guodong,

Thanks for the clarification.
It’s out of our work scope. We only support standard Nvidia adapters with valid entitlements. Please kindly seek assistance from the sales team.

Thanks