How to change FEC mode on ConnectX-7 NIC and QM9790 Switch?

Hi,

I have one server (Centos 7.9) with 2 infiniband ConnectX-7 single port adapters and one QM9790 IB switch. The port of each CX7 adapter has a 400G OSFP112 DR4 optical module. One port of the switch has two 800G OSFP DR8 optical module. Two single mode fibers are used to connecting one 800G module with two 400G modules. Each port of another 800G module is self-loop connected.

And now, the FEC mode of self-loop 800G module is presented as “Standard_RS-FEC - (544,514)”. The FEC modes of 800G module which connected with two 400G modules and 400G modules are both presented as “Ethernet_Consortium_LL_50G_RS_FEC_PLR -(272,257+1)

What makes the difference? And how can I configure the FEC mode to “Standard_RS-FEC - (544,514)”?

##########800G module with two 400G modules

[root@localhost ~]# mlxlink -d lid-3 -p 17/2 -c -e --show_histogram --rx_fec_histogram

Operational Info
----------------
State : Active
Physical state : LinkUp
Speed : IB-NDR
Width : 4x
FEC : Ethernet_Consortium_LL_50G_RS_FEC_PLR -(272,257+1)
Loopback Mode  : No Loopback
Auto Negotiation : ON

Supported Info
--------------
Enabled Link Speed : 0x00000080 (NDR)
Supported Cable Speed : 0x00000080 (NDR)

Troubleshooting Info
--------------------
Status Opcode  : 0
Group Opcode  : N/A
Recommendation : No issue was observed

Tool Information
----------------
Firmware Version                   : 31.2012.2200
amBER Version                      : 2.22
MFT Version                        : mft 4.26.1-3

Physical Counters and BER Info
------------------------------
Time Since Last Clear [Min] : 1344.4
Symbol Errors  : 0
Symbol BER  : 15E-255
Effective Physical Errors : 8
Effective Physical BER  : 2E-16
Raw Physical Errors Per Lane       : 83292107,2332348,4874171,4075639
Raw Physical BER   : 2E-9
Link Down Counter  : 6
Link Error Recovery Counter : 0

EYE Opening Info
----------------
FOM Mode  : SLRG_FOM_MODE_EYEO
Lane  :      0,     1,     2,     3
Initial FOM  :     75,    90,    86,    66
Last FOM  :     76,    94,    88,    72
Upper Grades :     78,    98,    86,    72
Mid Grades :    106,   107,    98,    96
Lower Grades :     84,    92,    86,    72

Histogram of FEC Errors
-----------------------
Header                          : Range 	Occurrences
Bin 0                              : [0]   	12604950840763
Bin 1                              : [1]   	94491311
Bin 2                              : [2]   	36808
Bin 3                              : [3]   	2307
Bin 4                              : [4]   	603
Bin 5                              : [5]   	2
Bin 6                              : [6]   	1
Bin 7                              : [7]   	1

##########self loop

[root@localhost ~]# mlxlink -d lid-3 -p 7/2 -c -e --show_histogram --rx_fec_histogram

Operational Info
----------------
State                               : Active
Physical state                : LinkUp
Speed                              : IB-NDR
Width                              : 4x
FEC                                  : Standard_RS-FEC - (544,514)
Loopback Mode            : No Loopback
Auto Negotiation          : ON

Supported Info
--------------
Enabled Link Speed                 : 0x00000080 (NDR)
Supported Cable Speed         : 0x00000080 (NDR)

Troubleshooting Info
--------------------
Status Opcode                      : 0
Group Opcode                       : N/A
Recommendation                 : No issue was observed

Tool Information
----------------
Firmware Version                : 31.2012.2200
amBER Version                   : 2.22
MFT Version                        : mft 4.26.1-3

Physical Counters and BER Info
------------------------------
Time Since Last Clear [Min]      : 1655.2
Symbol Errors                      : 0
Symbol BER                         : 15E-255
Effective Physical Errors          : 0
Effective Physical BER             : 15E-255
Raw Physical Errors Per Lane       : 932901108,55634833,131040331,100111746
Raw Physical BER                   : 3E-8
Link Down Counter                  : 5
Link Error Recovery Counter        : 0

EYE Opening Info
----------------
FOM Mode                           : SLRG_FOM_MODE_EYEO
Lane                               :      0,     1,     2,     3
Initial FOM                        :     89,    91,    87,    79
Last FOM                           :     92,    91,    86,    84
Upper Grades                       :     90,    87,    88,    84
Mid Grades                         :    114,   105,   102,   105
Lower Grades                       :     93,    88,    82,    83

Histogram of FEC Errors
-----------------------
Header                          : Range 	Occurrences
Bin 0                              : [0]   	7758328425319
Bin 1                              : [1]   	1217210427
Bin 2                              : [2]   	1234162
Bin 3                              : [3]   	2976
Bin 4                              : [4]   	111
Bin 5                              : [5]   	0
Bin 6                              : [6]   	1
Bin 7                              : [7]   	0
Bin 8                              : [8]   	0
Bin 9                              : [9]   	0
Bin 10                             : [10]   	0
Bin 11                             : [11]   	0
Bin 12                             : [12]   	0
Bin 13                             : [13]   	0
Bin 14                             : [14]   	0
Bin 15                             : [15]   	0

Hello Jliii,

Welcome, and thank you for posting your inquiry to the Mellanox community!

Changing the FEC mode for Infiniband devices is not supported. The FEC mode is configured during autonegotiation based on device capabilities and firmware.

All information on supported FEC algorithms and their implementation in NDR can be found in the Infiniband specification (Volume 2, Link/Phy Interface chapter, ‘Forward Error Correction’ section).

Best regards,
NVIDIA Enterprise Experience