SN2100/SN2700 100G 10G Rate Conversion

[shepard.siegel]

We have four SN2100 and SN2700 switches (all ONYX 3.10.4100) happily interconnecting themselves, CX5/6 NICs, and dozens of FPGA boards. The links are ALL 100 GbE, either over DAC or AOC. Fine. Everything works.

We’ve been frustrated and unsuccessful at getting any port on any SN2100/2700 to make link at 10G. We’re sure the switch and media are good, because the 10G switch makes 10G link fine with the CX5/6 or FPGA.

Since the switches “see” the cable I2c, but dont make 10G link, we suspect that for some reason they are not allowed. We are using a MLNX adapter to plug a SFP in into the QSFP on the switch. And with AOC or DAC, no link! But pull it out of the switch and plug to a CX5/CX6, 10G link as expected.

Any ideas on what to look at or try here so these fine100G switches can have some 10G love?

[dwaxman]

is the port explicitly configured to 10g on the switch?

Let’s see the connection status when failing to connect. Please run the following on the switch CLI:

There’s an mlxlink command (might require fae shell (I.e. #fae mlxlink -d <mst_dev> -p <#port> -m)

For getting the mst device use
#fae mst status

[shepard.siegel]

Sure! Here’s the status of port 15…

ar-sn2100-03 [standalone: master] (config fae) # mlxlink -d /dev/mst/mt52100_pci_cr0 -p 15 -m

Operational Info
----------------
State                           : Disable
Physical state                  : ETH_AN_FSM_ENABLE
Speed                           : N/A
Width                           : N/A
FEC                             : N/A
Loopback Mode                   : N/A
Auto Negotiation                : ON

Supported Info
--------------
Enabled Link Speed              : 0x00b00000 (100G)
Supported Cable Speed           : 0x00001001 (10G,1G)

Troubleshooting Info
--------------------
Status Opcode                   : 16
Group Opcode                    : PHY FW
Recommendation                  : Cable speed not enabled

Tool Information
----------------
Firmware Version                : 13.2010.4120
MFT Version                     : mft 4.22.1-11

Module Info
-----------
Identifier                      : QSA (QSFP->SFP)
Compliance                      : N/A
Cable Technology                : Passive
Cable Type                      : Passive copper cable
OUI                             : Other
Vendor Name                     : Amphenol
Vendor Part Number              : 624380003
Vendor Serial Number            : APF14030030RJC
Rev                             : A
Wavelength [nm]                 : N/A
Transfer Distance [m]           : 3
Attenuation (5g,7g,12g) [dB]    : N/A
FW Version                      : N/A
Digital Diagnostic Monitoring   : No
Power Class                     : N/A
CDR RX                          : N/A
CDR TX                          : N/A
LOS Alarm                       : N/A
Temperature [C]                 : N/A
Voltage [mV]                    : N/A
Bias Current [mA]               : N/A
Rx Power Current [dBm]          : N/A
Tx Power Current [dBm]          : N/A

I went into the web gui to enable the 10G and disable the 100G bit and now from mlxlink I see…

Operational Info
----------------
State                           : Active
Physical state                  : LinkUp
Speed                           : 10GbE
Width                           : 1x
FEC                             : No FEC
Loopback Mode                   : No Loopback
Auto Negotiation                : ON

Supported Info
--------------
Enabled Link Speed              : 0x00007000 (10G)
Supported Cable Speed           : 0x00001001 (10G,1G)

Troubleshooting Info
--------------------
Status Opcode                   : 0
Group Opcode                    : N/A
Recommendation                  : No issue was observed

Link is up and we are moving packets.
Thanks so much for this help @dwaxman !
Much appreciated! -Shep

[dwaxman]

Nice! Glad this was resolved.