Rate sdr instead hdr

Infrastructure & Networking Software And Drivers Mellanox OFED
1

I have a system with Infiniband HDR 200 cards and Mellanox QM8700 switches installed. After configuring everything and also trying to force the speed to hdr I always get speed sdr link. What can I do?
I carried out tests both with sm installed on the switch and externally with the usual partition. No improvement.
These are the details of my hardware:

[root@md15 ~]# ibstat
CA ‘mlx5_0’
CA type: MT4123
Number of ports: 1
Firmware version: 20.35.1012
Hardware version: 0
Node GUID: 0x946dae03005d2ae6
System image GUID: 0x946dae03005d2ae6
Port 1:
State: Active
Physical state: LinkUp
Rate: 10
Base lid: 20
LMC: 0
SM lid: 2
Capability mask: 0xa651e848
Port GUID: 0x946dae03005d2ae6
Link layer: InfiniBand
[root@md15 ~]# ibstat
ibstat ibstatus
[root@md15 ~]# ibstatus
Infiniband device ‘mlx5_0’ port 1 status:
default gid: fe80:0000:0000:0000:946d:ae03:005d:2ae6
base lid: 0x14
sm lid: 0x1
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 10 Gb/sec (4X SDR)
link_layer: InfiniBand

Partition:

Partition Name PKey Default Member IPoIB Rate MTU SL Scope
Default 0x7FFF full Yes 200 4K default default

QM8700:

QM8700 Software Version: X86_64 3.10.4006 2022-11-09 14:28:37 x86_64
QM8700 Firmware Version: 27.2010.4034
MLNX-OFED: 23.07-0.5.0.0

Do you have any suggestions for solving the problem and obtaining hdr connections?
Thank you
Matteo

2

Some details from switch port:
Port Info
Port number :

15

Port type :

IB

IB Subnet :

infiniband-default

Port description :

Logical port state :

Active

Physical port state :

LinkUp

Current line rate :

10.0 Gbps

Supported speeds :

sdr, hdr

Speed :

sdr

Supported widths :

1X, 2X, 4X

Width :

4X

Max supported MTUs :

4096

MTU :

4096

VL admin capabilities :

VL0 - VL7

Operational VLs :

VL0 - VL3

Threshold Level :

N/A

Port CountersClear Port 15 Counters
RX bytes :

75256

RX packets :

417

RX errors :

0

Symbol errors :

0

VL15 dropped packets :

0

TX bytes :

76952

TX packets :

441

TX wait :

0

TX discarded packets :

0

Transceiver Information
Identifier :

QSFP56

Cable/ Module type :

Passive copper, unequalized

Infiniband speeds :

unknown

Vendor :

Mellanox

Cable length :

2 m

Part number :

MCP1650-V002E26

Revision :

A3

Serial number :

MT2248VB82669

3

Please use Infiniband cables.
the cable in USE PN: MCP1650-V002E26. is a 200GbE cable and not an HDR cable.

4

What cable are y on using.
There’s no need to force any speed it should work in accordance to the cable and raise HDR.

Please use mlxlink with ‘m e c’ flags and attach the output for both sides. You can use device as lid- for inband access.

5

mellanox-qm8700 [standalone: master] (config) # show interfaces ib 1/01 transceiver
IB1/1 state:
identifier : QSFP56
cable/module type : Passive copper, unequalized
infiniband speeds : unknown
vendor : Mellanox
cable length : 2m
part number : MCP1650-V002E26
revision : A3
serial number : MT2248VB82448

6

mlxlink -d /dev/mst/mt4123_pciconf0 -p 1 -m -c

Operational Info

State : Active
Physical state : LinkUp
Speed : IB-SDR
Width : 4x
FEC : No FEC
Loopback Mode : No Loopback
Auto Negotiation : ON

Supported Info

Enabled Link Speed : 0x00000001 (SDR)
Supported Cable Speed : 0x00000001 (SDR)

Troubleshooting Info

Status Opcode : 0
Group Opcode : N/A
Recommendation : No issue was observed

Tool Information

Firmware Version : 20.35.1012
amBER Version : 2.17
MFT Version : mft 4.25.0-62

Module Info

Identifier : QSFP28
Compliance : N/A
Cable Technology : Copper cable unequalized
Cable Type : Passive copper cable
OUI : Mellanox
Vendor Name : Mellanox
Vendor Part Number : MCP1650-V002E26
Vendor Serial Number : MT2248VB82483
Rev : A3
Wavelength [nm] : N/A
Transfer Distance [m] : 2
Attenuation (5g,7g,12g) [dB] : 5,6,10
FW Version : N/A
Digital Diagnostic Monitoring : No
Power Class : N/A
CDR RX : N/A
CDR TX : N/A
LOS Alarm : N/A
Temperature [C] : N/A
Voltage [mV] : N/A
Bias Current [mA] : N/A
Rx Power Current [dBm] : N/A
Tx Power Current [dBm] : N/A
SNR Media Lanes [dB] : N/A
SNR Host Lanes [dB] : N/A
IB Cable Width : 1x,2x,4x
Memory Map Revision : 8
Linear Direct Drive : 0
Cable Breakout : Channels implemented [1,2,3,4]/Cable with single far-end with 4 channels implemented, or separable module with a 4-channel connector
SMF Length : N/A
MAX Power : 0
Cable Rx AMP : N/A
Cable Rx Emphasis : N/A
Cable Rx Post Emphasis : N/A
Cable Tx Equalization : N/A
Wavelength Tolerance : N/A
Module State : N/A
DataPath state [per lane] : N/A
Rx Output Valid [per lane] : N/A
Nominal bit rate : 26.500Gb/s
Rx Power Type : OMA
Manufacturing Date : 16_11_22
Active Set Host Compliance Code : N/A
Active Set Media Compliance Code : N/A
Error Code Response : N/A
Module FW Fault : N/A
DataPath FW Fault : N/A
Tx Fault [per lane] : N/A
Tx LOS [per lane] : N/A
Tx CDR LOL [per lane] : N/A
Rx LOS [per lane] : N/A
Rx CDR LOL [per lane] : N/A
Tx Adaptive EQ Fault [per lane] : N/A

Physical Counters and BER Info

Time Since Last Clear [Min] : 207.3
Symbol Errors : 0
Symbol BER : 15E-255
Effective Physical Errors : 0
Effective Physical BER : 15E-255
Raw Physical Errors Per Lane : 0,0,0,0
Raw Physical BER : 15E-255
Link Down Counter : 6
Link Error Recovery Counter : 0

7

Ok. But why the connection rate is 10Gbit/s instead 200Gbit/s? What is the differences between MCP1650-H and MCP1650-V?
Thank you

8

when there is no IB support for a cable - the devices on both ends will only try to Link up at SDR (2.5x4)
MCP1650 are HDR DAC cables