Issue Description:
Two SB7890 switches are connected via MMA1L30-CM optical modules, but there is a significant bandwidth reduction. One optical module is connected to port 1 of the first switch, and the other optical module is connected to port 3 of the second switch. Other servers are connected to the switches using 56G IB network cards.
When using the ib_write_bw command to test bandwidth on servers connected to a single switch, the results show around 50G, which is close to the expected performance. However, when testing bandwidth between servers on opposite switches, the throughput drops to around 28G.
The iblinkinfo command shows that the optical modules are correctly detected and operating in 4*25.78125G EDR mode.
Question: Why is there a significant bandwidth reduction when testing between the two switches, despite the correct detection of the EDR link on the optical modules? How can we solve it？
Image type: FS3
FW ISSU Version: 1
FW Version: 15.2010.4516
FW Release Date: 13.6.2024
Description: UID GuidsNumber
Base GUID: 043f720300031cd0 32
Base MAC: 043f72031cd0 32
Image VSD: N/A
Device VSD: N/A
PSID: MT_2640110032
Security Attributes: N/A
[root@g ~]# mlxlink -d lid-8 -p 3
Operational Info
State : Active
Physical state : LinkUp
Speed : IB-EDR
Width : 4x
FEC : Standard LL RS-FEC - RS(271,257)
Loopback Mode : No Loopback
Auto Negotiation : ON
Supported Info
Enabled Link Speed : 0x00000020 (EDR)
Supported Cable Speed : 0x00000020 (EDR)
Troubleshooting Info
Status Opcode : 33
Group Opcode : PHY FW
Recommendation :