How to configure MCX354A-FCBT Mellanox InfiniBand speed at 56Gbps ?

Can someone help me to configure my MCX354A-FCBT Mellanox InfiniBand speed at 56Gbps. I have a MCX354A-FCBT Mellanox configured for InfiniBand but the speed remains at 40Gbps (All the components can speed at 56Gbps (card/Switch/Cable)! Thank a lot for your help. Here is my configuration:

Operating System

Fedora release 24 (Twenty Four)

kernel 4.11.12-100.fc24.x86_64

Mellanox card

MCX354A-FCBT

[root@aigle ~]# mlxconfig -d /dev/mst/mt4099_pci_cr0 q

Device #1:


Device type: ConnectX3

PCI device: /dev/mst/mt4099_pci_cr0

Configurations: Next Boot

SRIOV_EN True(1)

NUM_OF_VFS 8

LINK_TYPE_P1 VPI(3)

LINK_TYPE_P2 VPI(3)

LOG_BAR_SIZE 3

BOOT_PKEY_P1 0

BOOT_PKEY_P2 0

BOOT_OPTION_ROM_EN_P1 True(1)

BOOT_VLAN_EN_P1 False(0)

BOOT_RETRY_CNT_P1 0

LEGACY_BOOT_PROTOCOL_P1 PXE(1)

BOOT_VLAN_P1 1

BOOT_OPTION_ROM_EN_P2 True(1)

BOOT_VLAN_EN_P2 False(0)

BOOT_RETRY_CNT_P2 0

LEGACY_BOOT_PROTOCOL_P2 PXE(1)

BOOT_VLAN_P2 1

IP_VER_P1 IPv4(0)

IP_VER_P2 IPv4(0)

[root@aigle ~]# mlxfwmanager --query

Querying Mellanox devices firmware …

Device #1:


Device Type: ConnectX3

Part Number: MCX354A-FCB_A2-A5

Description: ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6

PSID: MT_1090120019

PCI Device Name: /dev/mst/mt4099_pci_cr1

Port1 GUID: f45214030027f751

Port2 GUID: f45214030027f752

Versions: Current Available

FW 2.42.5000 N/A

PXE 3.4.0752 N/A

Status: No matching image found

continued…

Device #2:


Device Type: ConnectX3

Part Number: MCX354A-FCB_A2-A5

Description: ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6

PSID: MT_1090120019

PCI Device Name: /dev/mst/mt4099_pci_cr0

Port1 GUID: 0002c9030032e311

Port2 GUID: 0002c9030032e312

Versions: Current Available

FW 2.42.5000 N/A

PXE 3.4.0752 N/A

Status: No matching image found

[root@aigle ~]# ibstat

CA ‘mlx4_0’

CA type: MT4099

Number of ports: 2

Firmware version: 2.42.5000

Hardware version: 1

Node GUID: 0x0002c9030032e310

System image GUID: 0x0002c9030032e313

Port 1:

State: Active

Physical state: LinkUp

Rate: 40 (FDR10)

Base lid: 6

LMC: 0

SM lid: 1

Capability mask: 0x02514868

Port GUID: 0x0002c9030032e311

Link layer: InfiniBand

Port 2:

State: Active

Physical state: LinkUp

Rate: 40 (FDR10)

Base lid: 7

LMC: 0

SM lid: 1

Capability mask: 0x02514868

Port GUID: 0x0002c9030032e312

Link layer: InfiniBand

CA ‘mlx4_1’

CA type: MT4099

Number of ports: 2

Firmware version: 2.42.5000

Hardware version: 1

Node GUID: 0xf45214030027f750

System image GUID: 0xf45214030027f753

Port 1:

State: Active

Physical state: LinkUp

Rate: 40 (FDR10)

Base lid: 8

LMC: 0

SM lid: 1

Capability mask: 0x02514868

Port GUID: 0xf45214030027f751

Link layer: InfiniBand

Port 2:

State: Active

Physical state: LinkUp

continued…

Rate: 40 (FDR10)

Base lid: 9

LMC: 0

SM lid: 1

Capability mask: 0x02514868

Port GUID: 0xf45214030027f752

Link layer: InfiniBand

[root@aigle ~]# ibv_devinfo -v

hca_id: mlx4_0

transport: InfiniBand (0)

fw_ver: 2.42.5000

node_guid: 0002:c903:0032:e310

sys_image_guid: 0002:c903:0032:e313

vendor_id: 0x02c9

vendor_part_id: 4099

hw_ver: 0x1

board_id: MT_1090120019

phys_port_cnt: 2

max_mr_size: 0xffffffffffffffff

page_size_cap: 0xfffffe00

max_qp: 393144

max_qp_wr: 16351

device_cap_flags: 0x057e9c76

BAD_PKEY_CNTR

BAD_QKEY_CNTR

AUTO_PATH_MIG

CHANGE_PHY_PORT

UD_AV_PORT_ENFORCE

PORT_ACTIVE_EVENT

SYS_IMAGE_GUID

RC_RNR_NAK_GEN

XRC

Unknown flags: 0x056e8000

device_cap_exp_flags: 0x5000401600000000

EXP_DEVICE_QPG

EXP_UD_RSS

EXP_CROSS_CHANNEL

EXP_MR_ALLOCATE

EXT_ATOMICS

EXP_MASKED_ATOMICS

max_sge: 32

max_sge_rd: 30

max_cq: 65408

max_cqe: 4194303

max_mr: 524032

max_pd: 32764

max_qp_rd_atom: 16

max_ee_rd_atom: 0

max_res_rd_atom: 6290304

max_qp_init_rd_atom: 128

max_ee_init_rd_atom: 0

atomic_cap: ATOMIC_HCA (1)

log atomic arg sizes (mask) 0x8

masked_log_atomic_arg_sizes (mask) 0x8

masked_log_atomic_arg_sizes_network_endianness (mask) 0x0

max fetch and add bit boundary 64

log max atomic inline 3

max_ee: 0

max_rdd: 0

max_mw: 0

max_raw_ipv6_qp: 0

max_raw_ethy_qp: 0

max_mcast_grp: 131072

max_mcast_qp_attach: 244

max_total_mcast_qp_attach: 31981568

continued…

max_ah: 2147483647

max_fmr: 0

max_srq: 65472

max_srq_wr: 16383

max_srq_sge: 31

max_pkeys: 128

local_ca_ack_delay: 15

hca_core_clock: 427000

max_klm_list_size: 0

max_send_wqe_inline_klms: 0

max_umr_recursion_depth: 0

max_umr_stride_dimension: 0

general_odp_caps:

max_size: 0x0

rc_odp_caps:

NO SUPPORT

uc_odp_caps:

NO SUPPORT

ud_odp_caps:

NO SUPPORT

dc_odp_caps:

NO SUPPORT

xrc_odp_caps:

NO SUPPORT

raw_eth_odp_caps:

NO SUPPORT

max_dct: 0

max_device_ctx: 1016

Multi-Packet RQ is not supported

rx_pad_end_addr_align: 0

tso_caps:

max_tso: 0

packet_pacing_caps:

qp_rate_limit_min: 0kbps

qp_rate_limit_max: 0kbps

ooo_caps:

ooo_rc_caps = 0x0

ooo_xrc_caps = 0x0

ooo_dc_caps = 0x0

ooo_ud_caps = 0x0

sw_parsing_caps:

supported_qp:

tag matching not supported

Device ports:

port: 1

state: PORT_ACTIVE (4)

max_mtu: 4096 (5)

active_mtu: 4096 (5)

sm_lid: 1

port_lid: 6

port_lmc: 0x00

link_layer: InfiniBand

max_msg_sz: 0x40000000

port_cap_flags: 0x02514868

max_vl_num: 8 (4)

bad_pkey_cntr: 0x0

qkey_viol_cntr: 0x0

sm_sl: 0

pkey_tbl_len: 128

gid_tbl_len: 128

subnet_timeout: 18

init_type_reply: 0

active_width: 4X (2)

active_speed: 10.0 Gbps (8)

phys_state: LINK_UP (5)

GID[ 0]: fe80:0000:0000:0000:0002:c903:0032:e311

port: 2

state: PORT_ACTIVE (4)

max_mtu: 4096 (5)

active_mtu: 4096 (5)

sm_lid: 1

port_lid: 7

port_lmc: 0x00

link_layer: InfiniBand

max_msg_sz: 0x40000000

port_cap_flags: 0x02514868

max_vl_num: 8 (4)

bad_pkey_cntr: 0x0

qkey_viol_cntr: 0x0

sm_sl: 0

pkey_tbl_len: 128

gid_tbl_len: 128

subnet_timeout: 18

init_type_reply: 0

active_width: 4X (2)

active_speed: 10.0 Gbps (8)

phys_state: LINK_UP (5)

GID[ 0]: fe80:0000:0000:0000:0002:c903:0032:e312

continued…

Switch

Part Info


Type: SX6036

S/N: IL23190198

P/N: 712498-B21

Chassis system GUID: 00:02:C9:03:00:AC:6C:20

Asic FW version: 9.4.3580

LID: 1

Node GUID: 00:02:C9:03:00:AC:6C

Installed MLNX-OS Images


Partition 1 - Active Image (partition of next boot)

PPC_M460EX 3.6.4006 2017-07-03 16:17:35 ppc

Partition 2

PPC_M460EX 3.6.3004 2017-02-05 17:31:50 ppc

Port Info


Port number : 1

Port type : IB

IB Subnet : infiniband-default

Port description :

Logical port state : Active

Physical port state : LinkUp

Current line rate : 40.0 Gbps

Supported speeds : sdr, ddr, qdr, fdr10

Speed : fdr10

Supported widths : 1X, 4X

Width : 4X

Max supported MTUs : 4096

MTU : 4096

VL capabilities : VL0 - VL7

Operational VLs : VL0 - VL7

Supported LLR speeds : FDR10, FDR

LLR Status : Active

Transceiver Information

Identifier : QSFP+

Cable/ Module type : Passive copper, unequalized

Infiniband speeds : SDR , DDR , QDR , FDR

Vendor : Mellanox

Cable length : 2 m

Part number : MC2207130-002

Revision : A3

Serial number : MT1710VS05863

Subnet Manager (SM) Status


SM Status

Local SM running 1 hour 26 minutes 52 seconds

SM Priority 7 State running

Failures 0 Autostart true

Routing Engine Used minhop

SM version OpenSM4.7.0.MLNX20170511.3016205

reply…

Hi Jae-Hoon, Hi aviap

I’m sorry to be late. Thank a lot for your answers. Now, it seems clear that I can not use FDR because my switch is not compatible. I still tested to force FDR but “% the requested speed value is not allowed for this port”:

Mellanox-SX6036 [standalone: master] # configure terminal

Mellanox-SX6036 [standalone: master] (config) #

Mellanox-SX6036 [standalone: master] (config) # show running-config

##llanox-SX6036 [standalone: master] (config) # show running-config

Running database “initial”

Generated at 2017/10/01 20:10:51 +0200

Hostname: Mellanox-SX6036

Running-config temporary prefix mode setting

no cli default prefix-modes enable

Network interface configuration

interface ib 1/1 mtu 4K

interface ib 1/1 op-vls 8

interface ib 1/1 speed sdr ddr qdr fdr10

interface ib 1/1 width 5

no interface mgmt0 dhcp

interface mgmt0 ip address 192.168.0.252 /24

Network interface IPv6 configuration

no interface mgmt0 ipv6 address autoconfig default

no interface mgmt0 ipv6 enable

Mellanox-SX6036 [standalone: master] (config) # show interface ib status

reply…

Interface Description IB Subnet Speed Current line rate Logical port state Physical port state


IB1/1 infiniband-default fdr10 40.0 Gbps Active LinkUp

IB1/2 infiniband-default fdr10 40.0 Gbps Active LinkUp

IB1/3 infiniband-default fdr10 40.0 Gbps Active LinkUp

IB1/4 infiniband-default fdr10 40.0 Gbps Active LinkUp

reply…

IB1/5 infiniband-default - - Down Polling

IB1/6 infiniband-default - - Down Polling

IB1/7 infiniband-default - - Down Polling

IB1/8 infiniband-default - - Down Polling

IB1/9 infiniband-default - - Down Polling

IB1/10 infiniband-default - - Down Polling

IB1/11 infiniband-default - - Down Polling

IB1/12 infiniband-default - - Down Polling

IB1/13 infiniband-default - - Down Polling

IB1/14 infiniband-default - - Down Polling

IB1/15 infiniband-default - - Down Polling

IB1/16 infiniband-default - - Down Polling

IB1/17 infiniband-default - - Down Polling

reply…

IB1/18 infiniband-default - - Down Polling

IB1/19 infiniband-default - - Down Polling

IB1/20 infiniband-default - - Down Polling

IB1/21 infiniband-default - - Down Polling

IB1/22 infiniband-default - - Down Polling

IB1/23 infiniband-default - - Down Polling

IB1/24 infiniband-default - - Down Polling

IB1/25 infiniband-default - - Down Polling

IB1/26 infiniband-default - - Down Polling

IB1/27 infiniband-default - - Down Polling

reply…

IB1/28 infiniband-default - - Down Polling

IB1/29 infiniband-default - - Down Polling

IB1/30 infiniband-default - - Down Polling

IB1/31 infiniband-default - - Down Polling

IB1/32 infiniband-default - - Down Polling

IB1/33 infiniband-default - - Down Polling

IB1/34 infiniband-default - - Down Polling

IB1/35 infiniband-default - - Down Polling

IB1/36 infiniband-default - - Down Polling

reply…

Mellanox-SX6036 [standalone: master] (config) # interface ib 1/1 speed sdr ddr qdr fdr10 fdr force

% the requested speed value is not allowed for this port

Mellanox-SX6036 [standalone: master] (config) #

No matter if the switch is only FDR10 and not FDR it is just for educational purpose, I’ m learning InfiniBand technology. Thank a lot.

Best Regard

HPE Part Number 712498-B21 is SX6036T FDR10 switch.

Therefore in this case user can’t use FDR14 56Gb speed in fabric.

HPE Support document - HPE Support Center https://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-a00007395en_us

Why did you give a incorrect information to community users?

Best Regard,

Jae-Hoon Choi

Both QDR & FDR10 can support 40Gb speed, but there is a deffrent factor like belows.

QDR (PCIe 2.0, 8/10bit encording & only 10Gb ETH support) vs FDR10 (PCIe 3.0, 64/66bit encording & only 10Gb ETH support)

Therefor real performance will be diffrent in maximum speed in IB mode.

BR,

Jae-Hoon Choi

Hi!

You have SX6036T FDR10 switch.

Your switch’s port support up to FDR10 (40Gb, 64/66bit encording).

If you want FDR14(56Gb) speed, you must change your switch to SX6036F or SX6036G.

Best Regard,

Jae-Hoon Choi

From the output prints you have presented, it looks like your SX6036 switch should be good & supports 56Gb (FDR)

  • cables & nics are also fine and capable of fdr

  • cables present: Infiniband speeds : SDR , DDR , QDR , FDR

  • switch presents:

Supported LLR speeds : FDR10, FDR - which indicates you can set it to 56Gb

Supported speeds : sdr, ddr, qdr, fdr10 - here you see that fdr is missing

so, in my view - all you have to do is to run the following command on the switch that will add FDR on “supported speeds” list

run on CLI commnd:

(config) # interface ib <interface#> speed sdr ddr qdr fdr10 fdr force

(config) # configure write (to save the changes)

you should now see: Supported speeds : sdr, ddr, qdr, fdr10, fdr

this should enable you to uses 56Gb on the switch and on the nics as well

Hi Jae-Hoon, Hi aviap

I’m sorry to be late. Thank a lot for your answers. Now, it seems clear that I can not use FDR because my switch is not compatible. I still tested to force FDR but “% the requested speed value is not allowed for this port”:

Mellanox-SX6036 [standalone: master] # configure terminal

Mellanox-SX6036 [standalone: master] (config) #

Mellanox-SX6036 [standalone: master] (config) # show running-config

##llanox-SX6036 [standalone: master] (config) # show running-config

Running database “initial”

Generated at 2017/10/01 20:10:51 +0200

Hostname: Mellanox-SX6036

Running-config temporary prefix mode setting

no cli default prefix-modes enable

Network interface configuration

interface ib 1/1 mtu 4K

interface ib 1/1 op-vls 8

interface ib 1/1 speed sdr ddr qdr fdr10

interface ib 1/1 width 5

no interface mgmt0 dhcp

interface mgmt0 ip address 192.168.0.252 /24

Network interface IPv6 configuration

no interface mgmt0 ipv6 address autoconfig default

no interface mgmt0 ipv6 enable

Mellanox-SX6036 [standalone: master] (config) # show interface ib status

Interface Description IB Subnet Speed Current line rate Logical port state Physical port state


IB1/1 infiniband-default fdr10 40.0 Gbps Active LinkUp

IB1/2 infiniband-default fdr10 40.0 Gbps Active LinkUp

IB1/3 infiniband-default fdr10 40.0 Gbps Active LinkUp

IB1/4 infiniband-default fdr10 40.0 Gbps Active LinkUp

IB1/5 infiniband-default - - Down Polling

IB1/6 infiniband-default - - Down Polling

IB1/7 infiniband-default - - Down Polling

IB1/8 infiniband-default - - Down Polling

IB1/9 infiniband-default - - Down Polling

IB1/10 infiniband-default - - Down Polling

IB1/11 infiniband-default - - Down Polling

IB1/12 infiniband-default - - Down Polling

IB1/13 infiniband-default - - Down Polling

IB1/14 infiniband-default - - Down Polling

IB1/15 infiniband-default - - Down Polling

IB1/16 infiniband-default - - Down Polling

IB1/17 infiniband-default - - Down Polling

IB1/18 infiniband-default - - Down Polling

IB1/19 infiniband-default - - Down Polling

IB1/20 infiniband-default - - Down Polling

IB1/21 infiniband-default - - Down Polling

IB1/22 infiniband-default - - Down Polling

IB1/23 infiniband-default - - Down Polling

IB1/24 infiniband-default - - Down Polling

IB1/25 infiniband-default - - Down Polling

IB1/26 infiniband-default - - Down Polling

IB1/27 infiniband-default - - Down Polling

IB1/28 infiniband-default - - Down Polling

IB1/29 infiniband-default - - Down Polling

IB1/30 infiniband-default - - Down Polling

IB1/31 infiniband-default - - Down Polling

IB1/32 infiniband-default - - Down Polling

IB1/33 infiniband-default - - Down Polling

IB1/34 infiniband-default - - Down Polling

IB1/35 infiniband-default - - Down Polling

IB1/36 infiniband-default - - Down Polling

Mellanox-SX6036 [standalone: master] (config) # interface ib 1/1 speed sdr ddr qdr fdr10 fdr force

% the requested speed value is not allowed for this port

Mellanox-SX6036 [standalone: master] (config) #

No matter if the switch is only FDR10 and not FDR it is just for educational purpose, I’ m learning InfiniBand technology. Thank a lot.

Best Regard