how to debug "hangs" related to infiniband adapter

Hi,

on some of our machines we are facing (sporadically but yet “too often”) hangs that seem to be related with the Infiniband adapters.

Meaning: if I issue one of the following commands, it will never return to the shell and won’t react to CTRL+C:

  • ibstat (this seems particularly bad)
  • ping $OTHER_HOSTS_IP_ASSIGNED_TO_INFINIBAND_DEVICE
  • ls /nfs/mount/on/server/connected/via/infiniband
  • sudo ifdown ib0 && sudo ifup ib0

Of course, after a reboot everything is fine again :-/

I am very new to this, so my troubleshooting skills are weak. I have listed some basic information below and would be grateful for

further guidance how to debug this issue.

Thanks,

Jonas

$ grep ib0 /var/log/syslog # this is around the time when the problem happened

Dec 5 10:13:29 heinzel60 kernel: [70988.535625] ib0: ipoib_cm_tx_destroy_rss: 7 not completed for QP: 0x257 force cleanup.

Dec 5 11:55:50 heinzel60 kernel: [77129.295892] ib0: timing out; 7 sends not completed

Dec 5 11:55:55 heinzel60 kernel: [77134.300207] ib0: timing out; 7 sends not completed

Dec 5 11:56:00 heinzel60 kernel: [77139.304520] ib0: timing out; 7 sends not completed

Dec 5 11:56:05 heinzel60 kernel: [77144.308840] ib0: timing out; 7 sends not completed

Dec 5 11:56:10 heinzel60 kernel: [77149.313159] ib0: timing out; 7 sends not completed

Dec 5 11:56:10 heinzel60 kernel: [77149.313795] ib0: ipoib_cm_tx_destroy_rss: 7 not completed for QP: 0x265 force cleanup.

Dec 5 12:08:40 heinzel60 kernel: [77899.392921] ib0: timing out; 7 sends not completed

Dec 5 12:08:45 heinzel60 kernel: [77904.397237] ib0: timing out; 7 sends not completed

Dec 5 12:08:50 heinzel60 kernel: [77909.401558] ib0: timing out; 7 sends not completed

Dec 5 12:08:55 heinzel60 kernel: [77914.405874] ib0: timing out; 7 sends not completed

Dec 5 12:09:00 heinzel60 kernel: [77919.410197] ib0: timing out; 7 sends not completed

[…]

$ lspci | grep Mellanox

01:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]

$ ibstat # as said above, this worked only after reboot

CA ‘mlx4_0’

CA type: MT4099

Number of ports: 1

Firmware version: 2.42.5000

Hardware version: 1

Node GUID: 0xec0d9a0300062a80

System image GUID: 0xec0d9a0300062a83

Port 1:

State: Active

Physical state: LinkUp

Rate: 56

Base lid: 52

LMC: 0

SM lid: 25

Capability mask: 0x02514868

Port GUID: 0xec0d9a0300062a81

Link layer: InfiniBand

$ lsmod | egrep ‘ib|mlx’

ib_ucm 20480 0

ib_uverbs 106496 2 ib_ucm,rdma_ucm

mlx5_fpga_tools 16384 0

mlx5_ib 266240 0

mlx5_core 782336 2 mlx5_ib,mlx5_fpga_tools

mlxfw 20480 1 mlx5_core

ib_iser 49152 0

rdma_cm 61440 2 ib_iser,rdma_ucm

libiscsi_tcp 24576 1 iscsi_tcp

libiscsi 53248 3 libiscsi_tcp,iscsi_tcp,ib_iser

scsi_transport_iscsi 98304 4 iscsi_tcp,ib_iser,libiscsi

ib_ipoib 163840 0

ib_cm 53248 3 rdma_cm,ib_ucm,ib_ipoib

ib_umad 24576 0

mlx4_ib 208896 0

ib_core 282624 11 rdma_cm,ib_cm,iw_cm,mlx4_ib,mlx5_ib,ib_ucm,ib_iser,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib

libcrc32c 16384 1 raid456

mlx4_en 135168 0

vxlan 49152 2 mlx4_en,mlx5_core

ptp 20480 3 igb,mlx4_en,mlx5_core

libahci 32768 1 ahci

mlx4_core 348160 2 mlx4_en,mlx4_ib

mlx_compat 24576 16 rdma_cm,ib_cm,iw_cm,mlx4_en,mlx4_ib,mlx5_ib,mlx5_fpga_tools,ib_ucm,ib_core,ib_iser,ib_umad,ib_uverbs,mlx4_core,mlx5_core,rdma_ucm,ib_ipoib

Hi Jonas,

This issue that is related to the driver that allocates the minimum amount of buffers for Rx/Tx queues.

The default values of TX and RX buffers are 256/128.

We recommend using higher values, for ConnectX-3 CM(Connected mode).

I suggest to enlarge the IPoIB Rx/Tx queue buffer size to 1k and let us know if the issue still happens ?

  • vi /etc/modprobe.d/ib_ipoib.conf

options ib_ipoib send_queue_size=1024 recv_queue_size=1024

  • restart openibd service :

/etc/init.d/openibd restart

In case the issue is still occur try to enlarge to higher value 2048 etc .

If after enlarging the value to the maximum still not solving the issue, open a support ticket at support@mellanox.com mailto:support@mellanox.com

Thanks

Samer

Thank you, Samer.

I will give it a try and follow up at support@mellanox.com mailto:support@mellanox.com in case I still face issues.

Thanks,

Jonas

Hi Jonas,

This issue that is related to the driver that allocates the minimum amount of buffers for Rx/Tx queues.

The default values of TX and RX buffers are 256/128.

We recommend using higher values, for ConnectX-3 CM(Connected mode).

I suggest to enlarge the IPoIB Rx/Tx queue buffer size to 1k and let us know if the issue still happens ?

  • vi /etc/modprobe.d/ib_ipoib.conf

options ib_ipoib send_queue_size=1024 recv_queue_size=1024

  • restart openibd service :

/etc/init.d/openibd restart

In case the issue is still occur try to enlarge to higher value 2048 etc .

If after enlarging the value to the maximum still not solving the issue, open a support ticket at support@mellanox.com mailto:support@mellanox.com

Thanks

Samer

Hi Samer , I have observered similar behavior . I was not able to ping, status of the Ib was showing active and up . I had to bringdown ib0 and bring up to resolve the issue (configured as ib bond with 2 interface) . After that it started pinging to other host. Is this issue related to above fix. Here is the details of my card and OS . Can you please help.

Dmesg output shows as below

4532888.484428] ib0: timing out; 4 sends not completed

[4532893.487573] ib0: timing out; 4 sends not completed

[4532898.493744] ib0: timing out; 4 sends not completed

[4532903.497944] ib0: timing out; 4 sends not completed

[4532903.497959] ib0: ipoib_cm_tx_destroy_rss: 4 not completed for QP: 0x247e force cleanup.

[4532911.750191] ib0: timing out; 8 sends not completed

[4532916.754382] ib0: timing out; 8 sends not completed

[4532921.756578] ib0: timing out; 8 sends not completed

Port status show as good.

hca_id: mlx4_0

transport: InfiniBand (0)

fw_ver: 2.6.100

node_guid: 0002:c903:0007:56f4

sys_image_guid: 0002:c903:0007:56f7

vendor_id: 0x02c9

vendor_part_id: 25418

hw_ver: 0xA0

board_id: HP_08B0000001

phys_port_cnt: 2

Device ports:

port: 1

state: PORT_ACTIVE (4)

max_mtu: 4096 (5)

active_mtu: 4096 (5)

sm_lid: 6

port_lid: 5

port_lmc: 0x00

link_layer: InfiniBand

port: 2

state: PORT_ACTIVE (4)

max_mtu: 4096 (5)

active_mtu: 4096 (5)

sm_lid: 14

port_lid: 15

port_lmc: 0x00

link_layer: InfiniBand

Adapter

09:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX VPI PCIe 2.0 2.5GT/s - IB DDR / 10GigE] (rev a0)

OS Kernel

3.10.0-327.el7.x86_64

Oracle Linux 7

Driver details

CA ‘mlx4_0’

CA type: MT25418

Number of ports: 2

Firmware version: 2.6.100

Hardware version: a0

Node GUID: 0x0002c903000756f4

System image GUID: 0x0002c903000756f7

Port 1:

State: Active

Physical state: LinkUp

Rate: 20

Base lid: 5

LMC: 0

SM lid: 6

Capability mask: 0x02510868

Port GUID: 0x0002c903000756f5

Link layer: InfiniBand

Port 2:

State: Active

Physical state: LinkUp

Rate: 20

Base lid: 15

LMC: 0

SM lid: 14

Capability mask: 0x02510868

Port GUID: 0x0002c903000756f6

Link layer: InfiniBand