Ib1: transmit timeout: latency 128 msecs

Jun 2 12:39:45 localhost kernel: NETDEV WATCHDOG: ib1 (mlx5_core): transmit queue 23 timed out
Jun 2 12:39:45 localhost kernel: WARNING: CPU: 35 PID: 1224744 at net/sched/sch_generic.c:482 dev_watchdog+0x29a/0x2b0
Jun 2 12:39:45 localhost kernel: Modules linked in: nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 redirfs(OE) mptcp_diag xsk_diag tcp_diag udp_diag raw_diag inet_diag unix_diag af_packet_diag netlink_diag wekafsio(OE) wekafsgw(OE) igb_uio(OE) sch_ingress nft_counter nft_compat nf_tables nfnetlink mpin_user(OE) squashfs loop uio_pci_generic(OE) uio vfio_pci vfio_virqfd vfio_iommu_type1 vfio irqbypass cuse fuse rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) mlx5_core(OE) mlxdevm(OE) ib_uverbs(OE) ib_core(OE) mlx_compat(OE) psample mlxfw(OE) pci_hyperv_intf bonding intel_rapl_msr ipmi_ssif intel_rapl_common vfat fat nfit libnvdimm intel_sdsi pmt_crashlog pmt_telemetry pmt_class x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl intel_cstate acpi_ipmi mei_me ipmi_si isst_if_mbox_pci isst_if_mmio joydev idxd intel_uncore mei isst_if_common ipmi_devintf idxd_bus pcspkr intel_vsec hpwdt hpilo wmi ipmi_msghandler
Jun 2 12:39:45 localhost kernel: acpi_tad acpi_power_meter binfmt_misc knem(OE) xpmem(OE) xfs libcrc32c raid1 mgag200 i2c_algo_bit drm_shmem_helper drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm crc32c_intel bnxt_en nvme tls nvme_core t10_pi [last unloaded: asm]
Jun 2 12:39:45 localhost kernel: CPU: 35 PID: 1224744 Comm: wekanode Kdump: loaded Tainted: G W OE --------- - - 4.18.0-477.10.1.el8_8.x86_64 #1
Jun 2 12:39:45 localhost kernel: Hardware name: HPE Alletra Storage Server 4110/Alletra Storage Server 4110, BIOS 2.34 10/04/2024
Jun 2 12:39:45 localhost kernel: RIP: 0010:dev_watchdog+0x29a/0x2b0
Jun 2 12:39:45 localhost kernel: Code: e4 eb 9a 4c 8b 3c 24 c6 05 13 2d 6d 01 01 4c 89 ff e8 5a 39 fa ff 89 d9 4c 89 fe 48 c7 c7 60 52 da 91 48 89 c2 e8 a3 80 86 ff <0f> 0b e9 75 ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00
Jun 2 12:39:45 localhost kernel: RSP: 0000:ff31a5a98cde8e78 EFLAGS: 00013286
Jun 2 12:39:45 localhost kernel: RAX: 0000000000000000 RBX: 0000000000000017 RCX: 0000000000000027
Jun 2 12:39:45 localhost kernel: RDX: 0000000000000027 RSI: 00000001003b78f4 RDI: ff2a8bbafffde690
Jun 2 12:39:45 localhost kernel: RBP: 0000000000000023 R08: 0000000000000000 R09: c0000001003b78f4
Jun 2 12:39:45 localhost kernel: R10: 00000000166c79e0 R11: ff31a5a98cde8c90 R12: ff2a8b7cae81845c
Jun 2 12:39:45 localhost kernel: R13: 0000000000000180 R14: ff2a8b7cae818480 R15: ff2a8b7cae818000
Jun 2 12:39:45 localhost kernel: FS: 00007f3d5f86ab80(0000) GS:ff2a8bbafffc0000(0000) knlGS:0000000000000000
Jun 2 12:39:45 localhost kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 2 12:39:45 localhost kernel: CR2: 000000000150ec48 CR3: 000000318db5e004 CR4: 0000000000771ee0
Jun 2 12:39:45 localhost kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jun 2 12:39:45 localhost kernel: DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
Jun 2 12:39:45 localhost kernel: PKRU: 55555554
Jun 2 12:39:45 localhost kernel: Call Trace:
Jun 2 12:39:45 localhost kernel:
Jun 2 12:39:45 localhost kernel: ? pfifo_fast_enqueue+0x150/0x150
Jun 2 12:39:45 localhost kernel: call_timer_fn+0x2e/0x130
Jun 2 12:39:45 localhost kernel: run_timer_softirq+0x1d8/0x410
Jun 2 12:39:45 localhost kernel: ? sched_clock+0x5/0x10
Jun 2 12:39:45 localhost kernel: __do_softirq+0xdc/0x2cf
Jun 2 12:39:45 localhost kernel: irq_exit_rcu+0xd5/0xe0
Jun 2 12:39:45 localhost kernel: irq_exit+0xa/0x10
Jun 2 12:39:45 localhost kernel: smp_apic_timer_interrupt+0x74/0x130
Jun 2 12:39:45 localhost kernel: apic_timer_interrupt+0xf/0x20
Jun 2 12:39:45 localhost kernel:
Jun 2 12:39:45 localhost kernel: RIP: 0033:0x96ba838
Jun 2 12:39:45 localhost kernel: Code: e1 45 0f b7 74 07 4a 66 45 85 f6 74 d5 49 8d 1c 07 48 83 c3 40 31 ed 66 66 2e 0f 1f 84 00 00 00 00 00 0f b7 ed 48 89 df 89 ee d3 e2 05 00 ff c5 66 41 39 ee 75 eb eb a9 48 8b 04 24 0f b7 00
Jun 2 12:39:45 localhost kernel: RSP: 002b:00007f3d435dcdb0 EFLAGS: 00003246 ORIG_RAX: ffffffffffffff13
Jun 2 12:39:45 localhost kernel: RAX: 0000000000000500 RBX: 000000000f509300 RCX: 0000000014070620
Jun 2 12:39:45 localhost kernel: RDX: 000000000f52fa50 RSI: 0000000000000000 RDI: 000000000f509300
Jun 2 12:39:45 localhost kernel: RBP: 0000000000000000 R08: 00000000000012e5 R09: 00007f3c9d802b40
Jun 2 12:39:45 localhost kernel: R10: 0000000104caf940 R11: 0000000000000000 R12: 0000000000000002
Jun 2 12:39:45 localhost kernel: R13: 0000000000000001 R14: 0000000000000002 R15: 000000000f508dc0
Jun 2 12:39:45 localhost kernel: —[ end trace 4b18370c6340f3c0 ]—
Jun 2 12:39:45 localhost kernel: ib1: transmit timeout: latency 128 msecs
Jun 2 12:39:45 localhost kernel: ib1: queue stopped 0, tx_head 0, tx_tail 0, global_tx_head 0, global_tx_tail 0
Jun 2 12:40:01 localhost systemd[1]: Starting system activity accounting tool…
Jun 2 12:40:03 localhost systemd[1]: sysstat-collect.service: Succeeded.
Jun 2 12:40:03 localhost systemd[1]: Started system activity accounting tool.
Jun 2 12:40:09 localhost NetworkManager[1672]: [1780371609.1593] device (ib1): state change: activated → unavailable (reason ‘carrier-changed’, sys-iface-state: ‘managed’)
Jun 2 12:40:09 localhost dbus-daemon[1669]: [system] Activating via systemd: service name=‘org.freedesktop.nm_dispatcher’ unit=‘dbus-org.freedesktop.nm-dispatcher.service’ requested by ‘:1.2’ (uid=0 pid=1672 comm="/usr/sbin/NetworkManager --no-daemon “)
Jun 2 12:40:09 localhost systemd[1]: Starting Network Manager Script Dispatcher Service…
Jun 2 12:40:09 localhost dbus-daemon[1669]: [system] Successfully activated service ‘org.freedesktop.nm_dispatcher’
Jun 2 12:40:09 localhost systemd[1]: Started Network Manager Script Dispatcher Service.
Jun 2 12:40:19 localhost systemd[1]: NetworkManager-dispatcher.service: Succeeded.
Jun 2 12:40:27 localhost NetworkManager[1672]: [1780371627.5921] device (ib1): carrier: link connected
Jun 2 12:40:27 localhost NetworkManager[1672]: [1780371627.5922] device (ib1): state change: unavailable → disconnected (reason ‘carrier-changed’, sys-iface-state: ‘managed’)
Jun 2 12:40:27 localhost NetworkManager[1672]: [1780371627.5928] policy: auto-activating connection ‘ib1’ (0042427c-f79a-456d-ad83-68d6b8a7b29d)
Jun 2 12:40:27 localhost NetworkManager[1672]: [1780371627.5934] device (ib1): Activation: starting connection ‘ib1’ (0042427c-f79a-456d-ad83-68d6b8a7b29d)
Jun 2 12:40:27 localhost NetworkManager[1672]: [1780371627.5934] device (ib1): state change: disconnected → prepare (reason ‘none’, sys-iface-state: ‘managed’)
Jun 2 12:40:33 localhost NetworkManager[1672]: [1780371633.0099] device (ib1): state change: prepare → config (reason ‘none’, sys-iface-state: ‘managed’)
Jun 2 12:40:33 localhost NetworkManager[1672]: [1780371633.0103] device (ib1): state change: config → ip-config (reason ‘none’, sys-iface-state: ‘managed’)
Jun 2 12:40:33 localhost kernel: IPv6: ADDRCONF(NETDEV_UP): ib1: link is not ready
Jun 2 12:40:33 localhost kernel: IPv6: ADDRCONF(NETDEV_UP): ib1: link is not ready
Jun 2 12:40:35 localhost NetworkManager[1672]: [1780371635.0118] device (ib1): state change: ip-config → ip-check (reason ‘none’, sys-iface-state: ‘managed’)
Jun 2 12:40:35 localhost dbus-daemon[1669]: [system] Activating via systemd: service name=‘org.freedesktop.nm_dispatcher’ unit=‘dbus-org.freedesktop.nm-dispatcher.service’ requested by ‘:1.2’ (uid=0 pid=1672 comm=”/usr/sbin/NetworkManager --no-daemon ")
Jun 2 12:40:35 localhost systemd[1]: Starting Network Manager Script Dispatcher Service…
Jun 2 12:40:35 localhost dbus-daemon[1669]: [system] Successfully activated service ‘org.freedesktop.nm_dispatcher’
Jun 2 12:40:35 localhost systemd[1]: Started Network Manager Script Dispatcher Service.
Jun 2 12:40:35 localhost NetworkManager[1672]: [1780371635.0203] device (ib1): state change: ip-check → secondaries (reason ‘none’, sys-iface-state: ‘managed’)
Jun 2 12:40:35 localhost NetworkManager[1672]: [1780371635.0204] device (ib1): state change: secondaries → activated (reason ‘none’, sys-iface-state: ‘managed’)
Jun 2 12:40:35 localhost NetworkManager[1672]: [1780371635.0206] device (ib1): Activation: successful, device activated.
Jun 2 12:40:37 localhost root[1226334]: Starting wekanode slot 6 in drives0 with core 20
Jun 2 12:40:37 localhost NetworkManager[1672]: [1780371637.2505] device (ib1): carrier: link connected
Jun 2 12:40:37 localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): ib1: link becomes ready

Host Driver Version … MLNX_OFED_LINUX-23.10-0.5.5.0 (OFED-23.10-0.5.5): 4.18.0-477.10.1.el8_8.x86_64

Hello,

An issue occurred on the IB network on June 2.

The system is running Red Hat 8.8. After the log shown above was generated, all IB links were disconnected, resulting in a service interruption.

From the HPE iLO perspective, there were no indications of a physical link-down event, and no related logs were found on the IB switch. Therefore, we believe that no actual physical link failure occurred and that the issue likely originated at the OS level.

We suspect that the service interruption was caused by the OS resetting the HCA card.

We would like to understand why the error “ib1: transmit timeout: latency 128 msecs” occurred.

Could you confirm whether this issue is related to the kernel, HCA firmware, or driver?

Hi,

According to the log you provided, the link became unavailable at that time.

So I believe there is a link issue.

Thanks,

Suo

Jun 2 12:40:09 localhost NetworkManager[1672]: [1780371609.1593] device (ib1): state change: activated → unavailable (reason ‘carrier-changed’, sys-iface-state: ‘managed’)

In practice, there was no actual IB link down event.

There were no hardware events reported by either the IB switches or iLO.
Instead, it appears that the watchdog detected an IB hang condition and reset the driver or firmware.

We would like to verify whether there was any known issue with the firmware that could have caused this behavior.

Hi Donghyun,

In such case, I recommend you to open a support ticket in enterprise support portal.

Thanks,

Suo