Hello,
we are facing a strange issue in our 3 vcenter cluster (7.0.3), ESXI Version 7.0.3, 24585291.
The following errors occurs only on the esxi node that is currently identified as the HA-Primary.
Apart from the huge amount of erros inside the vmkernel.log, the main issue is that arp changes on some vms which are running on the current HA-Primary node wont work. The only way to fix that is to migrate to a none HA-Primary node.
cat /var/log/vmkernel.log | grep NMLX
2025-04-08T15:47:30.004Z cpu44:2098732)<NMLX_ERR> nmlx5_core: vmnic5: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:30.004Z cpu44:2098732)<NMLX_ERR> nmlx5_core: vmnic5: nmlx5_en_UpdateVportCounters - (nmlx5_core_en_main.c:1147) Failed to query vport counters (195887328)
2025-04-08T15:47:30.182Z cpu5:2098643)<NMLX_ERR> nmlx5_core: vmnic2: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:30.182Z cpu5:2098643)<NMLX_ERR> nmlx5_core: vmnic2: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:30.182Z cpu5:2098643)<NMLX_ERR> nmlx5_core: vmnic2: nmlx5_en_UpdateVportCounters - (nmlx5_core_en_main.c:1147) Failed to query vport counters (195887328)
2025-04-08T15:47:30.248Z cpu65:2098673)<NMLX_ERR> nmlx5_core: vmnic3: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:30.248Z cpu65:2098673)<NMLX_ERR> nmlx5_core: vmnic3: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:30.248Z cpu65:2098673)<NMLX_ERR> nmlx5_core: vmnic3: nmlx5_en_UpdateVportCounters - (nmlx5_core_en_main.c:1147) Failed to query vport counters (195887328)
2025-04-08T15:47:30.477Z cpu4:2098703)<NMLX_ERR> nmlx5_core: vmnic4: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:30.478Z cpu4:2098703)<NMLX_ERR> nmlx5_core: vmnic4: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:30.478Z cpu4:2098703)<NMLX_ERR> nmlx5_core: vmnic4: nmlx5_en_UpdateVportCounters - (nmlx5_core_en_main.c:1147) Failed to query vport counters (195887328)
2025-04-08T15:47:30.505Z cpu44:2098732)<NMLX_ERR> nmlx5_core: vmnic5: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:30.505Z cpu44:2098732)<NMLX_ERR> nmlx5_core: vmnic5: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:30.505Z cpu44:2098732)<NMLX_ERR> nmlx5_core: vmnic5: nmlx5_en_UpdateVportCounters - (nmlx5_core_en_main.c:1147) Failed to query vport counters (195887328)
2025-04-08T15:47:30.683Z cpu5:2098643)<NMLX_ERR> nmlx5_core: vmnic2: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:30.683Z cpu5:2098643)<NMLX_ERR> nmlx5_core: vmnic2: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:30.683Z cpu5:2098643)<NMLX_ERR> nmlx5_core: vmnic2: nmlx5_en_UpdateVportCounters - (nmlx5_core_en_main.c:1147) Failed to query vport counters (195887328)
2025-04-08T15:47:30.747Z cpu96:2097580)<NMLX_INF> nmlx5_core: vmnic3: nmlx5_en_L2TableIndexAdd - (nmlx5_core_en_main.c:7631) Add 0:15:5D:17:1C:11 to L2 table
2025-04-08T15:47:30.747Z cpu96:2097580)<NMLX_ERR> nmlx5_core: vmnic3: nmlx5_en_L2TableIndexAdd - (nmlx5_core_en_main.c:7662) Failed to set L2 table entry (195887328)
2025-04-08T15:47:30.747Z cpu96:2097580)<NMLX_ERR> nmlx5_core: vmnic3: nmlx5_en_RxQueueFiltersDbApply - (nmlx5_core_en_multiq.c:1712) Failed to add filter into L2 (Failure)
2025-04-08T15:47:30.747Z cpu96:2097580)<NMLX_ERR> nmlx5_core: vmnic3: nmlx5_en_RxQueueFiltersDbApply - (nmlx5_core_en_multiq.c:1738) done, status: Failure
2025-04-08T15:47:30.747Z cpu96:2097580)<NMLX_ERR> nmlx5_core: vmnic3: nmlx5_en_UplinkQApplyFilter - (nmlx5_core_en_multiq.c:1928) nmlx5_en_RxQueueFiltersDbApply failed - Failure
2025-04-08T15:47:30.747Z cpu96:2097580)<NMLX_ERR> nmlx5_core: vmnic3: nmlx5_en_UplinkQApplyFilter - (nmlx5_core_en_multiq.c:1964) done, status: Failure
2025-04-08T15:47:30.749Z cpu65:2098673)<NMLX_ERR> nmlx5_core: vmnic3: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:30.749Z cpu65:2098673)<NMLX_ERR> nmlx5_core: vmnic3: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:30.749Z cpu65:2098673)<NMLX_ERR> nmlx5_core: vmnic3: nmlx5_en_UpdateVportCounters - (nmlx5_core_en_main.c:1147) Failed to query vport counters (195887328)
2025-04-08T15:47:30.978Z cpu0:2098703)<NMLX_ERR> nmlx5_core: vmnic4: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:30.978Z cpu0:2098703)<NMLX_ERR> nmlx5_core: vmnic4: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:30.978Z cpu0:2098703)<NMLX_ERR> nmlx5_core: vmnic4: nmlx5_en_UpdateVportCounters - (nmlx5_core_en_main.c:1147) Failed to query vport counters (195887328)
2025-04-08T15:47:31.006Z cpu44:2098732)<NMLX_ERR> nmlx5_core: vmnic5: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:31.006Z cpu44:2098732)<NMLX_ERR> nmlx5_core: vmnic5: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:31.006Z cpu44:2098732)<NMLX_ERR> nmlx5_core: vmnic5: nmlx5_en_UpdateVportCounters - (nmlx5_core_en_main.c:1147) Failed to query vport counters (195887328)
2025-04-08T15:47:31.184Z cpu0:2098643)<NMLX_ERR> nmlx5_core: vmnic2: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:31.184Z cpu0:2098643)<NMLX_ERR> nmlx5_core: vmnic2: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:31.184Z cpu0:2098643)<NMLX_ERR> nmlx5_core: vmnic2: nmlx5_en_UpdateVportCounters - (nmlx5_core_en_main.c:1147) Failed to query vport counters (195887328)
2025-04-08T15:47:31.249Z cpu65:2098673)<NMLX_ERR> nmlx5_core: vmnic3: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:31.249Z cpu65:2098673)<NMLX_ERR> nmlx5_core: vmnic3: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:31.249Z cpu65:2098673)<NMLX_ERR> nmlx5_core: vmnic3: nmlx5_en_UpdateVportCounters - (nmlx5_core_en_main.c:1147) Failed to query vport counters (195887328)
2025-04-08T15:47:31.479Z cpu0:2098703)<NMLX_ERR> nmlx5_core: vmnic4: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:31.479Z cpu0:2098703)<NMLX_ERR> nmlx5_core: vmnic4: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:31.479Z cpu0:2098703)<NMLX_ERR> nmlx5_core: vmnic4: nmlx5_en_UpdateVportCounters - (nmlx5_core_en_main.c:1147) Failed to query vport counters (195887328)
2025-04-08T15:47:31.507Z cpu40:2098732)<NMLX_ERR> nmlx5_core: vmnic5: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:31.507Z cpu40:2098732)<NMLX_ERR> nmlx5_core: vmnic5: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:31.507Z cpu40:2098732)<NMLX_ERR> nmlx5_core: vmnic5: nmlx5_en_UpdateVportCounters - (nmlx5_core_en_main.c:1147) Failed to query vport counters (195887328)
2025-04-08T15:47:31.686Z cpu0:2098643)<NMLX_ERR> nmlx5_core: vmnic2: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:31.686Z cpu0:2098643)<NMLX_ERR> nmlx5_core: vmnic2: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:31.686Z cpu0:2098643)<NMLX_ERR> nmlx5_core: vmnic2: nmlx5_en_UpdateVportCounters - (nmlx5_core_en_main.c:1147) Failed to query vport counters (195887328)
2025-04-08T15:47:31.749Z cpu70:2098673)<NMLX_ERR> nmlx5_core: vmnic3: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:31.750Z cpu70:2098673)<NMLX_ERR> nmlx5_core: vmnic3: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:31.750Z cpu70:2098673)<NMLX_ERR> nmlx5_core: vmnic3: nmlx5_en_UpdateVportCounters - (nmlx5_core_en_main.c:1147) Failed to query vport counters (195887328)
2025-04-08T15:47:31.979Z cpu0:2098703)<NMLX_ERR> nmlx5_core: vmnic4: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:31.979Z cpu0:2098703)<NMLX_ERR> nmlx5_core: vmnic4: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:31.979Z cpu0:2098703)<NMLX_ERR> nmlx5_core: vmnic4: nmlx5_en_UpdateVportCounters - (nmlx5_core_en_main.c:1147) Failed to query vport counters (195887328)
2025-04-08T15:47:32.009Z cpu40:2098732)<NMLX_ERR> nmlx5_core: vmnic5: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:32.009Z cpu40:2098732)<NMLX_ERR> nmlx5_core: vmnic5: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:32.009Z cpu40:2098732)<NMLX_ERR> nmlx5_core: vmnic5: nmlx5_en_UpdateVportCounters - (nmlx5_core_en_main.c:1147) Failed to query vport counters (195887328)
2025-04-08T15:47:32.186Z cpu1:2098643)<NMLX_ERR> nmlx5_core: vmnic2: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:32.186Z cpu1:2098643)<NMLX_ERR> nmlx5_core: vmnic2: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:32.186Z cpu4:2098643)<NMLX_ERR> nmlx5_core: vmnic2: nmlx5_en_UpdateVportCounters - (nmlx5_core_en_main.c:1147) Failed to query vport counters (195887328)
2025-04-08T15:47:32.250Z cpu70:2098673)<NMLX_ERR> nmlx5_core: vmnic3: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:32.250Z cpu70:2098673)<NMLX_ERR> nmlx5_core: vmnic3: nmlx5_en_QueryOutOfBufferQCounter - (nmlx5_core_en_main.c:1312) Failed query Q Counter: 195887328
2025-04-08T15:47:32.250Z cpu70:2098673)<NMLX_ERR> nmlx5_core: vmnic3: nmlx5_en_UpdateVportCounters - (nmlx5_core_en_main.c:1147) Failed to query vport counters (195887328)
esxcli software vib list | grep nmlx
nmlx5-core 4.22.73.1004-1OEM.703.0.0.18644231 MEL VMwareCertified 2023-06-01
nmlx5-rdma 4.22.73.1004-1OEM.703.0.0.18644231 MEL VMwareCertified 2023-06-01
nmlx4-core 3.19.16.8-2vmw.703.0.20.19193900 VMW VMwareCertified 2021-03-17
nmlx4-en 3.19.16.8-2vmw.703.0.20.19193900 VMW VMwareCertified 2021-03-17
nmlx4-rdma 3.19.16.8-2vmw.703.0.20.19193900 VMW VMwareCertified 2021-03-17
lspci | grep Mellanox
0000:01:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] [vmnic2]
0000:01:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] [vmnic3]
0000:25:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] [vmnic4]
0000:25:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] [vmnic5]
esxcli network nic get -n vmnic2
Advertised Auto Negotiation: true
Advertised Link Modes: Auto, 1000BaseCX-SGMII/Full, 10000BaseKR/Full, 40000BaseCR4/Full, 100000BaseCR4/Full, 25000BaseTwinax/Full, 50000BaseKR2/Full
Auto Negotiation: true
Cable Type: DA
Current Message Level: -1
Driver Info:
Bus Info: 0000:01:00:0
Driver: nmlx5_core
Firmware Version: 16.26.1040
Version: 4.22.73.1004
Link Detected: true
Link Status: Up
Name: vmnic2
PHYAddress: 0
Pause Autonegotiate: false
Pause RX: true
Pause TX: true
Supported Ports: FIBRE, DA
Supports Auto Negotiation: true
Supports Pause: true
Supports Wakeon: false
Transceiver: internal
Virtual Address: …
Wakeon: None