Hello, I run Linux VMs on Azure with accelerated networking activated. A ConnectX-5 VF nic is exposed to the VM. In rare occasions the nic seems to not transmit any packet anymore. Linux reports high value for inflight packets in the byte queue limit. ethtool -S on the VF nic shows no packet being transmitted, while we see the rx_congst_umr error counter increasing.
Following are the details of the typical nic configuration, and also two ethtool -S ran consecutively at 1 second interval during the issue, along with the content of /proc/interrupts and /sys/class/net/${e}/queues/tx-*/byte_queue_limits/inflight. Would you have an idea how to troubleshoot further? thanks!
# lspci
2b7d:00:02.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] (rev 80)
# ethtool -i enP11133s1
driver: mlx5_core
version: 5.14.0-427.65.1.el9_4.x86_64
firmware-version: 16.30.5000 (MSF0000000012)
expansion-rom-version:
bus-info: 2b7d:00:02.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
# ethtool -c enP11133s1
Coalesce parameters for enP11133s1:
Adaptive RX: on TX: on
stats-block-usecs: n/a
sample-interval: n/a
pkt-rate-low: n/a
pkt-rate-high: n/a
rx-usecs: 8
rx-frames: 128
rx-usecs-irq: n/a
rx-frames-irq: n/a
tx-usecs: 8
tx-frames: 128
tx-usecs-irq: n/a
tx-frames-irq: n/a
rx-usecs-low: n/a
rx-frame-low: n/a
tx-usecs-low: n/a
tx-frame-low: n/a
rx-usecs-high: n/a
rx-frame-high: n/a
tx-usecs-high: n/a
tx-frame-high: n/a
CQE mode RX: on TX: off
# ethtool -g enP11133s1
Ring parameters for enP11133s1:
Pre-set maximums:
RX: 8192
RX Mini: n/a
RX Jumbo: n/a
TX: 8192
Current hardware settings:
RX: 8192
RX Mini: n/a
RX Jumbo: n/a
TX: 8192
RX Buf Len: n/a
CQE Size: n/a
TX Push: off
TCP data split: off
# ethtool -l enP11133s1
Channel parameters for enP11133s1:
Pre-set maximums:
RX: n/a
TX: n/a
Other: n/a
Combined: 16
Current hardware settings:
RX: n/a
TX: n/a
Other: n/a
Combined: 16
# ethtool -k enP11133s1
Features for enP11133s1:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: on [fixed]
tx-checksum-ip-generic: off
tx-checksum-ipv6: on [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: off
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off
tx-gre-csum-segmentation: off
tx-ipxip4-segmentation: off
tx-ipxip6-segmentation: off
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: on [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off
rx-udp-gro-forwarding: on
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]
before:
NIC statistics:
rx_packets: 104009754610
rx_bytes: 120879172668176
tx_packets: 135923618504
tx_bytes: 166463657131543
tx_tso_packets: 784014719
tx_tso_bytes: 2185850369783
tx_nop: 9716738702
tx_mpwqe_blks: 16072105203
tx_mpwqe_pkts: 16072147336
rx_csum_unnecessary: 22119468
rx_csum_none: 207575
rx_csum_complete: 103987427567
tx_csum_none: 19998618973
tx_csum_partial: 114808414439
tx_xmit_more: 186651341
tx_cqes: 134620378401
rx_congst_umr: 10021
rx_pp_alloc_fast: 52002899008
rx_pp_alloc_slow: 1304
rx_pp_alloc_empty: 1304
rx_pp_alloc_refill: 2036392
rx_pp_recycle_cached: 51874534256
rx_pp_recycle_ring: 130336912
ch_events: 68341034306
ch_poll: 68390650918
ch_arm: 66604096523
rx_if_down_packets: 171
rx_vport_unicast_packets: 104009754774
rx_vport_unicast_bytes: 120879172890267
tx_vport_unicast_packets: 135923515133
tx_vport_unicast_bytes: 166463648098868
tx_vport_multicast_packets: 687
tx_vport_multicast_bytes: 45818
tx_vport_broadcast_packets: 98992
tx_vport_broadcast_bytes: 5940507
ch0_events: 4413480665
ch0_poll: 4416620862
ch0_arm: 4314261572
ch1_events: 4146125773
ch1_poll: 4149667551
ch1_arm: 4055469921
ch2_events: 4299792341
ch2_poll: 4302650402
ch2_arm: 4205825886
ch3_events: 4084123679
ch3_poll: 4087370962
ch3_arm: 3995885453
ch4_events: 4431598260
ch4_poll: 4434828740
ch4_arm: 4335869996
ch5_events: 4204501467
ch5_poll: 4207212425
ch5_arm: 4033777807
ch6_events: 4373780852
ch6_poll: 4376839023
ch6_arm: 4279664279
ch7_events: 4110418756
ch7_poll: 4113805343
ch7_arm: 4021313755
ch8_events: 4379982651
ch8_poll: 4383228963
ch8_arm: 4285001032
ch9_events: 4114238806
ch9_poll: 4117740222
ch9_arm: 4017400999
ch10_events: 4421714024
ch10_poll: 4424797982
ch10_arm: 4325953285
ch11_events: 4151232296
ch11_poll: 4154488239
ch11_arm: 4064231670
ch12_events: 4297283053
ch12_poll: 4300197023
ch12_arm: 4198746228
ch13_events: 4137265800
ch13_poll: 4139692689
ch13_arm: 3963223482
ch14_events: 4600649457
ch14_poll: 4603385985
ch14_arm: 4419567187
ch15_events: 4174846428
ch15_poll: 4178124509
ch15_arm: 4087903973
rx0_packets: 6448464205
rx0_bytes: 7487373996616
rx0_csum_complete: 6446872207
rx0_csum_unnecessary: 1384423
rx0_csum_none: 207575
rx0_congst_umr: 437
rx0_pp_alloc_fast: 3224111211
rx0_pp_alloc_slow: 76
rx0_pp_alloc_empty: 76
rx0_pp_alloc_refill: 124425
rx0_pp_recycle_cached: 3216268057
rx0_pp_recycle_ring: 7963559
rx1_packets: 6670961580
rx1_bytes: 7786461152114
rx1_csum_complete: 6669582622
rx1_csum_unnecessary: 1378958
rx1_congst_umr: 650
rx1_pp_alloc_fast: 3335347612
rx1_pp_alloc_slow: 83
rx1_pp_alloc_empty: 83
rx1_pp_alloc_refill: 135505
rx1_pp_recycle_cached: 3326806223
rx1_pp_recycle_ring: 8672881
rx2_packets: 6054591230
rx2_bytes: 6990855128070
rx2_csum_complete: 6053211802
rx2_csum_unnecessary: 1379428
rx2_congst_umr: 670
rx2_pp_alloc_fast: 3027187201
rx2_pp_alloc_slow: 87
rx2_pp_alloc_empty: 87
rx2_pp_alloc_refill: 112232
rx2_pp_recycle_cached: 3020111383
rx2_pp_recycle_ring: 7184041
rx3_packets: 6397840279
rx3_bytes: 7437300687575
rx3_csum_complete: 6396458293
rx3_csum_unnecessary: 1381986
rx3_congst_umr: 581
rx3_pp_alloc_fast: 3198800922
rx3_pp_alloc_slow: 74
rx3_pp_alloc_empty: 74
rx3_pp_alloc_refill: 121180
rx3_pp_recycle_cached: 3191162219
rx3_pp_recycle_ring: 7755861
rx4_packets: 6501905100
rx4_bytes: 7559972982862
rx4_csum_complete: 6500523208
rx4_csum_unnecessary: 1381892
rx4_congst_umr: 507
rx4_pp_alloc_fast: 3250832552
rx4_pp_alloc_slow: 75
rx4_pp_alloc_empty: 75
rx4_pp_alloc_refill: 123853
rx4_pp_recycle_cached: 3243025558
rx4_pp_recycle_ring: 7926826
rx5_packets: 6870441834
rx5_bytes: 8008005259271
rx5_csum_complete: 6869055614
rx5_csum_unnecessary: 1386220
rx5_congst_umr: 830
rx5_pp_alloc_fast: 3435085208
rx5_pp_alloc_slow: 78
rx5_pp_alloc_empty: 78
rx5_pp_alloc_refill: 139610
rx5_pp_recycle_cached: 3426285335
rx5_pp_recycle_ring: 8935465
rx6_packets: 6293902315
rx6_bytes: 7294818063657
rx6_csum_complete: 6292519651
rx6_csum_unnecessary: 1382664
rx6_congst_umr: 557
rx6_pp_alloc_fast: 3146834631
rx6_pp_alloc_slow: 74
rx6_pp_alloc_empty: 74
rx6_pp_alloc_refill: 120367
rx6_pp_recycle_cached: 3139247122
rx6_pp_recycle_ring: 7703854
rx7_packets: 6519867981
rx7_bytes: 7575626289555
rx7_csum_complete: 6518484249
rx7_csum_unnecessary: 1383732
rx7_congst_umr: 351
rx7_pp_alloc_fast: 3259804000
rx7_pp_alloc_slow: 75
rx7_pp_alloc_empty: 75
rx7_pp_alloc_refill: 133909
rx7_pp_recycle_cached: 3251363237
rx7_pp_recycle_ring: 8570651
rx8_packets: 6472453931
rx8_bytes: 7534643211397
rx8_csum_complete: 6471071574
rx8_csum_unnecessary: 1382357
rx8_congst_umr: 735
rx8_pp_alloc_fast: 3236102192
rx8_pp_alloc_slow: 77
rx8_pp_alloc_empty: 77
rx8_pp_alloc_refill: 128707
rx8_pp_recycle_cached: 3227989186
rx8_pp_recycle_ring: 8237694
rx9_packets: 6677102412
rx9_bytes: 7798889123945
rx9_csum_complete: 6675721869
rx9_csum_unnecessary: 1380543
rx9_congst_umr: 700
rx9_pp_alloc_fast: 3338424233
rx9_pp_alloc_slow: 126
rx9_pp_alloc_empty: 126
rx9_pp_alloc_refill: 130841
rx9_pp_recycle_cached: 3330176947
rx9_pp_recycle_ring: 8374157
rx10_packets: 6364217606
rx10_bytes: 7394286290617
rx10_csum_complete: 6362834214
rx10_csum_unnecessary: 1383392
rx10_congst_umr: 576
rx10_pp_alloc_fast: 3181992582
rx10_pp_alloc_slow: 85
rx10_pp_alloc_empty: 85
rx10_pp_alloc_refill: 120037
rx10_pp_recycle_cached: 3174425581
rx10_pp_recycle_ring: 7683027
rx11_packets: 6480670695
rx11_bytes: 7529042264596
rx11_csum_complete: 6479287203
rx11_csum_unnecessary: 1383492
rx11_congst_umr: 601
rx11_pp_alloc_fast: 3240217525
rx11_pp_alloc_slow: 83
rx11_pp_alloc_empty: 83
rx11_pp_alloc_refill: 121656
rx11_pp_recycle_cached: 3232548272
rx11_pp_recycle_ring: 7786896
rx12_packets: 6223217093
rx12_bytes: 7205922129762
rx12_csum_complete: 6221831487
rx12_csum_unnecessary: 1385606
rx12_congst_umr: 575
rx12_pp_alloc_fast: 3111492036
rx12_pp_alloc_slow: 74
rx12_pp_alloc_empty: 74
rx12_pp_alloc_refill: 120370
rx12_pp_recycle_cached: 3103904492
rx12_pp_recycle_ring: 7703892
rx13_packets: 6544266555
rx13_bytes: 7594392681559
rx13_csum_complete: 6542881988
rx13_csum_unnecessary: 1384567
rx13_congst_umr: 816
rx13_pp_alloc_fast: 3272004775
rx13_pp_alloc_slow: 79
rx13_pp_alloc_empty: 79
rx13_pp_alloc_refill: 132426
rx13_pp_recycle_cached: 3263657668
rx13_pp_recycle_ring: 8475516
rx14_packets: 6955392761
rx14_bytes: 8110633259049
rx14_csum_complete: 6954013059
rx14_csum_unnecessary: 1379702
rx14_congst_umr: 574
rx14_pp_alloc_fast: 3477553812
rx14_pp_alloc_slow: 74
rx14_pp_alloc_empty: 74
rx14_pp_alloc_refill: 146402
rx14_pp_recycle_cached: 3468326197
rx14_pp_recycle_ring: 9369995
rx15_packets: 6534459035
rx15_bytes: 7570950148047
rx15_csum_complete: 6533078529
rx15_csum_unnecessary: 1380506
rx15_congst_umr: 863
rx15_pp_alloc_fast: 3267108516
rx15_pp_alloc_slow: 84
rx15_pp_alloc_empty: 84
rx15_pp_alloc_refill: 124872
rx15_pp_recycle_cached: 3259236779
rx15_pp_recycle_ring: 7992597
tx0_packets: 9177681073
tx0_bytes: 11254226376765
tx0_tso_packets: 53127845
tx0_tso_bytes: 146994119190
tx0_csum_partial: 7772718338
tx0_nop: 660764086
tx0_mpwqe_blks: 1074575531
tx0_mpwqe_pkts: 1074582746
tx0_csum_none: 1329934984
tx0_xmit_more: 10481170
tx0_cqes: 9092171890
tx1_packets: 7794877780
tx1_bytes: 9486385283539
tx1_tso_packets: 44479392
tx1_tso_bytes: 124745961213
tx1_csum_partial: 6533345637
tx1_nop: 575084230
tx1_mpwqe_blks: 970022701
tx1_mpwqe_pkts: 970024050
tx1_csum_none: 1197760362
tx1_xmit_more: 8026132
tx1_cqes: 7723079544
tx2_packets: 9147381580
tx2_bytes: 11219597798912
tx2_tso_packets: 52152135
tx2_tso_bytes: 144404732216
tx2_csum_partial: 7746513280
tx2_nop: 654744271
tx2_mpwqe_blks: 1065206055
tx2_mpwqe_pkts: 1065207458
tx2_csum_none: 1326841759
tx2_xmit_more: 8773017
tx2_cqes: 9064581863
tx3_packets: 7745520315
tx3_bytes: 9424046176462
tx3_tso_packets: 44507411
tx3_tso_bytes: 125120113942
tx3_csum_partial: 6491977125
tx3_nop: 572124476
tx3_mpwqe_blks: 961882020
tx3_mpwqe_pkts: 961888185
tx3_csum_none: 1189460335
tx3_xmit_more: 9936385
tx3_cqes: 7671500951
tx4_packets: 9217509308
tx4_bytes: 11345096730548
tx4_tso_packets: 53418160
tx4_tso_bytes: 147601645729
tx4_csum_partial: 7833977480
tx4_nop: 651582103
tx4_mpwqe_blks: 1050517957
tx4_mpwqe_pkts: 1050520040
tx4_csum_none: 1308329425
tx4_xmit_more: 12595000
tx4_cqes: 9129711389
tx5_packets: 7741990169
tx5_bytes: 9414523674913
tx5_tso_packets: 44183763
tx5_tso_bytes: 124023690172
tx5_csum_partial: 6485880001
tx5_nop: 552314355
tx5_mpwqe_blks: 960841864
tx5_mpwqe_pkts: 960843113
tx5_csum_none: 1192736334
tx5_xmit_more: 14209071
tx5_cqes: 7664407160
tx6_packets: 9140721203
tx6_bytes: 11218903864104
tx6_tso_packets: 52850320
tx6_tso_bytes: 146420501337
tx6_csum_partial: 7746403146
tx6_nop: 652038962
tx6_mpwqe_blks: 1058979062
tx6_mpwqe_pkts: 1058980438
tx6_csum_none: 1319273428
tx6_xmit_more: 9609986
tx6_cqes: 9056066278
tx7_packets: 7847795161
tx7_bytes: 9601002319347
tx7_tso_packets: 46046525
tx7_tso_bytes: 129344430148
tx7_csum_partial: 6616398352
tx7_nop: 568045411
tx7_mpwqe_blks: 935912301
tx7_mpwqe_pkts: 935913395
tx7_csum_none: 1165391419
tx7_xmit_more: 8525381
tx7_cqes: 7773264357
tx8_packets: 9051681212
tx8_bytes: 11129332815032
tx8_tso_packets: 52488444
tx8_tso_bytes: 145673299300
tx8_csum_partial: 7683796730
tx8_nop: 641047104
tx8_mpwqe_blks: 1037809710
tx8_mpwqe_pkts: 1037811085
tx8_csum_none: 1293319760
tx8_xmit_more: 11748355
tx8_cqes: 8965368023
tx9_packets: 7812377419
tx9_bytes: 9566570966203
tx9_tso_packets: 45407397
tx9_tso_bytes: 127508180981
tx9_csum_partial: 6589714653
tx9_nop: 560268655
tx9_mpwqe_blks: 928772059
tx9_mpwqe_pkts: 928775588
tx9_csum_none: 1157733612
tx9_xmit_more: 11523999
tx9_cqes: 7735923686
tx10_packets: 9139401359
tx10_bytes: 11243710228410
tx10_tso_packets: 53598355
tx10_tso_bytes: 148174444033
tx10_csum_partial: 7759397160
tx10_nop: 647330629
tx10_mpwqe_blks: 1040579776
tx10_mpwqe_pkts: 1040582377
tx10_csum_none: 1304615418
tx10_xmit_more: 14477432
tx10_cqes: 9049534982
tx11_packets: 7958280479
tx11_bytes: 9733645614983
tx11_tso_packets: 45752087
tx11_tso_bytes: 129013149046
tx11_csum_partial: 6703444309
tx11_nop: 575170424
tx11_mpwqe_blks: 960554431
tx11_mpwqe_pkts: 960555886
tx11_csum_none: 1189114410
tx11_xmit_more: 11768405
tx11_cqes: 7880790217
tx12_packets: 9214326004
tx12_bytes: 11356095581400
tx12_tso_packets: 52812793
tx12_tso_bytes: 145869442606
tx12_csum_partial: 7842956814
tx12_nop: 646576494
tx12_mpwqe_blks: 1036863432
tx12_mpwqe_pkts: 1036870176
tx12_csum_none: 1296765669
tx12_xmit_more: 9876175
tx12_cqes: 9129846276
tx13_packets: 7771739047
tx13_bytes: 9460232437077
tx13_tso_packets: 44625991
tx13_tso_bytes: 126064407221
tx13_csum_partial: 6517997408
tx13_nop: 548002615
tx13_mpwqe_blks: 954119763
tx13_mpwqe_pkts: 954121021
tx13_csum_none: 1189118295
tx13_xmit_more: 16528598
tx13_cqes: 7690586560
tx14_packets: 9327198615
tx14_bytes: 11444099882295
tx14_tso_packets: 53471737
tx14_tso_bytes: 148109721848
tx14_csum_partial: 7898024779
tx14_nop: 642659882
tx14_mpwqe_blks: 1085046860
tx14_mpwqe_pkts: 1085048744
tx14_csum_none: 1353706875
tx14_xmit_more: 18569646
tx14_cqes: 9233161893
tx15_packets: 7835137780
tx15_bytes: 9566187381553
tx15_tso_packets: 45092364
tx15_tso_bytes: 126782530801
tx15_csum_partial: 6585869227
tx15_nop: 568985005
tx15_mpwqe_blks: 950421681
tx15_mpwqe_pkts: 950423034
tx15_csum_none: 1184516888
tx15_xmit_more: 10002589
tx15_cqes: 7760383332
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15
4: 0 0 3611 0 0 0 0 0 0 0 0 0 0 0 0 0 IO-APIC 4-edge ttyS0
8: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IO-APIC 8-edge rtc0
9: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IO-APIC 9-fasteoi acpi
24: 0 150909051 1042065 149379077 0 70418610 14390803 56870401 112474 55662095 32669605 17360582 21023079 129188917 4326424 93861436 Hyper-V PCIe MSI 77309444096-edge mlx5_async0@pci:0240:00:02.0
25: 724 0 55 6591 1 63866 0 42887 0 38418 4263755032 46101 66 112621960 21734 39514 Hyper-V PCIe MSI 77309444097-edge mlx5_comp1@pci:0240:00:02.0
26: 0 2352 0 3502564496 0 29373 19601 18750 109210365 16588 41915 501087562 0 20743 10602 46259 Hyper-V PCIe MSI 77309444098-edge mlx5_comp2@pci:0240:00:02.0
27: 0 0 353 8706 0 477918495 1 3686124781 0 13104 62467 100597640 2557 43409 156502 3031 Hyper-V PCIe MSI 77309444099-edge mlx5_comp3@pci:0240:00:02.0
28: 0 0 53 5695 99494146 13620 31334 13007 3951675493 9642 91 10508 63364 33078 0 2337 Hyper-V PCIe MSI 77309444100-edge mlx5_comp4@pci:0240:00:02.0
29: 35 54 70 10047 37105 4040 30674 102167396 16292 10266 27833 16071 61248 4419 7340 10689 Hyper-V PCIe MSI 77309444101-edge mlx5_comp5@pci:0240:00:02.0
30: 0 262 14603 110210842 9501 3578839369 486165401 14199 10197 11604 8420 21814 1266 8833 9360 7790 Hyper-V PCIe MSI 77309444102-edge mlx5_comp6@pci:0240:00:02.0
31: 0 77 18869 24170 4231072938 15786 475 14496 9886 31192 4221 22148 33810 22291 203 109197150 Hyper-V PCIe MSI 77309444103-edge mlx5_comp7@pci:0240:00:02.0
32: 0 4077665806 0 0 0 0 0 415 0 0 0 1 0 0 0 0 Hyper-V PCIe MSI 77309444104-edge mlx5_comp8@pci:0240:00:02.0
33: 51497030 0 0 0 0 0 0 0 318 0 0 0 1 0 0 0 Hyper-V PCIe MSI 77309444105-edge mlx5_comp9@pci:0240:00:02.0
34: 0 0 0 62726 48 5775 43 5863 0 419937546 101252282 48454055 38105 1 43231 3512140862 Hyper-V PCIe MSI 77309444106-edge mlx5_comp10@pci:0240:00:02.0
35: 0 0 178561 2508 100 7998 54 956 0 19415 15045 3868725291 66 1560 516210967 6198 Hyper-V PCIe MSI 77309444107-edge mlx5_comp11@pci:0240:00:02.0
36: 0 46 0 502903455 101663 35353 3617230094 19949 39964 37889 31 25337 1970 1683 53 20272 Hyper-V PCIe MSI 77309444108-edge mlx5_comp12@pci:0240:00:02.0
37: 1 74 44 11848 197 24423 26 495532649 2547 3763078974 512 30262 23720 19792 9280 44505 Hyper-V PCIe MSI 77309444109-edge mlx5_comp13@pci:0240:00:02.0
38: 0 1 95909455 13699 239 16907 3609 18699 18534 19693 464 25842 19863 4013818918 339 17774 Hyper-V PCIe MSI 77309444110-edge mlx5_comp14@pci:0240:00:02.0
39: 0 108 4227 8941 213 109792302 120 38226 6573 15690 99 25456 11976 37372 3940671822 518488287 Hyper-V PCIe MSI 77309444111-edge mlx5_comp15@pci:0240:00:02.0
40: 0 0 4038730004 292 116746 1554 90264 4821 0 30480 0 5859 0 29772 105044550 368 Hyper-V PCIe MSI 77309444112-edge mlx5_comp16@pci:0240:00:02.0
NMI: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Non-maskable interrupts
LOC: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Local timer interrupts
SPU: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Performance monitoring interrupts
IWI: 1879699 1852710 1896833 1843913 1865220 1841230 1893002 1831720 1851141 1789618 1904516 1815972 1865510 1832217 1905321 1802954 IRQ work interrupts
RTR: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 APIC ICR read retries
RES: 55323541 43359059 52988184 44035092 54827063 55137009 51942961 46547186 53997968 48453658 55738968 51259695 55259872 55383204 67824836 45993465 Rescheduling interrupts
CAL: 1891134989 444940925 1914582910 475347289 1894798469 407571888 1954832215 527327041 1958503614 515750654 1963093120 518191207 1998609751 466210738 1915275369 513880276 Function call interrupts
TLB: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 TLB shootdowns
TRM: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Threshold APIC interrupts
DFR: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Deferred Error APIC interrupts
MCE: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Machine check exceptions
MCP: 3615 3615 3615 3615 3615 3615 3615 3615 3615 3615 3615 3615 3615 3615 3615 3615 Machine check polls
HYP: 1497790 5 48 673400 1087961 772567 0 32 0 11 49 13 3 12 15 7 Hypervisor callback interrupts
HRE: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Hyper-V reenlightenment interrupts
HVS: 1441309393 1351719889 1446526615 1352678368 1444433101 1348243275 1444985328 1353173258 1438402005 1340604615 1437721381 1347817400 1439038314 1350275399 1441700352 1350886985 Hyper-V stimer0 interrupts
ERR: 0
MIS: 0
PIN: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Posted-interrupt notification event
NPI: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Nested posted-interrupt event
PIW: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Posted-interrupt wakeup event
/sys/class/net/enP576s1/queues/tx-0/byte_queue_limits/inflight:184783
/sys/class/net/enP576s1/queues/tx-1/byte_queue_limits/inflight:455668
/sys/class/net/enP576s1/queues/tx-10/byte_queue_limits/inflight:139458
/sys/class/net/enP576s1/queues/tx-11/byte_queue_limits/inflight:81794
/sys/class/net/enP576s1/queues/tx-12/byte_queue_limits/inflight:25416
/sys/class/net/enP576s1/queues/tx-13/byte_queue_limits/inflight:179768
/sys/class/net/enP576s1/queues/tx-14/byte_queue_limits/inflight:112049
/sys/class/net/enP576s1/queues/tx-15/byte_queue_limits/inflight:169535
/sys/class/net/enP576s1/queues/tx-2/byte_queue_limits/inflight:144471
/sys/class/net/enP576s1/queues/tx-3/byte_queue_limits/inflight:128293
/sys/class/net/enP576s1/queues/tx-4/byte_queue_limits/inflight:528686
/sys/class/net/enP576s1/queues/tx-5/byte_queue_limits/inflight:140212
/sys/class/net/enP576s1/queues/tx-6/byte_queue_limits/inflight:235499
/sys/class/net/enP576s1/queues/tx-7/byte_queue_limits/inflight:30932
/sys/class/net/enP576s1/queues/tx-8/byte_queue_limits/inflight:92361
/sys/class/net/enP576s1/queues/tx-9/byte_queue_limits/inflight:397425
1s after:
NIC statistics:
rx_packets: 104009760946
rx_bytes: 120879175263066
tx_packets: 135923618504
tx_bytes: 166463657131543
tx_tso_packets: 784014719
tx_tso_bytes: 2185850369783
tx_nop: 9716738702
tx_mpwqe_blks: 16072105203
tx_mpwqe_pkts: 16072147336
rx_csum_unnecessary: 22119512
rx_csum_none: 207575
rx_csum_complete: 103987433859
tx_csum_none: 19998618973
tx_csum_partial: 114808414439
tx_xmit_more: 186651341
tx_cqes: 134620378401
rx_congst_umr: 15949
rx_pp_alloc_fast: 52002902464
rx_pp_alloc_slow: 1304
rx_pp_alloc_empty: 1304
rx_pp_alloc_refill: 2036392
rx_pp_recycle_cached: 51874537712
rx_pp_recycle_ring: 130336912
ch_events: 68341040234
ch_poll: 68390656846
ch_arm: 66604102451
rx_if_down_packets: 171
rx_vport_unicast_packets: 104009761116
rx_vport_unicast_bytes: 120879175486554
tx_vport_unicast_packets: 135923515133
tx_vport_unicast_bytes: 166463648098868
tx_vport_multicast_packets: 687
tx_vport_multicast_bytes: 45818
tx_vport_broadcast_packets: 98992
tx_vport_broadcast_bytes: 5940507
ch0_events: 4413481085
ch0_poll: 4416621282
ch0_arm: 4314261992
ch1_events: 4146126125
ch1_poll: 4149667903
ch1_arm: 4055470273
ch2_events: 4299792781
ch2_poll: 4302650842
ch2_arm: 4205826326
ch3_events: 4084124016
ch3_poll: 4087371299
ch3_arm: 3995885790
ch4_events: 4431598643
ch4_poll: 4434829123
ch4_arm: 4335870379
ch5_events: 4204501853
ch5_poll: 4207212811
ch5_arm: 4033778193
ch6_events: 4373781187
ch6_poll: 4376839358
ch6_arm: 4279664614
ch7_events: 4110419118
ch7_poll: 4113805705
ch7_arm: 4021314117
ch8_events: 4379983025
ch8_poll: 4383229337
ch8_arm: 4285001406
ch9_events: 4114239157
ch9_poll: 4117740573
ch9_arm: 4017401350
ch10_events: 4421714313
ch10_poll: 4424798271
ch10_arm: 4325953574
ch11_events: 4151232702
ch11_poll: 4154488645
ch11_arm: 4064232076
ch12_events: 4297283434
ch12_poll: 4300197404
ch12_arm: 4198746609
ch13_events: 4137266163
ch13_poll: 4139693052
ch13_arm: 3963223845
ch14_events: 4600649825
ch14_poll: 4603386353
ch14_arm: 4419567555
ch15_events: 4174846807
ch15_poll: 4178124888
ch15_arm: 4087904352
rx0_packets: 6448464653
rx0_bytes: 7487374194266
rx0_csum_complete: 6446872645
rx0_csum_unnecessary: 1384433
rx0_csum_none: 207575
rx0_congst_umr: 857
rx0_pp_alloc_fast: 3224111467
rx0_pp_alloc_slow: 76
rx0_pp_alloc_empty: 76
rx0_pp_alloc_refill: 124425
rx0_pp_recycle_cached: 3216268313
rx0_pp_recycle_ring: 7963559
rx1_packets: 6670961955
rx1_bytes: 7786461298781
rx1_csum_complete: 6669582995
rx1_csum_unnecessary: 1378960
rx1_congst_umr: 1002
rx1_pp_alloc_fast: 3335347868
rx1_pp_alloc_slow: 83
rx1_pp_alloc_empty: 83
rx1_pp_alloc_refill: 135505
rx1_pp_recycle_cached: 3326806479
rx1_pp_recycle_ring: 8672881
rx2_packets: 6054591699
rx2_bytes: 6990855316791
rx2_csum_complete: 6053212271
rx2_csum_unnecessary: 1379428
rx2_congst_umr: 1110
rx2_pp_alloc_fast: 3027187457
rx2_pp_alloc_slow: 87
rx2_pp_alloc_empty: 87
rx2_pp_alloc_refill: 112232
rx2_pp_recycle_cached: 3020111639
rx2_pp_recycle_ring: 7184041
rx3_packets: 6397840631
rx3_bytes: 7437300822494
rx3_csum_complete: 6396458645
rx3_csum_unnecessary: 1381986
rx3_congst_umr: 918
rx3_pp_alloc_fast: 3198801178
rx3_pp_alloc_slow: 74
rx3_pp_alloc_empty: 74
rx3_pp_alloc_refill: 121180
rx3_pp_recycle_cached: 3191162475
rx3_pp_recycle_ring: 7755861
rx4_packets: 6501905505
rx4_bytes: 7559973135166
rx4_csum_complete: 6500523603
rx4_csum_unnecessary: 1381902
rx4_congst_umr: 890
rx4_pp_alloc_fast: 3250832808
rx4_pp_alloc_slow: 75
rx4_pp_alloc_empty: 75
rx4_pp_alloc_refill: 123853
rx4_pp_recycle_cached: 3243025814
rx4_pp_recycle_ring: 7926826
rx5_packets: 6870442246
rx5_bytes: 8008005445176
rx5_csum_complete: 6869056026
rx5_csum_unnecessary: 1386220
rx5_congst_umr: 1216
rx5_pp_alloc_fast: 3435085336
rx5_pp_alloc_slow: 78
rx5_pp_alloc_empty: 78
rx5_pp_alloc_refill: 139610
rx5_pp_recycle_cached: 3426285463
rx5_pp_recycle_ring: 8935465
rx6_packets: 6293902678
rx6_bytes: 7294818212319
rx6_csum_complete: 6292520010
rx6_csum_unnecessary: 1382668
rx6_congst_umr: 892
rx6_pp_alloc_fast: 3146834887
rx6_pp_alloc_slow: 74
rx6_pp_alloc_empty: 74
rx6_pp_alloc_refill: 120367
rx6_pp_recycle_cached: 3139247378
rx6_pp_recycle_ring: 7703854
rx7_packets: 6519868363
rx7_bytes: 7575626446100
rx7_csum_complete: 6518484629
rx7_csum_unnecessary: 1383734
rx7_congst_umr: 713
rx7_pp_alloc_fast: 3259804128
rx7_pp_alloc_slow: 75
rx7_pp_alloc_empty: 75
rx7_pp_alloc_refill: 133909
rx7_pp_recycle_cached: 3251363365
rx7_pp_recycle_ring: 8570651
rx8_packets: 6472454323
rx8_bytes: 7534643377485
rx8_csum_complete: 6471071962
rx8_csum_unnecessary: 1382361
rx8_congst_umr: 1109
rx8_pp_alloc_fast: 3236102320
rx8_pp_alloc_slow: 77
rx8_pp_alloc_empty: 77
rx8_pp_alloc_refill: 128707
rx8_pp_recycle_cached: 3227989314
rx8_pp_recycle_ring: 8237694
rx9_packets: 6677102792
rx9_bytes: 7798889286125
rx9_csum_complete: 6675722249
rx9_csum_unnecessary: 1380543
rx9_congst_umr: 1051
rx9_pp_alloc_fast: 3338424361
rx9_pp_alloc_slow: 126
rx9_pp_alloc_empty: 126
rx9_pp_alloc_refill: 130841
rx9_pp_recycle_cached: 3330177075
rx9_pp_recycle_ring: 8374157
rx10_packets: 6364217916
rx10_bytes: 7394286428725
rx10_csum_complete: 6362834523
rx10_csum_unnecessary: 1383393
rx10_congst_umr: 865
rx10_pp_alloc_fast: 3181992838
rx10_pp_alloc_slow: 85
rx10_pp_alloc_empty: 85
rx10_pp_alloc_refill: 120037
rx10_pp_recycle_cached: 3174425837
rx10_pp_recycle_ring: 7683027
rx11_packets: 6480671116
rx11_bytes: 7529042404323
rx11_csum_complete: 6479287614
rx11_csum_unnecessary: 1383502
rx11_congst_umr: 1007
rx11_pp_alloc_fast: 3240217781
rx11_pp_alloc_slow: 83
rx11_pp_alloc_empty: 83
rx11_pp_alloc_refill: 121656
rx11_pp_recycle_cached: 3232548528
rx11_pp_recycle_ring: 7786896
rx12_packets: 6223217521
rx12_bytes: 7205922328739
rx12_csum_complete: 6221831915
rx12_csum_unnecessary: 1385606
rx12_congst_umr: 956
rx12_pp_alloc_fast: 3111492292
rx12_pp_alloc_slow: 74
rx12_pp_alloc_empty: 74
rx12_pp_alloc_refill: 120370
rx12_pp_recycle_cached: 3103904748
rx12_pp_recycle_ring: 7703892
rx13_packets: 6544266941
rx13_bytes: 7594392840762
rx13_csum_complete: 6542882374
rx13_csum_unnecessary: 1384567
rx13_congst_umr: 1179
rx13_pp_alloc_fast: 3272004903
rx13_pp_alloc_slow: 79
rx13_pp_alloc_empty: 79
rx13_pp_alloc_refill: 132426
rx13_pp_recycle_cached: 3263657796
rx13_pp_recycle_ring: 8475516
rx14_packets: 6955393161
rx14_bytes: 8110633413018
rx14_csum_complete: 6954013458
rx14_csum_unnecessary: 1379703
rx14_congst_umr: 942
rx14_pp_alloc_fast: 3477554068
rx14_pp_alloc_slow: 74
rx14_pp_alloc_empty: 74
rx14_pp_alloc_refill: 146402
rx14_pp_recycle_cached: 3468326453
rx14_pp_recycle_ring: 9369995
rx15_packets: 6534459446
rx15_bytes: 7570950312796
rx15_csum_complete: 6533078940
rx15_csum_unnecessary: 1380506
rx15_congst_umr: 1242
rx15_pp_alloc_fast: 3267108772
rx15_pp_alloc_slow: 84
rx15_pp_alloc_empty: 84
rx15_pp_alloc_refill: 124872
rx15_pp_recycle_cached: 3259237035
rx15_pp_recycle_ring: 7992597
tx0_packets: 9177681073
tx0_bytes: 11254226376765
tx0_tso_packets: 53127845
tx0_tso_bytes: 146994119190
tx0_csum_partial: 7772718338
tx0_nop: 660764086
tx0_mpwqe_blks: 1074575531
tx0_mpwqe_pkts: 1074582746
tx0_csum_none: 1329934984
tx0_xmit_more: 10481170
tx0_cqes: 9092171890
tx1_packets: 7794877780
tx1_bytes: 9486385283539
tx1_tso_packets: 44479392
tx1_tso_bytes: 124745961213
tx1_csum_partial: 6533345637
tx1_nop: 575084230
tx1_mpwqe_blks: 970022701
tx1_mpwqe_pkts: 970024050
tx1_csum_none: 1197760362
tx1_xmit_more: 8026132
tx1_cqes: 7723079544
tx2_packets: 9147381580
tx2_bytes: 11219597798912
tx2_tso_packets: 52152135
tx2_tso_bytes: 144404732216
tx2_csum_partial: 7746513280
tx2_nop: 654744271
tx2_mpwqe_blks: 1065206055
tx2_mpwqe_pkts: 1065207458
tx2_csum_none: 1326841759
tx2_xmit_more: 8773017
tx2_cqes: 9064581863
tx3_packets: 7745520315
tx3_bytes: 9424046176462
tx3_tso_packets: 44507411
tx3_tso_bytes: 125120113942
tx3_csum_partial: 6491977125
tx3_nop: 572124476
tx3_mpwqe_blks: 961882020
tx3_mpwqe_pkts: 961888185
tx3_csum_none: 1189460335
tx3_xmit_more: 9936385
tx3_cqes: 7671500951
tx4_packets: 9217509308
tx4_bytes: 11345096730548
tx4_tso_packets: 53418160
tx4_tso_bytes: 147601645729
tx4_csum_partial: 7833977480
tx4_nop: 651582103
tx4_mpwqe_blks: 1050517957
tx4_mpwqe_pkts: 1050520040
tx4_csum_none: 1308329425
tx4_xmit_more: 12595000
tx4_cqes: 9129711389
tx5_packets: 7741990169
tx5_bytes: 9414523674913
tx5_tso_packets: 44183763
tx5_tso_bytes: 124023690172
tx5_csum_partial: 6485880001
tx5_nop: 552314355
tx5_mpwqe_blks: 960841864
tx5_mpwqe_pkts: 960843113
tx5_csum_none: 1192736334
tx5_xmit_more: 14209071
tx5_cqes: 7664407160
tx6_packets: 9140721203
tx6_bytes: 11218903864104
tx6_tso_packets: 52850320
tx6_tso_bytes: 146420501337
tx6_csum_partial: 7746403146
tx6_nop: 652038962
tx6_mpwqe_blks: 1058979062
tx6_mpwqe_pkts: 1058980438
tx6_csum_none: 1319273428
tx6_xmit_more: 9609986
tx6_cqes: 9056066278
tx7_packets: 7847795161
tx7_bytes: 9601002319347
tx7_tso_packets: 46046525
tx7_tso_bytes: 129344430148
tx7_csum_partial: 6616398352
tx7_nop: 568045411
tx7_mpwqe_blks: 935912301
tx7_mpwqe_pkts: 935913395
tx7_csum_none: 1165391419
tx7_xmit_more: 8525381
tx7_cqes: 7773264357
tx8_packets: 9051681212
tx8_bytes: 11129332815032
tx8_tso_packets: 52488444
tx8_tso_bytes: 145673299300
tx8_csum_partial: 7683796730
tx8_nop: 641047104
tx8_mpwqe_blks: 1037809710
tx8_mpwqe_pkts: 1037811085
tx8_csum_none: 1293319760
tx8_xmit_more: 11748355
tx8_cqes: 8965368023
tx9_packets: 7812377419
tx9_bytes: 9566570966203
tx9_tso_packets: 45407397
tx9_tso_bytes: 127508180981
tx9_csum_partial: 6589714653
tx9_nop: 560268655
tx9_mpwqe_blks: 928772059
tx9_mpwqe_pkts: 928775588
tx9_csum_none: 1157733612
tx9_xmit_more: 11523999
tx9_cqes: 7735923686
tx10_packets: 9139401359
tx10_bytes: 11243710228410
tx10_tso_packets: 53598355
tx10_tso_bytes: 148174444033
tx10_csum_partial: 7759397160
tx10_nop: 647330629
tx10_mpwqe_blks: 1040579776
tx10_mpwqe_pkts: 1040582377
tx10_csum_none: 1304615418
tx10_xmit_more: 14477432
tx10_cqes: 9049534982
tx11_packets: 7958280479
tx11_bytes: 9733645614983
tx11_tso_packets: 45752087
tx11_tso_bytes: 129013149046
tx11_csum_partial: 6703444309
tx11_nop: 575170424
tx11_mpwqe_blks: 960554431
tx11_mpwqe_pkts: 960555886
tx11_csum_none: 1189114410
tx11_xmit_more: 11768405
tx11_cqes: 7880790217
tx12_packets: 9214326004
tx12_bytes: 11356095581400
tx12_tso_packets: 52812793
tx12_tso_bytes: 145869442606
tx12_csum_partial: 7842956814
tx12_nop: 646576494
tx12_mpwqe_blks: 1036863432
tx12_mpwqe_pkts: 1036870176
tx12_csum_none: 1296765669
tx12_xmit_more: 9876175
tx12_cqes: 9129846276
tx13_packets: 7771739047
tx13_bytes: 9460232437077
tx13_tso_packets: 44625991
tx13_tso_bytes: 126064407221
tx13_csum_partial: 6517997408
tx13_nop: 548002615
tx13_mpwqe_blks: 954119763
tx13_mpwqe_pkts: 954121021
tx13_csum_none: 1189118295
tx13_xmit_more: 16528598
tx13_cqes: 7690586560
tx14_packets: 9327198615
tx14_bytes: 11444099882295
tx14_tso_packets: 53471737
tx14_tso_bytes: 148109721848
tx14_csum_partial: 7898024779
tx14_nop: 642659882
tx14_mpwqe_blks: 1085046860
tx14_mpwqe_pkts: 1085048744
tx14_csum_none: 1353706875
tx14_xmit_more: 18569646
tx14_cqes: 9233161893
tx15_packets: 7835137780
tx15_bytes: 9566187381553
tx15_tso_packets: 45092364
tx15_tso_bytes: 126782530801
tx15_csum_partial: 6585869227
tx15_nop: 568985005
tx15_mpwqe_blks: 950421681
tx15_mpwqe_pkts: 950423034
tx15_csum_none: 1184516888
tx15_xmit_more: 10002589
tx15_cqes: 7760383332
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15
4: 0 0 3611 0 0 0 0 0 0 0 0 0 0 0 0 0 IO-APIC 4-edge ttyS0
8: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IO-APIC 8-edge rtc0
9: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IO-APIC 9-fasteoi acpi
24: 0 150909051 1042065 149379077 0 70418610 14390803 56870401 112474 55662095 32669605 17360582 21023146 129188917 4326424 93861436 Hyper-V PCIe MSI 77309444096-edge mlx5_async0@pci:0240:00:02.0
25: 724 0 55 6591 1 63866 0 42887 0 38418 4263755032 46101 66 112622379 21734 39514 Hyper-V PCIe MSI 77309444097-edge mlx5_comp1@pci:0240:00:02.0
26: 0 2352 0 3502564496 0 29373 19601 18750 109210716 16588 41915 501087562 0 20743 10602 46259 Hyper-V PCIe MSI 77309444098-edge mlx5_comp2@pci:0240:00:02.0
27: 0 0 353 8706 0 477918495 1 3686124781 0 13104 62467 100598079 2557 43409 156502 3031 Hyper-V PCIe MSI 77309444099-edge mlx5_comp3@pci:0240:00:02.0
28: 0 0 53 5695 99494483 13620 31334 13007 3951675493 9642 91 10508 63364 33078 0 2337 Hyper-V PCIe MSI 77309444100-edge mlx5_comp4@pci:0240:00:02.0
29: 35 54 70 10047 37105 4040 30674 102167779 16292 10266 27833 16071 61248 4419 7340 10689 Hyper-V PCIe MSI 77309444101-edge mlx5_comp5@pci:0240:00:02.0
30: 0 262 14603 110211228 9501 3578839369 486165401 14199 10197 11604 8420 21814 1266 8833 9360 7790 Hyper-V PCIe MSI 77309444102-edge mlx5_comp6@pci:0240:00:02.0
31: 0 77 18869 24170 4231072938 15786 475 14496 9886 31192 4221 22148 33810 22291 203 109197486 Hyper-V PCIe MSI 77309444103-edge mlx5_comp7@pci:0240:00:02.0
32: 0 4077666168 0 0 0 0 0 415 0 0 0 1 0 0 0 0 Hyper-V PCIe MSI 77309444104-edge mlx5_comp8@pci:0240:00:02.0
33: 51497402 0 0 0 0 0 0 0 318 0 0 0 1 0 0 0 Hyper-V PCIe MSI 77309444105-edge mlx5_comp9@pci:0240:00:02.0
34: 0 0 0 62726 48 5775 43 5863 0 419937546 101252632 48454055 38105 1 43231 3512140862 Hyper-V PCIe MSI 77309444106-edge mlx5_comp10@pci:0240:00:02.0
35: 0 0 178561 2508 100 7998 54 956 0 19415 15045 3868725581 66 1560 516210967 6198 Hyper-V PCIe MSI 77309444107-edge mlx5_comp11@pci:0240:00:02.0
36: 0 46 0 502903455 101663 35353 3617230498 19949 39964 37889 31 25337 1970 1683 53 20272 Hyper-V PCIe MSI 77309444108-edge mlx5_comp12@pci:0240:00:02.0
37: 1 74 44 11848 197 24423 26 495532649 2547 3763079354 512 30262 23720 19792 9280 44505 Hyper-V PCIe MSI 77309444109-edge mlx5_comp13@pci:0240:00:02.0
38: 0 1 95909817 13699 239 16907 3609 18699 18534 19693 464 25842 19863 4013818918 339 17774 Hyper-V PCIe MSI 77309444110-edge mlx5_comp14@pci:0240:00:02.0
39: 0 108 4227 8941 213 109792671 120 38226 6573 15690 99 25456 11976 37372 3940671822 518488287 Hyper-V PCIe MSI 77309444111-edge mlx5_comp15@pci:0240:00:02.0
40: 0 0 4038730004 292 116746 1554 90264 4821 0 30480 0 5859 0 29772 105044927 368 Hyper-V PCIe MSI 77309444112-edge mlx5_comp16@pci:0240:00:02.0
NMI: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Non-maskable interrupts
LOC: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Local timer interrupts
SPU: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Performance monitoring interrupts
IWI: 1879699 1852710 1896834 1843913 1865220 1841230 1893002 1831720 1851141 1789618 1904516 1815972 1865510 1832217 1905321 1802954 IRQ work interrupts
RTR: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 APIC ICR read retries
RES: 55323547 43359062 52988190 44035099 54827070 55137018 51942970 46547190 53997978 48453664 55738975 51259701 55259879 55383211 67824842 45993474 Rescheduling interrupts
CAL: 1891135599 444941464 1914583516 475347799 1894799178 407572419 1954832920 527327518 1958504208 515751111 1963093799 518191684 1998610334 466211287 1915275988 513880738 Function call interrupts
TLB: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 TLB shootdowns
TRM: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Threshold APIC interrupts
DFR: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Deferred Error APIC interrupts
MCE: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Machine check exceptions
MCP: 3615 3615 3615 3615 3615 3615 3615 3615 3615 3615 3615 3615 3615 3615 3615 3615 Machine check polls
HYP: 1497791 5 48 673400 1087961 772567 0 32 0 11 49 13 3 12 15 7 Hypervisor callback interrupts
HRE: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Hyper-V reenlightenment interrupts
HVS: 1441310554 1351721198 1446527831 1352679477 1444434265 1348244463 1444986564 1353174480 1438403164 1340605813 1437722652 1347818598 1439039009 1350276615 1441701539 1350888170 Hyper-V stimer0 interrupts
ERR: 0
MIS: 0
PIN: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Posted-interrupt notification event
NPI: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Nested posted-interrupt event
PIW: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Posted-interrupt wakeup event
/sys/class/net/enP576s1/queues/tx-0/byte_queue_limits/inflight:184783
/sys/class/net/enP576s1/queues/tx-1/byte_queue_limits/inflight:455668
/sys/class/net/enP576s1/queues/tx-10/byte_queue_limits/inflight:139458
/sys/class/net/enP576s1/queues/tx-11/byte_queue_limits/inflight:81794
/sys/class/net/enP576s1/queues/tx-12/byte_queue_limits/inflight:25416
/sys/class/net/enP576s1/queues/tx-13/byte_queue_limits/inflight:179768
/sys/class/net/enP576s1/queues/tx-14/byte_queue_limits/inflight:112049
/sys/class/net/enP576s1/queues/tx-15/byte_queue_limits/inflight:169535
/sys/class/net/enP576s1/queues/tx-2/byte_queue_limits/inflight:144471
/sys/class/net/enP576s1/queues/tx-3/byte_queue_limits/inflight:128293
/sys/class/net/enP576s1/queues/tx-4/byte_queue_limits/inflight:528686
/sys/class/net/enP576s1/queues/tx-5/byte_queue_limits/inflight:140212
/sys/class/net/enP576s1/queues/tx-6/byte_queue_limits/inflight:235499
/sys/class/net/enP576s1/queues/tx-7/byte_queue_limits/inflight:30932
/sys/class/net/enP576s1/queues/tx-8/byte_queue_limits/inflight:92361
/sys/class/net/enP576s1/queues/tx-9/byte_queue_limits/inflight:397425
we see rx_congst_umr counter increased from 10021 to 15949 during that 1s.
The inflight numbers stay high, tx_packets did not increase.
Compared to nominal values, usually the rx_congst_umr does not increase, the inflight numbers are 0, and there is a higher number of interrupts and packets being processed.