Packet stuck in Buffer in Connect-4 LX

Sending UDP packets through a server and 1 packet gets stuck in the buffer.
Eg send 1, it gets stuck, send 100, 99 gets transmitted with 1 stuck.
We can see the packet in TCPdump but the switch on the remote side does not receive it until either a new packet comes in or we bring down the interface…
Server1 → em2 UUT p2p1 [Connectx-4-Lx] → Switch → Server 2

If we change to another interface using a ConnectX-5 card no problems.

ethtool -S between the two cards doesn’t show anything different.

CentOS Linux release 7.9.2009 (Core)
Kernel 3.10.0-1160.2.2.el7.x86_64

*-network:0
description: Ethernet interface
product: MT27710 Family [ConnectX-4 Lx]
vendor: Mellanox Technologies
logical name: p2p1
version: 00
serial: 0c:42:a1:e9:9F:10
size: 1Gbit/s
width: 64 bits
clock: 33MHz
capabilities: bus_master cap_list rom ethernet physical tp autonegotiation
configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=5.0-0 duplex=full firmware=14.27.1016 (MT_2420110004) ip=192.168.18.1 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s

*-network:1
description: Ethernet interface
product: MT27800 Family [ConnectX-5]
vendor: Mellanox Technologies
logical name: em4
version: 00
serial: 1c:34:da:6f:d2:11
size: 1Gbit/s
width: 64 bits
clock: 33MHz
capabilities: bus_master cap_list rom ethernet physical tp autonegotiation
configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=5.0-0 duplex=full firmware=16.27.6106 (DEL0000000016) ip=192.168.20.1 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s

example of udp packet
tcpdump: listening on p3p2, link-type EN10MB (Ethernet), capture size 262144 bytes
19:56:09.866719 IP (tos 0x0, ttl 254, id 2640, offset 0, flags [DF], proto UDP (17), length 80)
192.168.167.101.mmcc > 192.168.200.102.mmcc: [udp sum ok] UDP, length 52

Any thoughts on what I am missing , some setting?