I have purchased two NVIDIA ethernet adapters as follows :
- NVIDIA MCX653105A-HDAT-SP ConnectX-6 VPI Adapter card (upto 200Gbps)
I installed these adapters in two servers (one adapter in each server), and they are directly connected through an optical cable.
When testing with the xdpsock program, the server receiving packets halts. This issue depends on the version of the mlx5_core driver being used:
- When using the driver embedded in my OS, the halt problem does not occur.
- When using the newly installed driver, the halt problem occurs.
When the halt occurs, the system stops silently without any message or error log.
I would like to use the latest stable driver for my adapter and enable AF_XDP zerocopy traffic communication. How can I avoid the halt problem?
I would like to use the latest stable driver for my adapter and enable AF_XDP zerocopy traffic communication. How can I avoid the halt problem?
Below is my test environment:
// cat /etc/lsb-release
DISTRIB_DESCRIPTION=“Ubuntu 20.04.6 LTS”
// mlx5_core driver version embedded on OS which does not halt when receiving packet with AF_XDP mode
modinfo mlx5_core
…
vermagic: 5.4.0-200-generic SMP mod_unload modversions
// newly installed mlx5_core version which occurs halt problem when receiving packet with AF_XDP mode
modinfo mlx5_core
version: 24.10-0.7.0
// mlx5_core firmware version
mlx5_core 0000:ca:00.0: firmware version: 20.43.1014
// packet sending program which is running on one server with mlx5_core driver version embedded on OS
./xdpsock -i ens7 -t -N -z -q 63
// packet receiving program which is running on the other server with mlx5_core driver version which is newly installed.
ethtool -U ens7np0 flow-type udp4 src-port 4242 dst-port 4242 action 16 loc 0
./xdpsock -i ens7np0 -r -N -z -q 16
sock0@ens7np0:16 rxdrop xdp-drv
pps pkts 1.00
rx 0 0
tx 0 0
…
[Problem] ← Whenever the sending program start to send packet, the server receiving packet is halted immediately.
// installation method of new driver
DOCA Online Repository doca-all
Installation Instructions:
export DOCA_URL=“Index of /public/repo/doca/2.9.0/ubuntu20.04/x86_64”
curl https://linux.mellanox.com/public/repo/doca/GPG-KEY-Mellanox.pub | gpg --dearmor > /etc/apt/trusted.gpg.d/GPG-KEY-Mellanox.pub
echo “deb [signed-by=/etc/apt/trusted.gpg.d/GPG-KEY-Mellanox.pub] $DOCA_URL ./” > /etc/apt/sources.list.d/doca.list
sudo apt-get update
sudo apt-get -y install doca-all