Ib_write speed loss

[tnhb000]

my os is ubuntu22.04 ,This problem has occurred.

[tnhb000]

but in centos7.9,speed is correct,like this :

[Levei_Luo]

hi,
which NIC is being used ?

Anyway, try to use numactl/taskset to bind the local numa core
and use ‘-x’ to specify the RoceV2 GID.

1. use ‘mst status -v’ to check the NIC belongs to which NUMA node

2. use ‘lscpu’ to query cpu core belonging to which NUMA

3. use ‘show_gids’ query the IPv4 based RoceV2.

ex:
[server]# show_gids
DEV PORT INDEX GID IPv4 VER DEV

---

mlx5_0 1 0 fe80:0000:0000:0000:a288:c2ff:fead:cda0 v1 eth0
mlx5_0 1 1 fe80:0000:0000:0000:a288:c2ff:fead:cda0 v2 eth0
mlx5_0 1 2 0000:0000:0000:0000:0000:ffff:c0a8:000b 192.168.0.11 v1 eth0
mlx5_0 1 3 0000:0000:0000:0000:0000:ffff:c0a8:000b 192.168.0.11 v2 eth0

here should select GID 3 ( the record includes ip’192.168.0.11’ and v2 here).

4. use numactl or taskset to bind local numa core
   also use ‘-x’ to specify the GID.

ex:
numactl --cpunodebind=0 ib_send_bw -d mlx5_0 -x 3 -F --report_gbits 192.168.0.11 -a

or

taskset -c 11 ib_send_bw -d mlx5_0 -x 3 -F --report_gbits 192.168.0.11 -a

Regards,
Levei