# Odd, unsymmetric ib\_send\_lat results? **URL:** \ **Category:** Mellanox OFED\ **Tags:** uname, iterations, bytes\ **Created:** [December 11, 2018, 1:54pm UTC](https://forums.developer.nvidia.com/t/odd-unsymmetric-ib-send-lat-results/206979 "2018-12-11T13:54:47Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![fangchin](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@fangchin](https://forums.developer.nvidia.com/u/fangchin)\ **Post date:** [December 11, 2018, 1:54pm UTC](https://forums.developer.nvidia.com/t/odd-unsymmetric-ib-send-lat-results/206979/1 "2018-12-11T13:54:47Z")
I have two small IB clusters set up for testing: - Both have SB7700 IB switch - Two servers, each with a MCX455A-ECAT ConnectX-4 VPI adapter, are connected to each switch. Essential system and software info: [root@fs00 ~]# uname -a Linux fs00 3.10.0-327.22.2.el7.x86\_64 #1 SMP Thu Jun 23 17:05:11 UTC 2016 x86\_64 x86\_64 x86\_64 GNU/Linux [root@fs00 ~]# rpm -qa |grep ofed ofed-scripts-3.3-OFED.3.3.1.0.0.x86\_64 mlnx-ofed-all-3.3-1.0.0.0.noarch I have been testing the two clusters using ib\_send\_lat and observed the following that I don’t understand: **Cluster A:** **Server** : I got what the latency numbers that I anticipated. Reverse the role of client and server, results more or less the same. Again, that’s what I anticipated. [root@fs01 ~]# ib\_send\_lat -a -c UD * * * - Waiting for client to connect… \* * * * Max msg size in UD is MTU 4096 Changing to this MTU * * * Send Latency Test Dual-port : OFF Device : mlx5\_0 Number of qps : 1 Transport type : IB Connection type : UD Using SRQ : OFF RX depth : 1000 Mtu : 4096[B] Link type : IB Max inline data : 188[B] rdma\_cm QPs : OFF Data ex. method : Ethernet * * * local address: LID 0x02 QPN 0x0028 PSN 0xd0b6fe remote address: LID 0x03 QPN 0x0028 PSN 0x91642e * * * #bytes #iterations t\_min[usec] t\_max[usec] t\_typical[usec] 2 1000 0.81 4.47 0.83 4 1000 0.82 3.88 0.83 8 1000 0.81 2.95 0.83 16 1000 0.82 3.31 0.84 32 1000 0.88 3.40 0.90 64 1000 0.88 3.27 0.90 128 1000 0.91 3.54 0.93 256 1000 1.23 3.55 1.25 512 1000 1.29 4.17 1.32 1024 1000 1.49 3.15 1.51 2048 1000 1.72 4.32 1.74 4096 1000 2.15 4.32 2.20 * * * ---
**Author:** ![fangchin](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@fangchin](https://forums.developer.nvidia.com/u/fangchin)\ **Post date:** [December 11, 2018, 1:55pm UTC](https://forums.developer.nvidia.com/t/odd-unsymmetric-ib-send-lat-results/206979/2 "2018-12-11T13:55:09Z")
continued… **Client** : [root@fs00 ~]# ib\_send\_lat -a -c UD 192.168.11.151 Max msg size in UD is MTU 4096 Changing to this MTU * * * Send Latency Test Dual-port : OFF Device : mlx5\_0 Number of qps : 1 Transport type : IB Connection type : UD Using SRQ : OFF TX depth : 1 Mtu : 4096[B] Link type : IB Max inline data : 188[B] rdma\_cm QPs : OFF Data ex. method : Ethernet * * * local address: LID 0x03 QPN 0x0028 PSN 0x91642e remote address: LID 0x02 QPN 0x0028 PSN 0xd0b6fe * * * #bytes #iterations t\_min[usec] t\_max[usec] t\_typical[usec] 2 1000 0.81 8.37 0.83 4 1000 0.82 3.87 0.83 8 1000 0.81 2.97 0.83 16 1000 0.82 3.31 0.84 32 1000 0.88 3.41 0.89 64 1000 0.88 3.27 0.90 128 1000 0.91 3.55 0.93 256 1000 1.23 3.56 1.25 512 1000 1.30 4.15 1.32 1024 1000 1.48 3.17 1.51 2048 1000 1.72 4.32 1.74 4096 1000 2.16 4.32 2.20 * * * ---
**Author:** ![fangchin](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@fangchin](https://forums.developer.nvidia.com/u/fangchin)\ **Post date:** [December 11, 2018, 1:56pm UTC](https://forums.developer.nvidia.com/t/odd-unsymmetric-ib-send-lat-results/206979/3 "2018-12-11T13:56:18Z")
continued… local address: LID 0x03 QPN 0x002c PSN 0x8f46d remote address: LID 0x02 QPN 0x002c PSN 0x9c2fe5 * * * #bytes #iterations t\_min[usec] t\_max[usec] t\_typical[usec] 2 1000 0.76 5.30 0.78 4 1000 0.78 4.56 0.79 8 1000 0.76 3.80 0.78 16 1000 0.77 3.39 0.79 32 1000 0.83 3.07 0.84 64 1000 0.84 5.82 0.86 128 1000 0.86 3.95 0.88 256 1000 1.17 4.01 1.19 512 1000 1.25 4.64 1.27 1024 1000 1.45 3.70 1.46 2048 1000 1.67 5.21 1.70 4096 1000 2.13 4.72 2.16 * * * **Client:** [root@fs11 ~]# ib\_send\_lat -a -c UD 192.168.12.150 Max msg size in UD is MTU 4096 Changing to this MTU * * * Send Latency Test Dual-port : OFF Device : mlx5\_0 Number of qps : 1 Transport type : IB Connection type : UD Using SRQ : OFF TX depth : 1 Mtu : 4096[B] Link type : IB Max inline data : 188[B] rdma\_cm QPs : OFF Data ex. method : Ethernet * * * local address: LID 0x02 QPN 0x002c PSN 0x9c2fe5 remote address: LID 0x03 QPN 0x002c PSN 0x8f46d * * * #bytes #iterations t\_min[usec] t\_max[usec] t\_typical[usec] 2 1000 0.76 5.29 0.78 4 1000 0.77 4.57 0.79 8 1000 0.77 3.80 0.78 16 1000 0.77 3.38 0.79 32 1000 0.83 3.06 0.84 64 1000 0.84 5.77 0.86 128 1000 0.86 3.95 0.88 256 1000 1.17 3.97 1.19 512 1000 1.25 4.65 1.27 1024 1000 1.44 3.69 1.47 2048 1000 1.67 5.18 1.70 4096 1000 2.13 4.68 2.16 * * * I am very puzzled by the above outcome. Would appreciate any hints as to what I can do to figure out what’s causing the large latency. ---
**Author:** ![fangchin](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@fangchin](https://forums.developer.nvidia.com/u/fangchin)\ **Post date:** [December 11, 2018, 1:55pm UTC](https://forums.developer.nvidia.com/t/odd-unsymmetric-ib-send-lat-results/206979/4 "2018-12-11T13:55:35Z")
continued… **Cluster B:** As shown below, in **Direction I** , the client side max latency is about 10X larger. What’s odd is that once I reversed the role of client and server, both showed the latency numbers that I anticipated. **Server** : **Direct I** [root@fs11 ~]# ib\_send\_lat -a -c UD * * * - Waiting for client to connect… \* * * * Max msg size in UD is MTU 4096 Changing to this MTU * * * Send Latency Test Dual-port : OFF Device : mlx5\_0 Number of qps : 1 Transport type : IB Connection type : UD Using SRQ : OFF RX depth : 1000 Mtu : 4096[B] Link type : IB Max inline data : 188[B] rdma\_cm QPs : OFF Data ex. method : Ethernet * * * local address: LID 0x02 QPN 0x002b PSN 0x79fb69 remote address: LID 0x03 QPN 0x002b PSN 0xfbae7e * * * #bytes #iterations t\_min[usec] t\_max[usec] t\_typical[usec] 2 1000 0.76 4.93 0.78 4 1000 0.77 3.60 0.79 8 1000 0.76 4.16 0.78 16 1000 0.77 3.54 0.79 32 1000 0.83 3.60 0.85 64 1000 0.83 3.74 0.85 128 1000 0.86 3.52 0.88 256 1000 1.18 4.68 1.20 512 1000 1.25 3.88 1.27 1024 1000 1.44 4.71 1.46 2048 1000 1.68 4.20 1.70 4096 1000 2.13 3.91 2.16 * * * **Client** : [root@fs10 ~]# ib\_send\_lat -a -c UD 192.168.12.151 Max msg size in UD is MTU 4096 Changing to this MTU * * * ---
**Author:** ![fangchin](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@fangchin](https://forums.developer.nvidia.com/u/fangchin)\ **Post date:** [December 11, 2018, 1:57pm UTC](https://forums.developer.nvidia.com/t/odd-unsymmetric-ib-send-lat-results/206979/5 "2018-12-11T13:57:05Z")
reply… I ran the ib\_send\_lat in each host, so client and server were in the same host. The numbers all look reasonable. Now both cables and the switch for the subnet are not involved. Any suggestion to what to test to narrow down the causes of the spikes? When you suggested a loopback test, did you mean to test the two cables? I.e. take each cable, loop-back between two ports, see if the link comes up? I do have SM running on the switch. BTW, I have updated both MLNX\_OFED and firmware for HCAs to the latest. An example from running hca\_self\_test.ofed ---- Performing Adapter Device Self Test ---- Number of CAs Detected … 1 PCI Device Check … PASS Kernel Arch … x86\_64 Host Driver Version … MLNX\_OFED\_LINUX-3.3-1.0.4.0 (OFED-3.3-1.0.4): modules Host Driver RPM Check … PASS Firmware on CA #0 HCA … v12.16.1006 Firmware Check on CA #0 (HCA) … NA REASON: NO required fw version Host Driver Initialization … PASS Number of CA Ports Active … 1 Port State of Port #1 on CA #0 (HCA)… UP 4X EDR (InfiniBand) Error Counter Check on CA #0 (HCA)… PASS Kernel Syslog Check … PASS Node GUID on CA #0 (HCA) … 7c:fe:90:03:00:29:26:b6 ------------------ DONE --------------------- I repeated the tests that I did before, I still observed spikes. Host fs10 **Server:** [root@fs10 ~]# ib\_send\_lat -a -c UD * * * - Waiting for client to connect… \* * * * Max msg size in UD is MTU 4096 Changing to this MTU * * * Send Latency Test Dual-port : OFF Device : mlx5\_0 Number of qps : 1 Transport type : IB Connection type : UD Using SRQ : OFF RX depth : 1000 Mtu : 4096[B] Link type : IB Max inline data : 188[B] rdma\_cm QPs : OFF Data ex. method : Ethernet * * * local address: LID 0x03 QPN 0x0031 PSN 0xc0fb9e remote address: LID 0x03 QPN 0x0030 PSN 0x3bd5ea * * * #bytes #iterations t\_min[usec] t\_max[usec] t\_typical[usec] 2 1000 0.67 4.41 0.69 4 1000 0.67 4.77 0.69 8 1000 0.67 4.77 0.69 16 1000 0.67 4.28 0.69 32 1000 0.71 4.93 0.72 64 1000 0.71 5.22 0.72 128 1000 0.75 4.80 0.76 256 1000 1.06 4.20 1.08 512 1000 1.14 4.79 1.16 1024 1000 1.27 5.08 1.29 2048 1000 1.54 5.62 1.55 4096 1000 2.04 5.71 2.06 * * * ---
**Author:** ![fangchin](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@fangchin](https://forums.developer.nvidia.com/u/fangchin)\ **Post date:** [December 11, 2018, 1:55pm UTC](https://forums.developer.nvidia.com/t/odd-unsymmetric-ib-send-lat-results/206979/6 "2018-12-11T13:55:57Z")
continued… Send Latency Test Dual-port : OFF Device : mlx5\_0 Number of qps : 1 Transport type : IB Connection type : UD Using SRQ : OFF TX depth : 1 Mtu : 4096[B] Link type : IB Max inline data : 188[B] rdma\_cm QPs : OFF Data ex. method : Ethernet * * * local address: LID 0x03 QPN 0x002a PSN 0x544e64 remote address: LID 0x02 QPN 0x002a PSN 0x7babed * * * #bytes #iterations t\_min[usec] t\_max[usec] t\_typical[usec] 2 1000 0.76 **45.78** 0.78 4 1000 0.77 **30.98** 0.79 8 1000 0.76 **37.99** 0.78 16 1000 0.77 **43.70** 0.79 32 1000 0.83 **47.34** 0.85 64 1000 0.84 **39.94** 0.86 128 1000 0.86 **41.16** 0.88 256 1000 1.18 **37.54** 1.20 512 1000 1.24 **42.94** 1.26 1024 1000 1.43 **39.50** 1.45 2048 1000 1.66 **42.06** 1.69 4096 1000 2.11 **40.37** 2.15 * * * **Direct II** **Server:** [root@fs10 ~]# ib\_send\_lat -a -c UD * * * - Waiting for client to connect… \* * * * Max msg size in UD is MTU 4096 Changing to this MTU * * * Send Latency Test Dual-port : OFF Device : mlx5\_0 Number of qps : 1 Transport type : IB Connection type : UD Using SRQ : OFF RX depth : 1000 Mtu : 4096[B] Link type : IB Max inline data : 188[B] rdma\_cm QPs : OFF Data ex. method : Ethernet * * * ---
**Author:** ![fangchin](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@fangchin](https://forums.developer.nvidia.com/u/fangchin)\ **Post date:** [December 11, 2018, 1:57pm UTC](https://forums.developer.nvidia.com/t/odd-unsymmetric-ib-send-lat-results/206979/7 "2018-12-11T13:57:31Z")
continued… **Client** : [root@fs10 ~]# ib\_send\_lat -a -c UD 192.168.12.150 Max msg size in UD is MTU 4096 Changing to this MTU * * * Send Latency Test Dual-port : OFF Device : mlx5\_0 Number of qps : 1 Transport type : IB Connection type : UD Using SRQ : OFF TX depth : 1 Mtu : 4096[B] Link type : IB Max inline data : 188[B] rdma\_cm QPs : OFF Data ex. method : Ethernet * * * local address: LID 0x03 QPN 0x0030 PSN 0x3bd5ea remote address: LID 0x03 QPN 0x0031 PSN 0xc0fb9e * * * #bytes #iterations t\_min[usec] t\_max[usec] t\_typical[usec] 2 1000 0.67 5.92 0.68 4 1000 0.67 4.93 0.69 8 1000 0.67 4.80 0.68 16 1000 0.67 4.29 0.69 32 1000 0.70 4.95 0.72 64 1000 0.70 5.21 0.72 128 1000 0.75 4.81 0.76 256 1000 1.07 4.20 1.08 512 1000 1.14 4.80 1.16 1024 1000 1.27 5.08 1.29 2048 1000 1.53 5.63 1.55 4096 1000 2.04 5.70 2.06 * * * Host fs11 **Server:** [root@fs11 ~]# ib\_send\_lat -a -c UD * * * - Waiting for client to connect… \* * * * Max msg size in UD is MTU 4096 Changing to this MTU * * * Send Latency Test Dual-port : OFF Device : mlx5\_0 Number of qps : 1 Transport type : IB Connection type : UD Using SRQ : OFF RX depth : 1000 Mtu : 4096[B] Link type : IB Max inline data : 188[B] rdma\_cm QPs : OFF Data ex. method : Ethernet * * * local address: LID 0x02 QPN 0x002d PSN 0xf82dfe remote address: LID 0x02 QPN 0x002c PSN 0x49619e * * * #bytes #iterations t\_min[usec] t\_max[usec] t\_typical[usec] 2 1000 0.68 2.60 0.69 4 1000 0.67 2.34 0.69 8 1000 0.67 2.06 0.69 16 1000 0.68 1.95 0.69 32 1000 0.71 1.83 0.72 64 1000 0.71 1.82 0.72 128 1000 0.75 1.91 0.76 256 1000 1.07 3.26 1.09 512 1000 1.14 2.50 1.15 1024 1000 1.28 2.71 1.30 2048 1000 1.54 2.83 1.56 4096 1000 2.05 2.76 2.07 * * * ---
**Author:** ![fangchin](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@fangchin](https://forums.developer.nvidia.com/u/fangchin)\ **Post date:** [December 11, 2018, 1:57pm UTC](https://forums.developer.nvidia.com/t/odd-unsymmetric-ib-send-lat-results/206979/8 "2018-12-11T13:57:49Z")
continued… **Client:** [root@fs11 ~]# ib\_send\_lat -a -c UD 192.168.12.151 Max msg size in UD is MTU 4096 Changing to this MTU * * * Send Latency Test Dual-port : OFF Device : mlx5\_0 Number of qps : 1 Transport type : IB Connection type : UD Using SRQ : OFF TX depth : 1 Mtu : 4096[B] Link type : IB Max inline data : 188[B] rdma\_cm QPs : OFF Data ex. method : Ethernet * * * local address: LID 0x02 QPN 0x002c PSN 0x49619e remote address: LID 0x02 QPN 0x002d PSN 0xf82dfe * * * #bytes #iterations t\_min[usec] t\_max[usec] t\_typical[usec] 2 1000 0.67 5.75 0.69 4 1000 0.67 2.37 0.69 8 1000 0.67 5.52 0.69 16 1000 0.67 1.86 0.69 32 1000 0.70 2.01 0.72 64 1000 0.71 1.85 0.72 128 1000 0.75 1.90 0.76 256 1000 1.06 5.13 1.08 512 1000 1.13 2.27 1.15 1024 1000 1.28 2.74 1.30 2048 1000 1.53 2.86 1.56 4096 1000 2.04 6.10 2.07 * * * ---
**Author:** ![fangchin](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@fangchin](https://forums.developer.nvidia.com/u/fangchin)\ **Post date:** [December 11, 2018, 1:58pm UTC](https://forums.developer.nvidia.com/t/odd-unsymmetric-ib-send-lat-results/206979/9 "2018-12-11T13:58:21Z")
continued… I also ran **ibdiagnet -pc -lw 4x -ls 25 -P all=1 --pm\_pause\_time 600 --get\_cable\_info** on all the four servers. All resulted * * * Summary -I- Stage Warnings Errors Comment -I- Discovery 0 0 -I- Lids Check 0 0 -I- Links Check 0 0 -I- Subnet Manager 0 0 -I- Port Counters 0 0 -I- Nodes Information 0 0 -I- Speed / Width checks 0 0 -I- Partition Keys 0 0 -I- Alias GUIDs 0 0 -I- Temperature Sensing 0 0 -I- Cable Diagnostic (Plugin) 0 0 ---
**Author:** ![fangchin](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@fangchin](https://forums.developer.nvidia.com/u/fangchin)\ **Post date:** [December 11, 2018, 1:58pm UTC](https://forums.developer.nvidia.com/t/odd-unsymmetric-ib-send-lat-results/206979/10 "2018-12-11T13:58:29Z")
reply… 6 days ago, I reported that I fixed my IPoIB setup. I just found sometime a while ago to revisit this issue. Indeed, as I suspected, the original, incorrect IPoIB setup was the cause of jitters observed, most likely due to the fact that for 10G Ethernet ports on servers, I have been using some inexpensive third party SFP+ DACs So, problem solved for now. ---
**Author:** ![alekseys1](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@alekseys1](https://forums.developer.nvidia.com/u/alekseys1)\ **Post date:** [July 29, 2016, 2:09pm UTC](https://forums.developer.nvidia.com/t/odd-unsymmetric-ib-send-lat-results/206979/11 "2016-07-29T14:09:24Z")
The last column is the most important as it shows the typical value, and these values are close. One of the hosts, probably, has something different in the settings -BIOS, OS tuning, some daemon running, management interrupts and it cause to spikes in some iterations. Try to run loopback tests. ---
**Author:** ![fangchin](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@fangchin](https://forums.developer.nvidia.com/u/fangchin)\ **Post date:** [July 30, 2016, 6:24pm UTC](https://forums.developer.nvidia.com/t/odd-unsymmetric-ib-send-lat-results/206979/12 "2016-07-30T18:24:00Z")
I ran the ib\_send\_lat in each host, so client and server were in the same host. The numbers all look reasonable. Now both cables and the switch for the subnet are not involved. Any suggestion to what to test to narrow down the causes of the spikes? When you suggested a loopback test, did you mean to test the two cables? I.e. take each cable, loop-back between two ports, see if the link comes up? I do have SM running on the switch. BTW, I have updated both MLNX\_OFED and firmware for HCAs to the latest. An example from running hca\_self\_test.ofed ---- Performing Adapter Device Self Test ---- Number of CAs Detected … 1 PCI Device Check … PASS Kernel Arch … x86\_64 Host Driver Version … MLNX\_OFED\_LINUX-3.3-1.0.4.0 (OFED-3.3-1.0.4): modules Host Driver RPM Check … PASS Firmware on CA #0 HCA … v12.16.1006 Firmware Check on CA #0 (HCA) … NA REASON: NO required fw version Host Driver Initialization … PASS Number of CA Ports Active … 1 Port State of Port #1 on CA #0 (HCA)… UP 4X EDR (InfiniBand) Error Counter Check on CA #0 (HCA)… PASS Kernel Syslog Check … PASS Node GUID on CA #0 (HCA) … 7c:fe:90:03:00:29:26:b6 ------------------ DONE --------------------- I repeated the tests that I did before, I still observed spikes. Host fs10 **Server:** [root@fs10 ~]# ib\_send\_lat -a -c UD * * * - Waiting for client to connect… \* * * * Max msg size in UD is MTU 4096 Changing to this MTU * * * Send Latency Test Dual-port : OFF Device : mlx5\_0 Number of qps : 1 Transport type : IB Connection type : UD Using SRQ : OFF RX depth : 1000 Mtu : 4096[B] Link type : IB Max inline data : 188[B] rdma\_cm QPs : OFF Data ex. method : Ethernet * * * local address: LID 0x03 QPN 0x0031 PSN 0xc0fb9e remote address: LID 0x03 QPN 0x0030 PSN 0x3bd5ea * * * #bytes #iterations t\_min[usec] t\_max[usec] t\_typical[usec] 2 1000 0.67 4.41 0.69 4 1000 0.67 4.77 0.69 8 1000 0.67 4.77 0.69 16 1000 0.67 4.28 0.69 32 1000 0.71 4.93 0.72 64 1000 0.71 5.22 0.72 128 1000 0.75 4.80 0.76 256 1000 1.06 4.20 1.08 512 1000 1.14 4.79 1.16 1024 1000 1.27 5.08 1.29 2048 1000 1.54 5.62 1.55 4096 1000 2.04 5.71 2.06 * * * **Client** : [root@fs10 ~]# ib\_send\_lat -a -c UD 192.168.12.150 Max msg size in UD is MTU 4096 Changing to this MTU * * * Send Latency Test Dual-port : OFF Device : mlx5\_0 Number of qps : 1 Transport type : IB Connection type : UD Using SRQ : OFF TX depth : 1 Mtu : 4096[B] Link type : IB Max inline data : 188[B] rdma\_cm QPs : OFF Data ex. method : Ethernet * * * local address: LID 0x03 QPN 0x0030 PSN 0x3bd5ea remote address: LID 0x03 QPN 0x0031 PSN 0xc0fb9e * * * #bytes #iterations t\_min[usec] t\_max[usec] t\_typical[usec] 2 1000 0.67 5.92 0.68 4 1000 0.67 4.93 0.69 8 1000 0.67 4.80 0.68 16 1000 0.67 4.29 0.69 32 1000 0.70 4.95 0.72 64 1000 0.70 5.21 0.72 128 1000 0.75 4.81 0.76 256 1000 1.07 4.20 1.08 512 1000 1.14 4.80 1.16 1024 1000 1.27 5.08 1.29 2048 1000 1.53 5.63 1.55 4096 1000 2.04 5.70 2.06 * * * Host fs11 **Server:** [root@fs11 ~]# ib\_send\_lat -a -c UD * * * - Waiting for client to connect… \* * * * Max msg size in UD is MTU 4096 Changing to this MTU * * * Send Latency Test Dual-port : OFF Device : mlx5\_0 Number of qps : 1 Transport type : IB Connection type : UD Using SRQ : OFF RX depth : 1000 Mtu : 4096[B] Link type : IB Max inline data : 188[B] rdma\_cm QPs : OFF Data ex. method : Ethernet * * * local address: LID 0x02 QPN 0x002d PSN 0xf82dfe remote address: LID 0x02 QPN 0x002c PSN 0x49619e * * * #bytes #iterations t\_min[usec] t\_max[usec] t\_typical[usec] 2 1000 0.68 2.60 0.69 4 1000 0.67 2.34 0.69 8 1000 0.67 2.06 0.69 16 1000 0.68 1.95 0.69 32 1000 0.71 1.83 0.72 64 1000 0.71 1.82 0.72 128 1000 0.75 1.91 0.76 256 1000 1.07 3.26 1.09 512 1000 1.14 2.50 1.15 1024 1000 1.28 2.71 1.30 2048 1000 1.54 2.83 1.56 4096 1000 2.05 2.76 2.07 * * * **Client:** [root@fs11 ~]# ib\_send\_lat -a -c UD 192.168.12.151 Max msg size in UD is MTU 4096 Changing to this MTU * * * Send Latency Test Dual-port : OFF Device : mlx5\_0 Number of qps : 1 Transport type : IB Connection type : UD Using SRQ : OFF TX depth : 1 Mtu : 4096[B] Link type : IB Max inline data : 188[B] rdma\_cm QPs : OFF Data ex. method : Ethernet * * * local address: LID 0x02 QPN 0x002c PSN 0x49619e remote address: LID 0x02 QPN 0x002d PSN 0xf82dfe * * * #bytes #iterations t\_min[usec] t\_max[usec] t\_typical[usec] 2 1000 0.67 5.75 0.69 4 1000 0.67 2.37 0.69 8 1000 0.67 5.52 0.69 16 1000 0.67 1.86 0.69 32 1000 0.70 2.01 0.72 64 1000 0.71 1.85 0.72 128 1000 0.75 1.90 0.76 256 1000 1.06 5.13 1.08 512 1000 1.13 2.27 1.15 1024 1000 1.28 2.74 1.30 2048 1000 1.53 2.86 1.56 4096 1000 2.04 6.10 2.07 * * * ---
**Author:** ![fangchin](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@fangchin](https://forums.developer.nvidia.com/u/fangchin)\ **Post date:** [July 31, 2016, 7:42am UTC](https://forums.developer.nvidia.com/t/odd-unsymmetric-ib-send-lat-results/206979/13 "2016-07-31T07:42:47Z")
I also ran **ibdiagnet -pc -lw 4x -ls 25 -P all=1 --pm\_pause\_time 600 --get\_cable\_info** on all the four servers. All resulted * * * Summary -I- Stage Warnings Errors Comment -I- Discovery 0 0 -I- Lids Check 0 0 -I- Links Check 0 0 -I- Subnet Manager 0 0 -I- Port Counters 0 0 -I- Nodes Information 0 0 -I- Speed / Width checks 0 0 -I- Partition Keys 0 0 -I- Alias GUIDs 0 0 -I- Temperature Sensing 0 0 -I- Cable Diagnostic (Plugin) 0 0 ---
**Author:** ![alekseys1](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@alekseys1](https://forums.developer.nvidia.com/u/alekseys1)\ **Post date:** [August 1, 2016, 2:31pm UTC](https://forums.developer.nvidia.com/t/odd-unsymmetric-ib-send-lat-results/206979/14 "2016-08-01T14:31:34Z")
You should take a look on the typical value (the right column) , which is good. There is always a chance that one of the iterations can have higher value then typical. ---
**Author:** ![fangchin](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@fangchin](https://forums.developer.nvidia.com/u/fangchin)\ **Post date:** [August 22, 2016, 7:29pm UTC](https://forums.developer.nvidia.com/t/odd-unsymmetric-ib-send-lat-results/206979/15 "2016-08-22T19:29:21Z")
6 days ago, I reported that I fixed my IPoIB setup. I just found sometime a while ago to revisit this issue. Indeed, as I suspected, the original, incorrect IPoIB setup was the cause of jitters observed, most likely due to the fact that for 10G Ethernet ports on servers, I have been using some inexpensive third party SFP+ DACs So, problem solved for now.