Hi guys:
I am currently conducting research and using ib_write_bw and ib_write_lat
I am using EDR x 4.
From my understanding, EDR x 4 uses 64/66 encoding on the physical link, and the signaling Bandwidth is 25.78Gbps * 4, making the theoretical bandwidth 25.78Gbps * 4 * (64/66) = 100Gbps
However, when I was doing ib_write_bw, I noticed the bandwidth ceiling is 96.6Gbps instead of 100Gbps. I know usually people are satisfied with 96Gbps, but just out of curiosity. Why?
Or is the actual signaling BW for EDR 25Gbps, instead of 25.78Gbps? In that case after applying encoding loss we get 25Gbps * 4 * (64/66) = 96.6Gbps
Or is it because of processing overhead?
The latency of sending 2097152B in ib_write_lat test is ~176us, but I am also wondering isn’t the theoretical latency of sending 2097152B in 100Gbps: 1s/(((100Gbps/8) * 1024 * 1024 * 1024)/2097152) = 156us?
Where is the middle 176us - 156us = 20us goes to?
Is there any methods I could find out?
We are doing usec level datacenter application related research, so every usec is pretty important to us.
~> ib_write_bw -d mlx5_1 -a -F --report_gbits --disable_pcie_relaxed 192.17.103.4
---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : mlx5_1
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: OFF
ibv_wr* API : ON
TX depth : 128
CQ Moderation : 100
Mtu : 4096[B]
Link type : IB
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x02 QPN 0x0154 PSN 0x76f7ac RKey 0x1bdfdf VAddr 0x007f3e25da2000
remote address: LID 0x01 QPN 0x0154 PSN 0xc8c064 RKey 0x1c0100 VAddr 0x007f8d8d6ee000
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
2 5000 0.14 0.11 6.688079
4 5000 0.28 0.25 7.727968
8 5000 0.55 0.47 7.413184
16 5000 1.11 1.00 7.773782
32 5000 2.23 1.99 7.782522
64 5000 4.45 3.99 7.783585
128 5000 8.90 7.58 7.404378
256 5000 17.81 15.68 7.653848
512 5000 35.31 30.64 7.479293
1024 5000 70.02 57.50 7.018787
2048 5000 94.71 87.01 5.310566
4096 5000 95.81 93.54 2.854465
8192 5000 95.67 94.43 1.440854
16384 5000 95.88 95.73 0.730324
32768 5000 96.08 96.07 0.366466
65536 5000 96.38 96.38 0.183824
131072 5000 96.16 96.13 0.091677
262144 5000 96.39 96.39 0.045962
524288 5000 96.39 96.37 0.022976
1048576 5000 96.52 96.52 0.011506
2097152 5000 96.52 96.51 0.005753
4194304 5000 96.49 96.49 0.002876
8388608 5000 96.50 96.50 0.001438
---------------------------------------------------------------------------------------
Appreciate so much,
Jerry