Ib_write_bw test - why sweeping through all sizes is retricted to 8MB?

Infrastructure & Networking Software And Drivers Mellanox OFED
1

why is the message size retricted to 8MB when we sweep through all the sizes in the ib_write_bw test? any specific reason for this?

Calling ctx_find_dev()...                                                                                                                                                
Calling ibv_open_device()...                                                                                                                                             
Calling check_link()...                                                                                                                                                  
Calling create_comm_struct()...                                                                                                                                          
Connected: Mon Sep  2 23:45:46 2024                                                                                                                                      
                                                                                                                                                                         
---------------------------------------------------------------------------------------                                                                                  
                    RDMA_Write BW Test                                                                                                                                   
 Dual-port       : OFF          Device         : roce_ai0                                                                                                                
 Number of qps   : 1            Transport type : IB                                                                                                                      
 Connection type : RC           Using SRQ      : OFF                                                                                                                     
 TX depth        : 5                                                                                                                                                     
 CQ Moderation   : 5                                                                                                                                                     
 CPU freq        : 2994[MHz]                                                                                                                                             
 Mtu             : 4096[B]                                                                                                                                               
 Link type       : Ethernet                                                                                                                                              
 GID index       : 1                                                                                                                                                     
 rdma_cm QPs     : OFF                                                                                                                                                   
 Data ex. method : Ethernet                                                                                                                                              
---------------------------------------------------------------------------------------                                                                                  
 local address:  LID 0000 QPN 0x0004 PSN 0x776db2 RKey 0x000303 VAddr 0x007f9c62084000                                                                                   
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:40:02:01:03                                                                                                                  
 remote address: LID 0000 QPN 0x0004 PSN 0x011e0a RKey 0x000303 VAddr 0x007f2f0e486000                                                                                   
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:40:02:01:02                                                                                                                  
---------------------------------------------------------------------------------------                                                                                  
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]                                                                                         
 2          5           0.001513            0.001513            0.094574                                                                                                 
 4          5           0.009852            0.009852            0.307878                                                                                                 
 8          5           0.019839            0.019839            0.309981                                                                                                 
 16         5           0.040076            0.040076            0.313093                                                                                                 
 32         5           0.079851            0.079851            0.311918                                                                                                 
 64         5                0.11               0.11               0.212732                                                                                              
 128        5                0.22               0.22               0.214193                                                                                              
 256        5                0.63               0.63               0.309405                                                                                              
 512        5                0.92               0.92               0.224705                                                                                              
 1024       5                1.63               1.63               0.198437                                                                                              
 2048       5                4.77               4.77               0.291342                                                                                              
 4096       5                8.19               8.19               0.250035                                                                                              
 8192       5                12.02              12.02              0.183414                                                                                              
 16384      5                16.16              16.16              0.123288                                                                                              
 32768      5                18.91              18.91              0.072130                                                                                              
 65536      5                20.67              20.67              0.039418                                                                                              
 131072     5                21.61              21.61              0.020612                                                                                              
 262144     5                22.12              22.12              0.010550                                                                                              
 524288     5                18.61              18.61              0.004436                                                                                              
 1048576    5                20.33              20.33              0.002424                                                                                              
 2097152    5                22.50              22.50              0.001341                                                                                              
 4194304    5                21.51              21.51              0.000641                                                                                              
 8388608    5                20.26              20.26              0.000302                                                                                              
---------------------------------------------------------------------------------------                                                                                  
Completed: Mon Sep  2 23:45:53 2024
2

The message size restriction to 8MB in the ib_write_bw test or similar benchmarks/tests for InfiniBand (IB) networks is often due to a combination of factors including hardware limitations, software configurations, and test design considerations. Here are some reasons why this limit might be in place:

1.‌Hardware Limitations‌: InfiniBand networks, like any other network technology, have physical limitations on the size of data packets that can be transmitted. These limitations are set by the hardware design and are intended to balance between data throughput, latency, and network efficiency.

2.‌Memory Management‌: Larger message sizes can lead to increased memory usage and management overhead, both on the sender and receiver sides. Limiting the message size can help manage memory resources more efficiently, especially in high-performance, low-latency applications.

3.‌Test Design‌: The ib_write_bw test is designed to measure the bandwidth of an InfiniBand link under different conditions. By sweeping through different message sizes, the test aims to provide insights into how the network performs with varying loads. Limiting the maximum message size allows for a more controlled and focused test, ensuring that the results are meaningful and relevant.

4.‌Software Configurations‌: The software stack implementing the InfiniBand protocol might have its own limitations on message sizes. These limitations could be due to buffer management, flow control, or other protocol-specific considerations.

5.‌Compatibility and Standardization‌: Standardizing on a maximum message size can help ensure compatibility between different InfiniBand implementations and devices. This compatibility is crucial for maintaining a stable and reliable network ecosystem.

In summary, the 8MB limit in the ib_write_bw test is likely a result of careful consideration of hardware capabilities, software design, test objectives, and the need for compatibility and standardization within the InfiniBand ecosystem.