About NCCL benchmark result

I have used 3 node of HGX(Insuper SMX4 A100 40G * 8ea) with four HDR Infiniband card(Mellanox) for nccl test.

I got a result like as bellow.

ubuntu@hpc-pr-a-pod01:~/nccl_test$ mpirun -mca pml ucx -np 24 --hostfile /home/ubuntu/nccl_test/hostfile /home/ubuntu/nccl_test/nccl-tests/build/all_reduce_perf -b 8 -e 256M -g 1

WARNING: There was an error initializing an OpenFabrics device.

Local host: hpc-pr-a-pod10
Local device: mlx5_0

nThread 1 nGpus 1 minBytes 8 maxBytes 268435456 step: 1048576(bytes) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0

Using devices

Rank 0 Group 0 Pid 3552208 on hpc-pr-a-pod01 device 0 [0x10] NVIDIA A100-SXM4-40GB

Rank 1 Group 0 Pid 3552209 on hpc-pr-a-pod01 device 1 [0x16] NVIDIA A100-SXM4-40GB

Rank 2 Group 0 Pid 3552210 on hpc-pr-a-pod01 device 2 [0x49] NVIDIA A100-SXM4-40GB

Rank 3 Group 0 Pid 3552211 on hpc-pr-a-pod01 device 3 [0x4d] NVIDIA A100-SXM4-40GB

Rank 4 Group 0 Pid 3552212 on hpc-pr-a-pod01 device 4 [0x8a] NVIDIA A100-SXM4-40GB

Rank 5 Group 0 Pid 3552213 on hpc-pr-a-pod01 device 5 [0x8f] NVIDIA A100-SXM4-40GB

Rank 6 Group 0 Pid 3552214 on hpc-pr-a-pod01 device 6 [0xc6] NVIDIA A100-SXM4-40GB

Rank 7 Group 0 Pid 3552218 on hpc-pr-a-pod01 device 7 [0xca] NVIDIA A100-SXM4-40GB

Rank 8 Group 0 Pid 1736754 on hpc-pr-a-pod03 device 0 [0x10] NVIDIA A100-SXM4-40GB

Rank 9 Group 0 Pid 1736755 on hpc-pr-a-pod03 device 1 [0x16] NVIDIA A100-SXM4-40GB

Rank 10 Group 0 Pid 1736756 on hpc-pr-a-pod03 device 2 [0x49] NVIDIA A100-SXM4-40GB

Rank 11 Group 0 Pid 1736757 on hpc-pr-a-pod03 device 3 [0x4d] NVIDIA A100-SXM4-40GB

Rank 12 Group 0 Pid 1736758 on hpc-pr-a-pod03 device 4 [0x8a] NVIDIA A100-SXM4-40GB

Rank 13 Group 0 Pid 1736759 on hpc-pr-a-pod03 device 5 [0x8f] NVIDIA A100-SXM4-40GB

Rank 14 Group 0 Pid 1736761 on hpc-pr-a-pod03 device 6 [0xc6] NVIDIA A100-SXM4-40GB

Rank 15 Group 0 Pid 1736765 on hpc-pr-a-pod03 device 7 [0xca] NVIDIA A100-SXM4-40GB

Rank 16 Group 0 Pid 2190460 on hpc-pr-a-pod10 device 0 [0x10] NVIDIA A100-SXM4-40GB

Rank 17 Group 0 Pid 2190461 on hpc-pr-a-pod10 device 1 [0x16] NVIDIA A100-SXM4-40GB

Rank 18 Group 0 Pid 2190462 on hpc-pr-a-pod10 device 2 [0x49] NVIDIA A100-SXM4-40GB

Rank 19 Group 0 Pid 2190463 on hpc-pr-a-pod10 device 3 [0x4d] NVIDIA A100-SXM4-40GB

Rank 20 Group 0 Pid 2190464 on hpc-pr-a-pod10 device 4 [0x8a] NVIDIA A100-SXM4-40GB

Rank 21 Group 0 Pid 2190465 on hpc-pr-a-pod10 device 5 [0x8f] NVIDIA A100-SXM4-40GB

Rank 22 Group 0 Pid 2190466 on hpc-pr-a-pod10 device 6 [0xc6] NVIDIA A100-SXM4-40GB

Rank 23 Group 0 Pid 2190469 on hpc-pr-a-pod10 device 7 [0xca] NVIDIA A100-SXM4-40GB

[hpc-pr-a-pod01:3552188] 23 more processes have sent help message help-mpi-btl-openib.txt / error in device init
[hpc-pr-a-pod01:3552188] Set MCA parameter “orte_base_help_aggregate” to 0 to see all help / error messages

out-of-place in-place

size count type redop root time algbw busbw #wrong time algbw busbw #wrong

(B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)

       8             2     float     sum      -1    43.85    0.00    0.00      0    43.51    0.00    0.00      0
 1048584        262146     float     sum      -1    125.0    8.39   16.08      0    125.6    8.35   16.00      0
 2097160        524290     float     sum      -1    176.4   11.89   22.79      0    166.4   12.60   24.15      0
 3145736        786434     float     sum      -1    190.3   16.53   31.69      0    190.0   16.56   31.74      0
 4194312       1048578     float     sum      -1    220.5   19.02   36.46      0    213.1   19.68   37.72      0
 5242888       1310722     float     sum      -1    229.0   22.89   43.88      0    212.2   24.71   47.36      0
 6291464       1572866     float     sum      -1    232.2   27.09   51.93      0    231.7   27.16   52.05      0
 7340040       1835010     float     sum      -1    258.6   28.38   54.40      0    259.6   28.28   54.19      0
 8388616       2097154     float     sum      -1    277.6   30.21   57.91      0    276.2   30.38   58.22      0
 9437192       2359298     float     sum      -1    304.8   30.96   59.35      0    302.6   31.19   59.78      0
10485768       2621442     float     sum      -1    309.1   33.93   65.03      0    304.5   34.44   66.01      0
11534344       2883586     float     sum      -1    314.7   36.65   70.24      0    311.5   37.03   70.97      0
12582920       3145730     float     sum      -1    329.0   38.25   73.31      0    325.9   38.61   73.99      0
13631496       3407874     float     sum      -1    350.8   38.86   74.48      0    346.6   39.33   75.38      0
14680072       3670018     float     sum      -1    368.5   39.84   76.36      0    361.9   40.57   77.75      0
15728648       3932162     float     sum      -1    381.1   41.27   79.11      0    380.4   41.35   79.25      0
16777224       4194306     float     sum      -1    397.7   42.18   80.85      0    392.5   42.75   81.94      0
17825800       4456450     float     sum      -1    422.0   42.24   80.96      0    416.7   42.78   82.00      0
18874376       4718594     float     sum      -1    460.5   40.99   78.56      0    470.5   40.12   76.89      0
19922952       4980738     float     sum      -1    489.5   40.70   78.01      0    488.8   40.76   78.12      0
20971528       5242882     float     sum      -1    500.2   41.93   80.36      0    492.3   42.60   81.65      0
22020104       5505026     float     sum      -1    497.1   44.30   84.91      0    501.2   43.94   84.21      0
23068680       5767170     float     sum      -1    521.4   44.25   84.81      0    510.6   45.18   86.59      0
24117256       6029314     float     sum      -1    539.3   44.72   85.71      0    534.9   45.09   86.42      0
25165832       6291458     float     sum      -1    549.7   45.78   87.75      0    555.0   45.34   86.90      0
26214408       6553602     float     sum      -1    582.5   45.01   86.26      0    575.2   45.57   87.35      0
27262984       6815746     float     sum      -1    601.7   45.31   86.84      0    602.3   45.27   86.76      0
28311560       7077890     float     sum      -1    620.7   45.61   87.42      0    659.8   42.91   82.24      0
29360136       7340034     float     sum      -1    657.8   44.63   85.55      0    670.0   43.82   83.99      0
30408712       7602178     float     sum      -1    704.7   43.15   82.71      0    688.2   44.19   84.69      0
31457288       7864322     float     sum      -1    687.3   45.77   87.73      0    693.9   45.34   86.89      0
32505864       8126466     float     sum      -1    713.4   45.57   87.34      0    709.2   45.83   87.85      0
33554440       8388610     float     sum      -1    728.3   46.07   88.30      0    727.5   46.12   88.40      0
34603016       8650754     float     sum      -1    763.3   45.33   86.89      0    764.3   45.27   86.78      0
35651592       8912898     float     sum      -1    788.1   45.24   86.71      0    787.2   45.29   86.80      0
36700168       9175042     float     sum      -1    799.9   45.88   87.94      0    800.8   45.83   87.84      0
37748744       9437186     float     sum      -1    817.4   46.18   88.52      0    814.7   46.33   88.81      0
38797320       9699330     float     sum      -1    846.6   45.83   87.83      0    879.0   44.14   84.60      0
39845896       9961474     float     sum      -1    913.2   43.63   83.63      0    873.8   45.60   87.40      0
40894472      10223618     float     sum      -1    897.7   45.56   87.31      0    906.5   45.11   86.47      0
41943048      10485762     float     sum      -1    907.3   46.23   88.60      0    925.7   45.31   86.84      0
42991624      10747906     float     sum      -1    925.9   46.43   89.00      0    939.8   45.74   87.68      0
44040200      11010050     float     sum      -1    958.5   45.95   88.07      0    961.7   45.79   87.77      0
45088776      11272194     float     sum      -1    996.8   45.24   86.70      0    994.2   45.35   86.92      0
46137352      11534338     float     sum      -1   1029.8   44.80   85.87      0    997.7   46.24   88.63      0
47185928      11796482     float     sum      -1   1067.3   44.21   84.74      0   1029.4   45.84   87.85      0
48234504      12058626     float     sum      -1   1060.9   45.46   87.14      0   1051.4   45.88   87.93      0
49283080      12320770     float     sum      -1   1080.9   45.59   87.39      0   1069.6   46.08   88.31      0
50331656      12582914     float     sum      -1   1103.1   45.63   87.45      0   1100.6   45.73   87.65      0
51380232      12845058     float     sum      -1   1118.7   45.93   88.03      0   1122.4   45.78   87.74      0
52428808      13107202     float     sum      -1   1156.5   45.33   86.89      0   1148.2   45.66   87.52      0
53477384      13369346     float     sum      -1   1180.9   45.29   86.80      0   1201.1   44.52   85.34      0
54525960      13631490     float     sum      -1   1208.1   45.13   86.51      0   1206.2   45.20   86.64      0
55574536      13893634     float     sum      -1   1248.7   44.51   85.30      0   1223.2   45.43   87.08      0
56623112      14155778     float     sum      -1   1248.0   45.37   86.96      0   1251.0   45.26   86.75      0
57671688      14417922     float     sum      -1   1284.4   44.90   86.06      0   1288.7   44.75   85.77      0
58720264      14680066     float     sum      -1   1313.0   44.72   85.71      0   1309.1   44.86   85.97      0
59768840      14942210     float     sum      -1   1317.9   45.35   86.92      0   1351.3   44.23   84.77      0
60817416      15204354     float     sum      -1   1357.9   44.79   85.84      0   1347.0   45.15   86.54      0
61865992      15466498     float     sum      -1   1387.9   44.57   85.43      0   1371.9   45.10   86.43      0
62914568      15728642     float     sum      -1   1385.8   45.40   87.02      0   1378.6   45.64   87.47      0
63963144      15990786     float     sum      -1   1415.4   45.19   86.62      0   1405.0   45.52   87.25      0
65011720      16252930     float     sum      -1   1435.9   45.28   86.78      0   1439.3   45.17   86.57      0
66060296      16515074     float     sum      -1   1462.6   45.17   86.57      0   1515.2   43.60   83.56      0
67108872      16777218     float     sum      -1   1481.6   45.29   86.81      0   1481.7   45.29   86.81      0
68157448      17039362     float     sum      -1   1511.6   45.09   86.42      0   1509.4   45.16   86.55      0
69206024      17301506     float     sum      -1   1540.6   44.92   86.10      0   1546.5   44.75   85.77      0
70254600      17563650     float     sum      -1   1547.4   45.40   87.02      0   1540.5   45.60   87.41      0
71303176      17825794     float     sum      -1   1592.2   44.78   85.84      0   1584.5   45.00   86.25      0
72351752      18087938     float     sum      -1   1610.3   44.93   86.12      0   1619.3   44.68   85.64      0
73400328      18350082     float     sum      -1   1827.6   40.16   76.98      0   1628.0   45.09   86.42      0
74448904      18612226     float     sum      -1   1640.8   45.37   86.97      0   1639.2   45.42   87.05      0
75497480      18874370     float     sum      -1   1687.6   44.74   85.74      0   1680.5   44.93   86.11      0
76546056      19136514     float     sum      -1   1693.6   45.20   86.63      0   1717.3   44.57   85.43      0
77594632      19398658     float     sum      -1   1714.2   45.26   86.76      0   1750.7   44.32   84.95      0
78643208      19660802     float     sum      -1   1742.3   45.14   86.51      0   1735.7   45.31   86.84      0
79691784      19922946     float     sum      -1   1791.0   44.50   85.28      0   1780.4   44.76   85.79      0
80740360      20185090     float     sum      -1   1787.3   45.17   86.58      0   1794.2   45.00   86.25      0
81788936      20447234     float     sum      -1   1810.5   45.18   86.59      0   1835.4   44.56   85.41      0
82837512      20709378     float     sum      -1   1843.1   44.95   86.15      0   1843.8   44.93   86.11      0
83886088      20971522     float     sum      -1   1891.5   44.35   85.00      0   1871.3   44.83   85.92      0
84934664      21233666     float     sum      -1   1877.1   45.25   86.72      0   1889.5   44.95   86.15      0
85983240      21495810     float     sum      -1   1765.2   48.71   93.36      0   1767.5   48.65   93.24      0
87031816      21757954     float     sum      -1   1785.4   48.75   93.43      0   1820.7   47.80   91.62      0
88080392      22020098     float     sum      -1   1799.6   48.94   93.81      0   1807.3   48.74   93.41      0
89128968      22282242     float     sum      -1   1805.6   49.36   94.61      0   1803.3   49.42   94.73      0
90177544      22544386     float     sum      -1   1833.9   49.17   94.25      0   1831.3   49.24   94.38      0
91226120      22806530     float     sum      -1   1852.3   49.25   94.39      0   1853.5   49.22   94.33      0
92274696      23068674     float     sum      -1   1873.2   49.26   94.41      0   1893.0   48.75   93.43      0
93323272      23330818     float     sum      -1   1880.2   49.63   95.13      0   1874.9   49.78   95.40      0
94371848      23592962     float     sum      -1   1936.1   48.74   93.43      0   1935.4   48.76   93.46      0
95420424      23855106     float     sum      -1   1953.5   48.85   93.62      0   1949.7   48.94   93.80      0
96469000      24117250     float     sum      -1   1972.7   48.90   93.73      0   1962.7   49.15   94.21      0
97517576      24379394     float     sum      -1   2015.3   48.39   92.75      0   2033.7   47.95   91.90      0
98566152      24641538     float     sum      -1   1995.5   49.39   94.67      0   1996.2   49.38   94.64      0
99614728      24903682     float     sum      -1   2003.7   49.72   95.29      0   1998.5   49.84   95.53      0

100663304 25165826 float sum -1 2015.1 49.95 95.74 0 2021.3 49.80 95.45 0
101711880 25427970 float sum -1 2040.4 49.85 95.54 0 2035.4 49.97 95.78 0
102760456 25690114 float sum -1 2052.9 50.06 95.94 0 2061.6 49.84 95.54 0
103809032 25952258 float sum -1 2065.7 50.25 96.32 0 2068.4 50.19 96.20 0
104857608 26214402 float sum -1 2096.4 50.02 95.87 0 2095.3 50.04 95.92 0
105906184 26476546 float sum -1 2158.9 49.06 94.02 0 2173.3 48.73 93.40 0
106954760 26738690 float sum -1 2219.3 48.19 92.37 0 2203.2 48.54 93.04 0
108003336 27000834 float sum -1 2178.4 49.58 95.03 0 2179.6 49.55 94.97 0
109051912 27262978 float sum -1 2201.1 49.54 94.96 0 2201.3 49.54 94.95 0
110100488 27525122 float sum -1 2197.1 50.11 96.05 0 2201.7 50.01 95.85 0
111149064 27787266 float sum -1 2261.7 49.14 94.19 0 2221.7 50.03 95.89 0
112197640 28049410 float sum -1 2275.0 49.32 94.52 0 2241.4 50.06 95.94 0
113246216 28311554 float sum -1 2256.9 50.18 96.18 0 2262.4 50.06 95.94 0
114294792 28573698 float sum -1 2284.4 50.03 95.90 0 2268.0 50.39 96.59 0
115343368 28835842 float sum -1 2324.8 49.61 95.09 0 2326.5 49.58 95.03 0
116391944 29097986 float sum -1 2364.6 49.22 94.34 0 2338.2 49.78 95.41 0
117440520 29360130 float sum -1 2369.4 49.57 95.00 0 2371.5 49.52 94.92 0
118489096 29622274 float sum -1 2393.2 49.51 94.89 0 2384.9 49.68 95.22 0
119537672 29884418 float sum -1 2430.8 49.18 94.25 0 2393.5 49.94 95.73 0
120586248 30146562 float sum -1 2409.0 50.06 95.94 0 2416.1 49.91 95.66 0
121634824 30408706 float sum -1 2440.4 49.84 95.53 0 2432.5 50.00 95.84 0
122683400 30670850 float sum -1 2455.2 49.97 95.77 0 2462.6 49.82 95.49 0
123731976 30932994 float sum -1 2490.2 49.69 95.24 0 2476.9 49.95 95.75 0
124780552 31195138 float sum -1 2484.5 50.22 96.26 0 2486.7 50.18 96.17 0
125829128 31457282 float sum -1 2540.9 49.52 94.92 0 2534.8 49.64 95.15 0
126877704 31719426 float sum -1 2561.2 49.54 94.95 0 2582.9 49.12 94.15 0
127926280 31981570 float sum -1 2564.7 49.88 95.60 0 2567.9 49.82 95.48 0
128974856 32243714 float sum -1 2592.3 49.75 95.36 0 2582.7 49.94 95.72 0
130023432 32505858 float sum -1 2599.3 50.02 95.88 0 2597.6 50.06 95.94 0
131072008 32768002 float sum -1 2636.0 49.72 95.30 0 2608.2 50.25 96.32 0
132120584 33030146 float sum -1 2633.9 50.16 96.14 0 2634.8 50.14 96.11 0
133169160 33292290 float sum -1 2669.7 49.88 95.61 0 2662.7 50.01 95.86 0
134217736 33554434 float sum -1 2680.0 50.08 95.99 0 2687.1 49.95 95.73 0
135266312 33816578 float sum -1 2695.7 50.18 96.18 0 2730.6 49.54 94.95 0
136314888 34078722 float sum -1 2722.0 50.08 95.98 0 2711.8 50.27 96.35 0
137363464 34340866 float sum -1 2759.6 49.78 95.41 0 2767.1 49.64 95.15 0
138412040 34603010 float sum -1 2817.6 49.12 94.16 0 2773.9 49.90 95.64 0
139460616 34865154 float sum -1 2772.4 50.30 96.41 0 2788.8 50.01 95.85 0
140509192 35127298 float sum -1 2808.0 50.04 95.91 0 2800.6 50.17 96.16 0
141557768 35389442 float sum -1 2819.1 50.21 96.24 0 2818.3 50.23 96.27 0
142606344 35651586 float sum -1 2836.6 50.27 96.36 0 2829.1 50.41 96.61 0
143654920 35913730 float sum -1 2852.5 50.36 96.53 0 2843.5 50.52 96.83 0
144703496 36175874 float sum -1 2867.6 50.46 96.72 0 2868.6 50.44 96.69 0
145752072 36438018 float sum -1 2917.3 49.96 95.76 0 2895.7 50.33 96.47 0
146800648 36700162 float sum -1 2931.1 50.08 95.99 0 2925.0 50.19 96.20 0
147849224 36962306 float sum -1 2948.4 50.15 96.11 0 2951.9 50.09 96.00 0
148897800 37224450 float sum -1 2987.9 49.83 95.52 0 3014.9 49.39 94.66 0
149946376 37486594 float sum -1 3011.3 49.79 95.44 0 3010.3 49.81 95.47 0
150994952 37748738 float sum -1 3014.7 50.09 96.00 0 3009.7 50.17 96.16 0
152043528 38010882 float sum -1 3030.0 50.18 96.18 0 3028.6 50.20 96.22 0
153092104 38273026 float sum -1 3062.3 49.99 95.82 0 3058.0 50.06 95.95 0
154140680 38535170 float sum -1 3088.0 49.92 95.67 0 3076.1 50.11 96.04 0
155189256 38797314 float sum -1 3094.4 50.15 96.12 0 3110.0 49.90 95.64 0
156237832 39059458 float sum -1 3109.0 50.25 96.32 0 3109.3 50.25 96.31 0
157286408 39321602 float sum -1 3180.2 49.46 94.79 0 3152.1 49.90 95.64 0
158334984 39583746 float sum -1 3179.1 49.80 95.46 0 3190.5 49.63 95.12 0
159383560 39845890 float sum -1 3206.7 49.70 95.27 0 3196.6 49.86 95.57 0
160432136 40108034 float sum -1 3208.7 50.00 95.83 0 3210.2 49.98 95.79 0
161480712 40370178 float sum -1 3233.9 49.93 95.71 0 3242.9 49.79 95.44 0
162529288 40632322 float sum -1 3285.2 49.47 94.82 0 3238.5 50.19 96.19 0
163577864 40894466 float sum -1 3260.6 50.17 96.15 0 3258.5 50.20 96.22 0
164626440 41156610 float sum -1 3306.8 49.78 95.42 0 3309.1 49.75 95.35 0
165675016 41418754 float sum -1 3366.0 49.22 94.34 0 3349.0 49.47 94.82 0
166723592 41680898 float sum -1 3337.3 49.96 95.75 0 3340.2 49.91 95.67 0
167772168 41943042 float sum -1 3368.0 49.81 95.48 0 3354.2 50.02 95.87 0
168820744 42205186 float sum -1 3396.1 49.71 95.28 0 3406.6 49.56 94.98 0
169869320 42467330 float sum -1 3414.8 49.74 95.34 0 3416.2 49.72 95.30 0
170917896 42729474 float sum -1 3420.4 49.97 95.78 0 3420.5 49.97 95.77 0
171966472 42991618 float sum -1 3478.4 49.44 94.76 0 3446.9 49.89 95.62 0
173015048 43253762 float sum -1 3472.4 49.83 95.50 0 3477.7 49.75 95.35 0
174063624 43515906 float sum -1 3494.4 49.81 95.47 0 3486.8 49.92 95.68 0
175112200 43778050 float sum -1 3508.1 49.92 95.67 0 3513.7 49.84 95.52 0
176160776 44040194 float sum -1 3525.2 49.97 95.78 0 3522.3 50.01 95.86 0
177209352 44302338 float sum -1 3549.4 49.93 95.69 0 3573.3 49.59 95.05 0
178257928 44564482 float sum -1 3573.6 49.88 95.61 0 3566.8 49.98 95.79 0
179306504 44826626 float sum -1 3598.6 49.83 95.50 0 3599.6 49.81 95.47 0
180355080 45088770 float sum -1 3652.9 49.37 94.63 0 3656.6 49.32 94.54 0
181403656 45350914 float sum -1 3680.6 49.29 94.46 0 3675.8 49.35 94.59 0
182452232 45613058 float sum -1 3671.7 49.69 95.24 0 3665.2 49.78 95.41 0
183500808 45875202 float sum -1 3683.8 49.81 95.47 0 3676.2 49.92 95.67 0
184549384 46137346 float sum -1 3709.8 49.75 95.35 0 3712.9 49.70 95.27 0
185597960 46399490 float sum -1 3730.0 49.76 95.37 0 3744.4 49.57 95.00 0
186646536 46661634 float sum -1 3741.9 49.88 95.60 0 3740.5 49.90 95.64 0
187695112 46923778 float sum -1 3772.5 49.75 95.36 0 3762.2 49.89 95.62 0
188743688 47185922 float sum -1 3806.4 49.59 95.04 0 3803.6 49.62 95.11 0
189792264 47448066 float sum -1 3828.3 49.58 95.02 0 3824.4 49.63 95.12 0
190840840 47710210 float sum -1 3854.0 49.52 94.91 0 3847.9 49.60 95.06 0
191889416 47972354 float sum -1 3877.7 49.49 94.85 0 3853.6 49.79 95.44 0
192937992 48234498 float sum -1 3895.5 49.53 94.93 0 3895.7 49.53 94.93 0
193986568 48496642 float sum -1 3906.9 49.65 95.17 0 3920.1 49.48 94.85 0
195035144 48758786 float sum -1 3907.5 49.91 95.67 0 3918.2 49.78 95.41 0
196083720 49020930 float sum -1 3972.2 49.36 94.62 0 3965.6 49.45 94.77 0
197132296 49283074 float sum -1 4007.2 49.19 94.29 0 3996.9 49.32 94.53 0
198180872 49545218 float sum -1 4010.2 49.42 94.72 0 3999.5 49.55 94.97 0
199229448 49807362 float sum -1 4019.1 49.57 95.01 0 4074.7 48.89 93.71 0
200278024 50069506 float sum -1 4039.3 49.58 95.03 0 4048.7 49.47 94.81 0
201326600 50331650 float sum -1 4073.2 49.43 94.73 0 4070.5 49.46 94.80 0
202375176 50593794 float sum -1 4075.7 49.65 95.17 0 4064.0 49.80 95.44 0
203423752 50855938 float sum -1 4104.5 49.56 94.99 0 4101.9 49.59 95.05 0
204472328 51118082 float sum -1 4126.1 49.56 94.98 0 4143.8 49.34 94.58 0
205520904 51380226 float sum -1 4141.6 49.62 95.11 0 4148.5 49.54 94.95 0
206569480 51642370 float sum -1 4167.0 49.57 95.01 0 4162.8 49.62 95.11 0
207618056 51904514 float sum -1 4189.3 49.56 94.99 0 4174.4 49.74 95.33 0
208666632 52166658 float sum -1 4203.9 49.64 95.14 0 4205.2 49.62 95.11 0
209715208 52428802 float sum -1 4219.0 49.71 95.27 0 4224.9 49.64 95.14 0
210763784 52690946 float sum -1 4258.2 49.50 94.87 0 4251.6 49.57 95.01 0
211812360 52953090 float sum -1 4292.2 49.35 94.58 0 4297.7 49.29 94.46 0
212860936 53215234 float sum -1 4374.6 48.66 93.26 0 4332.0 49.14 94.18 0
213909512 53477378 float sum -1 4321.2 49.50 94.88 0 4320.6 49.51 94.89 0
214958088 53739522 float sum -1 4354.4 49.37 94.62 0 4365.6 49.24 94.37 0
216006664 54001666 float sum -1 4366.4 49.47 94.82 0 4369.7 49.43 94.75 0
217055240 54263810 float sum -1 4388.9 49.46 94.79 0 4409.4 49.23 94.35 0
218103816 54525954 float sum -1 4400.3 49.57 95.00 0 4395.2 49.62 95.11 0
219152392 54788098 float sum -1 4437.4 49.39 94.66 0 4421.3 49.57 95.00 0
220200968 55050242 float sum -1 4489.0 49.05 94.02 0 4450.2 49.48 94.84 0
221249544 55312386 float sum -1 4485.7 49.32 94.54 0 4485.2 49.33 94.55 0
222298120 55574530 float sum -1 4512.6 49.26 94.42 0 4538.9 48.98 93.87 0
223346696 55836674 float sum -1 4529.7 49.31 94.51 0 4509.9 49.52 94.92 0
224395272 56098818 float sum -1 4530.3 49.53 94.94 0 4542.0 49.40 94.69 0
225443848 56360962 float sum -1 4557.5 49.47 94.81 0 4549.5 49.55 94.98 0
226492424 56623106 float sum -1 4586.0 49.39 94.66 0 4596.6 49.27 94.44 0
227541000 56885250 float sum -1 4657.5 48.85 93.64 0 4628.7 49.16 94.22 0
228589576 57147394 float sum -1 4670.5 48.94 93.81 0 4661.1 49.04 94.00 0
229638152 57409538 float sum -1 4664.5 49.23 94.36 0 4661.9 49.26 94.41 0
230686728 57671682 float sum -1 4676.2 49.33 94.55 0 4683.3 49.26 94.41 0
231735304 57933826 float sum -1 4710.1 49.20 94.30 0 4705.1 49.25 94.40 0
232783880 58195970 float sum -1 4737.8 49.13 94.17 0 4735.9 49.15 94.21 0
233832456 58458114 float sum -1 4738.9 49.34 94.57 0 4775.5 48.96 93.85 0
234881032 58720258 float sum -1 4754.9 49.40 94.68 0 4750.7 49.44 94.76 0
235929608 58982402 float sum -1 4779.2 49.37 94.62 0 4785.4 49.30 94.50 0
236978184 59244546 float sum -1 4804.1 49.33 94.55 0 4808.4 49.28 94.46 0
238026760 59506690 float sum -1 4826.3 49.32 94.53 0 4825.1 49.33 94.55 0
239075336 59768834 float sum -1 4869.2 49.10 94.11 0 4836.7 49.43 94.74 0
240123912 60030978 float sum -1 4863.9 49.37 94.62 0 4858.4 49.42 94.73 0
241172488 60293122 float sum -1 4902.2 49.20 94.29 0 4887.1 49.35 94.58 0
242221064 60555266 float sum -1 4912.7 49.31 94.50 0 4907.4 49.36 94.60 0
243269640 60817410 float sum -1 4983.1 48.82 93.57 0 4964.0 49.01 93.93 0
244318216 61079554 float sum -1 5004.6 48.82 93.57 0 4998.6 48.88 93.68 0
245366792 61341698 float sum -1 4998.9 49.08 94.08 0 5021.1 48.87 93.66 0
246415368 61603842 float sum -1 5011.3 49.17 94.25 0 5013.5 49.15 94.21 0
247463944 61865986 float sum -1 5048.8 49.01 93.94 0 5047.8 49.02 93.96 0
248512520 62128130 float sum -1 5072.4 48.99 93.90 0 5065.3 49.06 94.04 0
249561096 62390274 float sum -1 5065.3 49.27 94.43 0 5085.6 49.07 94.06 0
250609672 62652418 float sum -1 5084.5 49.29 94.47 0 5084.8 49.29 94.46 0
251658248 62914562 float sum -1 5116.5 49.19 94.27 0 5120.2 49.15 94.20 0
252706824 63176706 float sum -1 5144.3 49.12 94.15 0 5168.5 48.89 93.71 0
253755400 63438850 float sum -1 5166.5 49.12 94.14 0 5193.0 48.86 93.66 0
254803976 63700994 float sum -1 5178.1 49.21 94.31 0 5178.4 49.21 94.31 0
255852552 63963138 float sum -1 5203.3 49.17 94.25 0 5223.6 48.98 93.88 0
256901128 64225282 float sum -1 5229.6 49.12 94.16 0 5223.9 49.18 94.26 0
257949704 64487426 float sum -1 5236.0 49.26 94.42 0 5244.7 49.18 94.27 0
258998280 64749570 float sum -1 5301.3 48.86 93.64 0 5307.5 48.80 93.53 0
260046856 65011714 float sum -1 5340.9 48.69 93.32 0 5366.2 48.46 92.88 0
261095432 65273858 float sum -1 5338.9 48.90 93.73 0 5332.8 48.96 93.84 0
262144008 65536002 float sum -1 5355.2 48.95 93.82 0 5349.1 49.01 93.93 0
263192584 65798146 float sum -1 5385.2 48.87 93.67 0 5397.8 48.76 93.46 0
264241160 66060290 float sum -1 5401.5 48.92 93.76 0 5430.2 48.66 93.27 0
265289736 66322434 float sum -1 5405.0 49.08 94.07 0 5398.6 49.14 94.19 0
266338312 66584578 float sum -1 5418.2 49.16 94.22 0 5463.3 48.75 93.44 0
267386888 66846722 float sum -1 5462.5 48.95 93.82 0 5445.7 49.10 94.11 0

Out of bounds values : 0 OK

Avg bus bandwidth : 89.8902

Is this result normal? or the performance is low?