I have used 3 node of HGX(Insuper SMX4 A100 40G * 8ea) with four HDR Infiniband card(Mellanox) for nccl test.
I got a result like as bellow.
ubuntu@hpc-pr-a-pod01:~/nccl_test$ mpirun -mca pml ucx -np 24 --hostfile /home/ubuntu/nccl_test/hostfile /home/ubuntu/nccl_test/nccl-tests/build/all_reduce_perf -b 8 -e 256M -g 1
WARNING: There was an error initializing an OpenFabrics device.
Local host: hpc-pr-a-pod10
Local device: mlx5_0
nThread 1 nGpus 1 minBytes 8 maxBytes 268435456 step: 1048576(bytes) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
Using devices
Rank 0 Group 0 Pid 3552208 on hpc-pr-a-pod01 device 0 [0x10] NVIDIA A100-SXM4-40GB
Rank 1 Group 0 Pid 3552209 on hpc-pr-a-pod01 device 1 [0x16] NVIDIA A100-SXM4-40GB
Rank 2 Group 0 Pid 3552210 on hpc-pr-a-pod01 device 2 [0x49] NVIDIA A100-SXM4-40GB
Rank 3 Group 0 Pid 3552211 on hpc-pr-a-pod01 device 3 [0x4d] NVIDIA A100-SXM4-40GB
Rank 4 Group 0 Pid 3552212 on hpc-pr-a-pod01 device 4 [0x8a] NVIDIA A100-SXM4-40GB
Rank 5 Group 0 Pid 3552213 on hpc-pr-a-pod01 device 5 [0x8f] NVIDIA A100-SXM4-40GB
Rank 6 Group 0 Pid 3552214 on hpc-pr-a-pod01 device 6 [0xc6] NVIDIA A100-SXM4-40GB
Rank 7 Group 0 Pid 3552218 on hpc-pr-a-pod01 device 7 [0xca] NVIDIA A100-SXM4-40GB
Rank 8 Group 0 Pid 1736754 on hpc-pr-a-pod03 device 0 [0x10] NVIDIA A100-SXM4-40GB
Rank 9 Group 0 Pid 1736755 on hpc-pr-a-pod03 device 1 [0x16] NVIDIA A100-SXM4-40GB
Rank 10 Group 0 Pid 1736756 on hpc-pr-a-pod03 device 2 [0x49] NVIDIA A100-SXM4-40GB
Rank 11 Group 0 Pid 1736757 on hpc-pr-a-pod03 device 3 [0x4d] NVIDIA A100-SXM4-40GB
Rank 12 Group 0 Pid 1736758 on hpc-pr-a-pod03 device 4 [0x8a] NVIDIA A100-SXM4-40GB
Rank 13 Group 0 Pid 1736759 on hpc-pr-a-pod03 device 5 [0x8f] NVIDIA A100-SXM4-40GB
Rank 14 Group 0 Pid 1736761 on hpc-pr-a-pod03 device 6 [0xc6] NVIDIA A100-SXM4-40GB
Rank 15 Group 0 Pid 1736765 on hpc-pr-a-pod03 device 7 [0xca] NVIDIA A100-SXM4-40GB
Rank 16 Group 0 Pid 2190460 on hpc-pr-a-pod10 device 0 [0x10] NVIDIA A100-SXM4-40GB
Rank 17 Group 0 Pid 2190461 on hpc-pr-a-pod10 device 1 [0x16] NVIDIA A100-SXM4-40GB
Rank 18 Group 0 Pid 2190462 on hpc-pr-a-pod10 device 2 [0x49] NVIDIA A100-SXM4-40GB
Rank 19 Group 0 Pid 2190463 on hpc-pr-a-pod10 device 3 [0x4d] NVIDIA A100-SXM4-40GB
Rank 20 Group 0 Pid 2190464 on hpc-pr-a-pod10 device 4 [0x8a] NVIDIA A100-SXM4-40GB
Rank 21 Group 0 Pid 2190465 on hpc-pr-a-pod10 device 5 [0x8f] NVIDIA A100-SXM4-40GB
Rank 22 Group 0 Pid 2190466 on hpc-pr-a-pod10 device 6 [0xc6] NVIDIA A100-SXM4-40GB
Rank 23 Group 0 Pid 2190469 on hpc-pr-a-pod10 device 7 [0xca] NVIDIA A100-SXM4-40GB
[hpc-pr-a-pod01:3552188] 23 more processes have sent help message help-mpi-btl-openib.txt / error in device init
[hpc-pr-a-pod01:3552188] Set MCA parameter “orte_base_help_aggregate” to 0 to see all help / error messages
out-of-place in-place
size count type redop root time algbw busbw #wrong time algbw busbw #wrong
(B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
8 2 float sum -1 43.85 0.00 0.00 0 43.51 0.00 0.00 0
1048584 262146 float sum -1 125.0 8.39 16.08 0 125.6 8.35 16.00 0
2097160 524290 float sum -1 176.4 11.89 22.79 0 166.4 12.60 24.15 0
3145736 786434 float sum -1 190.3 16.53 31.69 0 190.0 16.56 31.74 0
4194312 1048578 float sum -1 220.5 19.02 36.46 0 213.1 19.68 37.72 0
5242888 1310722 float sum -1 229.0 22.89 43.88 0 212.2 24.71 47.36 0
6291464 1572866 float sum -1 232.2 27.09 51.93 0 231.7 27.16 52.05 0
7340040 1835010 float sum -1 258.6 28.38 54.40 0 259.6 28.28 54.19 0
8388616 2097154 float sum -1 277.6 30.21 57.91 0 276.2 30.38 58.22 0
9437192 2359298 float sum -1 304.8 30.96 59.35 0 302.6 31.19 59.78 0
10485768 2621442 float sum -1 309.1 33.93 65.03 0 304.5 34.44 66.01 0
11534344 2883586 float sum -1 314.7 36.65 70.24 0 311.5 37.03 70.97 0
12582920 3145730 float sum -1 329.0 38.25 73.31 0 325.9 38.61 73.99 0
13631496 3407874 float sum -1 350.8 38.86 74.48 0 346.6 39.33 75.38 0
14680072 3670018 float sum -1 368.5 39.84 76.36 0 361.9 40.57 77.75 0
15728648 3932162 float sum -1 381.1 41.27 79.11 0 380.4 41.35 79.25 0
16777224 4194306 float sum -1 397.7 42.18 80.85 0 392.5 42.75 81.94 0
17825800 4456450 float sum -1 422.0 42.24 80.96 0 416.7 42.78 82.00 0
18874376 4718594 float sum -1 460.5 40.99 78.56 0 470.5 40.12 76.89 0
19922952 4980738 float sum -1 489.5 40.70 78.01 0 488.8 40.76 78.12 0
20971528 5242882 float sum -1 500.2 41.93 80.36 0 492.3 42.60 81.65 0
22020104 5505026 float sum -1 497.1 44.30 84.91 0 501.2 43.94 84.21 0
23068680 5767170 float sum -1 521.4 44.25 84.81 0 510.6 45.18 86.59 0
24117256 6029314 float sum -1 539.3 44.72 85.71 0 534.9 45.09 86.42 0
25165832 6291458 float sum -1 549.7 45.78 87.75 0 555.0 45.34 86.90 0
26214408 6553602 float sum -1 582.5 45.01 86.26 0 575.2 45.57 87.35 0
27262984 6815746 float sum -1 601.7 45.31 86.84 0 602.3 45.27 86.76 0
28311560 7077890 float sum -1 620.7 45.61 87.42 0 659.8 42.91 82.24 0
29360136 7340034 float sum -1 657.8 44.63 85.55 0 670.0 43.82 83.99 0
30408712 7602178 float sum -1 704.7 43.15 82.71 0 688.2 44.19 84.69 0
31457288 7864322 float sum -1 687.3 45.77 87.73 0 693.9 45.34 86.89 0
32505864 8126466 float sum -1 713.4 45.57 87.34 0 709.2 45.83 87.85 0
33554440 8388610 float sum -1 728.3 46.07 88.30 0 727.5 46.12 88.40 0
34603016 8650754 float sum -1 763.3 45.33 86.89 0 764.3 45.27 86.78 0
35651592 8912898 float sum -1 788.1 45.24 86.71 0 787.2 45.29 86.80 0
36700168 9175042 float sum -1 799.9 45.88 87.94 0 800.8 45.83 87.84 0
37748744 9437186 float sum -1 817.4 46.18 88.52 0 814.7 46.33 88.81 0
38797320 9699330 float sum -1 846.6 45.83 87.83 0 879.0 44.14 84.60 0
39845896 9961474 float sum -1 913.2 43.63 83.63 0 873.8 45.60 87.40 0
40894472 10223618 float sum -1 897.7 45.56 87.31 0 906.5 45.11 86.47 0
41943048 10485762 float sum -1 907.3 46.23 88.60 0 925.7 45.31 86.84 0
42991624 10747906 float sum -1 925.9 46.43 89.00 0 939.8 45.74 87.68 0
44040200 11010050 float sum -1 958.5 45.95 88.07 0 961.7 45.79 87.77 0
45088776 11272194 float sum -1 996.8 45.24 86.70 0 994.2 45.35 86.92 0
46137352 11534338 float sum -1 1029.8 44.80 85.87 0 997.7 46.24 88.63 0
47185928 11796482 float sum -1 1067.3 44.21 84.74 0 1029.4 45.84 87.85 0
48234504 12058626 float sum -1 1060.9 45.46 87.14 0 1051.4 45.88 87.93 0
49283080 12320770 float sum -1 1080.9 45.59 87.39 0 1069.6 46.08 88.31 0
50331656 12582914 float sum -1 1103.1 45.63 87.45 0 1100.6 45.73 87.65 0
51380232 12845058 float sum -1 1118.7 45.93 88.03 0 1122.4 45.78 87.74 0
52428808 13107202 float sum -1 1156.5 45.33 86.89 0 1148.2 45.66 87.52 0
53477384 13369346 float sum -1 1180.9 45.29 86.80 0 1201.1 44.52 85.34 0
54525960 13631490 float sum -1 1208.1 45.13 86.51 0 1206.2 45.20 86.64 0
55574536 13893634 float sum -1 1248.7 44.51 85.30 0 1223.2 45.43 87.08 0
56623112 14155778 float sum -1 1248.0 45.37 86.96 0 1251.0 45.26 86.75 0
57671688 14417922 float sum -1 1284.4 44.90 86.06 0 1288.7 44.75 85.77 0
58720264 14680066 float sum -1 1313.0 44.72 85.71 0 1309.1 44.86 85.97 0
59768840 14942210 float sum -1 1317.9 45.35 86.92 0 1351.3 44.23 84.77 0
60817416 15204354 float sum -1 1357.9 44.79 85.84 0 1347.0 45.15 86.54 0
61865992 15466498 float sum -1 1387.9 44.57 85.43 0 1371.9 45.10 86.43 0
62914568 15728642 float sum -1 1385.8 45.40 87.02 0 1378.6 45.64 87.47 0
63963144 15990786 float sum -1 1415.4 45.19 86.62 0 1405.0 45.52 87.25 0
65011720 16252930 float sum -1 1435.9 45.28 86.78 0 1439.3 45.17 86.57 0
66060296 16515074 float sum -1 1462.6 45.17 86.57 0 1515.2 43.60 83.56 0
67108872 16777218 float sum -1 1481.6 45.29 86.81 0 1481.7 45.29 86.81 0
68157448 17039362 float sum -1 1511.6 45.09 86.42 0 1509.4 45.16 86.55 0
69206024 17301506 float sum -1 1540.6 44.92 86.10 0 1546.5 44.75 85.77 0
70254600 17563650 float sum -1 1547.4 45.40 87.02 0 1540.5 45.60 87.41 0
71303176 17825794 float sum -1 1592.2 44.78 85.84 0 1584.5 45.00 86.25 0
72351752 18087938 float sum -1 1610.3 44.93 86.12 0 1619.3 44.68 85.64 0
73400328 18350082 float sum -1 1827.6 40.16 76.98 0 1628.0 45.09 86.42 0
74448904 18612226 float sum -1 1640.8 45.37 86.97 0 1639.2 45.42 87.05 0
75497480 18874370 float sum -1 1687.6 44.74 85.74 0 1680.5 44.93 86.11 0
76546056 19136514 float sum -1 1693.6 45.20 86.63 0 1717.3 44.57 85.43 0
77594632 19398658 float sum -1 1714.2 45.26 86.76 0 1750.7 44.32 84.95 0
78643208 19660802 float sum -1 1742.3 45.14 86.51 0 1735.7 45.31 86.84 0
79691784 19922946 float sum -1 1791.0 44.50 85.28 0 1780.4 44.76 85.79 0
80740360 20185090 float sum -1 1787.3 45.17 86.58 0 1794.2 45.00 86.25 0
81788936 20447234 float sum -1 1810.5 45.18 86.59 0 1835.4 44.56 85.41 0
82837512 20709378 float sum -1 1843.1 44.95 86.15 0 1843.8 44.93 86.11 0
83886088 20971522 float sum -1 1891.5 44.35 85.00 0 1871.3 44.83 85.92 0
84934664 21233666 float sum -1 1877.1 45.25 86.72 0 1889.5 44.95 86.15 0
85983240 21495810 float sum -1 1765.2 48.71 93.36 0 1767.5 48.65 93.24 0
87031816 21757954 float sum -1 1785.4 48.75 93.43 0 1820.7 47.80 91.62 0
88080392 22020098 float sum -1 1799.6 48.94 93.81 0 1807.3 48.74 93.41 0
89128968 22282242 float sum -1 1805.6 49.36 94.61 0 1803.3 49.42 94.73 0
90177544 22544386 float sum -1 1833.9 49.17 94.25 0 1831.3 49.24 94.38 0
91226120 22806530 float sum -1 1852.3 49.25 94.39 0 1853.5 49.22 94.33 0
92274696 23068674 float sum -1 1873.2 49.26 94.41 0 1893.0 48.75 93.43 0
93323272 23330818 float sum -1 1880.2 49.63 95.13 0 1874.9 49.78 95.40 0
94371848 23592962 float sum -1 1936.1 48.74 93.43 0 1935.4 48.76 93.46 0
95420424 23855106 float sum -1 1953.5 48.85 93.62 0 1949.7 48.94 93.80 0
96469000 24117250 float sum -1 1972.7 48.90 93.73 0 1962.7 49.15 94.21 0
97517576 24379394 float sum -1 2015.3 48.39 92.75 0 2033.7 47.95 91.90 0
98566152 24641538 float sum -1 1995.5 49.39 94.67 0 1996.2 49.38 94.64 0
99614728 24903682 float sum -1 2003.7 49.72 95.29 0 1998.5 49.84 95.53 0
100663304 25165826 float sum -1 2015.1 49.95 95.74 0 2021.3 49.80 95.45 0
101711880 25427970 float sum -1 2040.4 49.85 95.54 0 2035.4 49.97 95.78 0
102760456 25690114 float sum -1 2052.9 50.06 95.94 0 2061.6 49.84 95.54 0
103809032 25952258 float sum -1 2065.7 50.25 96.32 0 2068.4 50.19 96.20 0
104857608 26214402 float sum -1 2096.4 50.02 95.87 0 2095.3 50.04 95.92 0
105906184 26476546 float sum -1 2158.9 49.06 94.02 0 2173.3 48.73 93.40 0
106954760 26738690 float sum -1 2219.3 48.19 92.37 0 2203.2 48.54 93.04 0
108003336 27000834 float sum -1 2178.4 49.58 95.03 0 2179.6 49.55 94.97 0
109051912 27262978 float sum -1 2201.1 49.54 94.96 0 2201.3 49.54 94.95 0
110100488 27525122 float sum -1 2197.1 50.11 96.05 0 2201.7 50.01 95.85 0
111149064 27787266 float sum -1 2261.7 49.14 94.19 0 2221.7 50.03 95.89 0
112197640 28049410 float sum -1 2275.0 49.32 94.52 0 2241.4 50.06 95.94 0
113246216 28311554 float sum -1 2256.9 50.18 96.18 0 2262.4 50.06 95.94 0
114294792 28573698 float sum -1 2284.4 50.03 95.90 0 2268.0 50.39 96.59 0
115343368 28835842 float sum -1 2324.8 49.61 95.09 0 2326.5 49.58 95.03 0
116391944 29097986 float sum -1 2364.6 49.22 94.34 0 2338.2 49.78 95.41 0
117440520 29360130 float sum -1 2369.4 49.57 95.00 0 2371.5 49.52 94.92 0
118489096 29622274 float sum -1 2393.2 49.51 94.89 0 2384.9 49.68 95.22 0
119537672 29884418 float sum -1 2430.8 49.18 94.25 0 2393.5 49.94 95.73 0
120586248 30146562 float sum -1 2409.0 50.06 95.94 0 2416.1 49.91 95.66 0
121634824 30408706 float sum -1 2440.4 49.84 95.53 0 2432.5 50.00 95.84 0
122683400 30670850 float sum -1 2455.2 49.97 95.77 0 2462.6 49.82 95.49 0
123731976 30932994 float sum -1 2490.2 49.69 95.24 0 2476.9 49.95 95.75 0
124780552 31195138 float sum -1 2484.5 50.22 96.26 0 2486.7 50.18 96.17 0
125829128 31457282 float sum -1 2540.9 49.52 94.92 0 2534.8 49.64 95.15 0
126877704 31719426 float sum -1 2561.2 49.54 94.95 0 2582.9 49.12 94.15 0
127926280 31981570 float sum -1 2564.7 49.88 95.60 0 2567.9 49.82 95.48 0
128974856 32243714 float sum -1 2592.3 49.75 95.36 0 2582.7 49.94 95.72 0
130023432 32505858 float sum -1 2599.3 50.02 95.88 0 2597.6 50.06 95.94 0
131072008 32768002 float sum -1 2636.0 49.72 95.30 0 2608.2 50.25 96.32 0
132120584 33030146 float sum -1 2633.9 50.16 96.14 0 2634.8 50.14 96.11 0
133169160 33292290 float sum -1 2669.7 49.88 95.61 0 2662.7 50.01 95.86 0
134217736 33554434 float sum -1 2680.0 50.08 95.99 0 2687.1 49.95 95.73 0
135266312 33816578 float sum -1 2695.7 50.18 96.18 0 2730.6 49.54 94.95 0
136314888 34078722 float sum -1 2722.0 50.08 95.98 0 2711.8 50.27 96.35 0
137363464 34340866 float sum -1 2759.6 49.78 95.41 0 2767.1 49.64 95.15 0
138412040 34603010 float sum -1 2817.6 49.12 94.16 0 2773.9 49.90 95.64 0
139460616 34865154 float sum -1 2772.4 50.30 96.41 0 2788.8 50.01 95.85 0
140509192 35127298 float sum -1 2808.0 50.04 95.91 0 2800.6 50.17 96.16 0
141557768 35389442 float sum -1 2819.1 50.21 96.24 0 2818.3 50.23 96.27 0
142606344 35651586 float sum -1 2836.6 50.27 96.36 0 2829.1 50.41 96.61 0
143654920 35913730 float sum -1 2852.5 50.36 96.53 0 2843.5 50.52 96.83 0
144703496 36175874 float sum -1 2867.6 50.46 96.72 0 2868.6 50.44 96.69 0
145752072 36438018 float sum -1 2917.3 49.96 95.76 0 2895.7 50.33 96.47 0
146800648 36700162 float sum -1 2931.1 50.08 95.99 0 2925.0 50.19 96.20 0
147849224 36962306 float sum -1 2948.4 50.15 96.11 0 2951.9 50.09 96.00 0
148897800 37224450 float sum -1 2987.9 49.83 95.52 0 3014.9 49.39 94.66 0
149946376 37486594 float sum -1 3011.3 49.79 95.44 0 3010.3 49.81 95.47 0
150994952 37748738 float sum -1 3014.7 50.09 96.00 0 3009.7 50.17 96.16 0
152043528 38010882 float sum -1 3030.0 50.18 96.18 0 3028.6 50.20 96.22 0
153092104 38273026 float sum -1 3062.3 49.99 95.82 0 3058.0 50.06 95.95 0
154140680 38535170 float sum -1 3088.0 49.92 95.67 0 3076.1 50.11 96.04 0
155189256 38797314 float sum -1 3094.4 50.15 96.12 0 3110.0 49.90 95.64 0
156237832 39059458 float sum -1 3109.0 50.25 96.32 0 3109.3 50.25 96.31 0
157286408 39321602 float sum -1 3180.2 49.46 94.79 0 3152.1 49.90 95.64 0
158334984 39583746 float sum -1 3179.1 49.80 95.46 0 3190.5 49.63 95.12 0
159383560 39845890 float sum -1 3206.7 49.70 95.27 0 3196.6 49.86 95.57 0
160432136 40108034 float sum -1 3208.7 50.00 95.83 0 3210.2 49.98 95.79 0
161480712 40370178 float sum -1 3233.9 49.93 95.71 0 3242.9 49.79 95.44 0
162529288 40632322 float sum -1 3285.2 49.47 94.82 0 3238.5 50.19 96.19 0
163577864 40894466 float sum -1 3260.6 50.17 96.15 0 3258.5 50.20 96.22 0
164626440 41156610 float sum -1 3306.8 49.78 95.42 0 3309.1 49.75 95.35 0
165675016 41418754 float sum -1 3366.0 49.22 94.34 0 3349.0 49.47 94.82 0
166723592 41680898 float sum -1 3337.3 49.96 95.75 0 3340.2 49.91 95.67 0
167772168 41943042 float sum -1 3368.0 49.81 95.48 0 3354.2 50.02 95.87 0
168820744 42205186 float sum -1 3396.1 49.71 95.28 0 3406.6 49.56 94.98 0
169869320 42467330 float sum -1 3414.8 49.74 95.34 0 3416.2 49.72 95.30 0
170917896 42729474 float sum -1 3420.4 49.97 95.78 0 3420.5 49.97 95.77 0
171966472 42991618 float sum -1 3478.4 49.44 94.76 0 3446.9 49.89 95.62 0
173015048 43253762 float sum -1 3472.4 49.83 95.50 0 3477.7 49.75 95.35 0
174063624 43515906 float sum -1 3494.4 49.81 95.47 0 3486.8 49.92 95.68 0
175112200 43778050 float sum -1 3508.1 49.92 95.67 0 3513.7 49.84 95.52 0
176160776 44040194 float sum -1 3525.2 49.97 95.78 0 3522.3 50.01 95.86 0
177209352 44302338 float sum -1 3549.4 49.93 95.69 0 3573.3 49.59 95.05 0
178257928 44564482 float sum -1 3573.6 49.88 95.61 0 3566.8 49.98 95.79 0
179306504 44826626 float sum -1 3598.6 49.83 95.50 0 3599.6 49.81 95.47 0
180355080 45088770 float sum -1 3652.9 49.37 94.63 0 3656.6 49.32 94.54 0
181403656 45350914 float sum -1 3680.6 49.29 94.46 0 3675.8 49.35 94.59 0
182452232 45613058 float sum -1 3671.7 49.69 95.24 0 3665.2 49.78 95.41 0
183500808 45875202 float sum -1 3683.8 49.81 95.47 0 3676.2 49.92 95.67 0
184549384 46137346 float sum -1 3709.8 49.75 95.35 0 3712.9 49.70 95.27 0
185597960 46399490 float sum -1 3730.0 49.76 95.37 0 3744.4 49.57 95.00 0
186646536 46661634 float sum -1 3741.9 49.88 95.60 0 3740.5 49.90 95.64 0
187695112 46923778 float sum -1 3772.5 49.75 95.36 0 3762.2 49.89 95.62 0
188743688 47185922 float sum -1 3806.4 49.59 95.04 0 3803.6 49.62 95.11 0
189792264 47448066 float sum -1 3828.3 49.58 95.02 0 3824.4 49.63 95.12 0
190840840 47710210 float sum -1 3854.0 49.52 94.91 0 3847.9 49.60 95.06 0
191889416 47972354 float sum -1 3877.7 49.49 94.85 0 3853.6 49.79 95.44 0
192937992 48234498 float sum -1 3895.5 49.53 94.93 0 3895.7 49.53 94.93 0
193986568 48496642 float sum -1 3906.9 49.65 95.17 0 3920.1 49.48 94.85 0
195035144 48758786 float sum -1 3907.5 49.91 95.67 0 3918.2 49.78 95.41 0
196083720 49020930 float sum -1 3972.2 49.36 94.62 0 3965.6 49.45 94.77 0
197132296 49283074 float sum -1 4007.2 49.19 94.29 0 3996.9 49.32 94.53 0
198180872 49545218 float sum -1 4010.2 49.42 94.72 0 3999.5 49.55 94.97 0
199229448 49807362 float sum -1 4019.1 49.57 95.01 0 4074.7 48.89 93.71 0
200278024 50069506 float sum -1 4039.3 49.58 95.03 0 4048.7 49.47 94.81 0
201326600 50331650 float sum -1 4073.2 49.43 94.73 0 4070.5 49.46 94.80 0
202375176 50593794 float sum -1 4075.7 49.65 95.17 0 4064.0 49.80 95.44 0
203423752 50855938 float sum -1 4104.5 49.56 94.99 0 4101.9 49.59 95.05 0
204472328 51118082 float sum -1 4126.1 49.56 94.98 0 4143.8 49.34 94.58 0
205520904 51380226 float sum -1 4141.6 49.62 95.11 0 4148.5 49.54 94.95 0
206569480 51642370 float sum -1 4167.0 49.57 95.01 0 4162.8 49.62 95.11 0
207618056 51904514 float sum -1 4189.3 49.56 94.99 0 4174.4 49.74 95.33 0
208666632 52166658 float sum -1 4203.9 49.64 95.14 0 4205.2 49.62 95.11 0
209715208 52428802 float sum -1 4219.0 49.71 95.27 0 4224.9 49.64 95.14 0
210763784 52690946 float sum -1 4258.2 49.50 94.87 0 4251.6 49.57 95.01 0
211812360 52953090 float sum -1 4292.2 49.35 94.58 0 4297.7 49.29 94.46 0
212860936 53215234 float sum -1 4374.6 48.66 93.26 0 4332.0 49.14 94.18 0
213909512 53477378 float sum -1 4321.2 49.50 94.88 0 4320.6 49.51 94.89 0
214958088 53739522 float sum -1 4354.4 49.37 94.62 0 4365.6 49.24 94.37 0
216006664 54001666 float sum -1 4366.4 49.47 94.82 0 4369.7 49.43 94.75 0
217055240 54263810 float sum -1 4388.9 49.46 94.79 0 4409.4 49.23 94.35 0
218103816 54525954 float sum -1 4400.3 49.57 95.00 0 4395.2 49.62 95.11 0
219152392 54788098 float sum -1 4437.4 49.39 94.66 0 4421.3 49.57 95.00 0
220200968 55050242 float sum -1 4489.0 49.05 94.02 0 4450.2 49.48 94.84 0
221249544 55312386 float sum -1 4485.7 49.32 94.54 0 4485.2 49.33 94.55 0
222298120 55574530 float sum -1 4512.6 49.26 94.42 0 4538.9 48.98 93.87 0
223346696 55836674 float sum -1 4529.7 49.31 94.51 0 4509.9 49.52 94.92 0
224395272 56098818 float sum -1 4530.3 49.53 94.94 0 4542.0 49.40 94.69 0
225443848 56360962 float sum -1 4557.5 49.47 94.81 0 4549.5 49.55 94.98 0
226492424 56623106 float sum -1 4586.0 49.39 94.66 0 4596.6 49.27 94.44 0
227541000 56885250 float sum -1 4657.5 48.85 93.64 0 4628.7 49.16 94.22 0
228589576 57147394 float sum -1 4670.5 48.94 93.81 0 4661.1 49.04 94.00 0
229638152 57409538 float sum -1 4664.5 49.23 94.36 0 4661.9 49.26 94.41 0
230686728 57671682 float sum -1 4676.2 49.33 94.55 0 4683.3 49.26 94.41 0
231735304 57933826 float sum -1 4710.1 49.20 94.30 0 4705.1 49.25 94.40 0
232783880 58195970 float sum -1 4737.8 49.13 94.17 0 4735.9 49.15 94.21 0
233832456 58458114 float sum -1 4738.9 49.34 94.57 0 4775.5 48.96 93.85 0
234881032 58720258 float sum -1 4754.9 49.40 94.68 0 4750.7 49.44 94.76 0
235929608 58982402 float sum -1 4779.2 49.37 94.62 0 4785.4 49.30 94.50 0
236978184 59244546 float sum -1 4804.1 49.33 94.55 0 4808.4 49.28 94.46 0
238026760 59506690 float sum -1 4826.3 49.32 94.53 0 4825.1 49.33 94.55 0
239075336 59768834 float sum -1 4869.2 49.10 94.11 0 4836.7 49.43 94.74 0
240123912 60030978 float sum -1 4863.9 49.37 94.62 0 4858.4 49.42 94.73 0
241172488 60293122 float sum -1 4902.2 49.20 94.29 0 4887.1 49.35 94.58 0
242221064 60555266 float sum -1 4912.7 49.31 94.50 0 4907.4 49.36 94.60 0
243269640 60817410 float sum -1 4983.1 48.82 93.57 0 4964.0 49.01 93.93 0
244318216 61079554 float sum -1 5004.6 48.82 93.57 0 4998.6 48.88 93.68 0
245366792 61341698 float sum -1 4998.9 49.08 94.08 0 5021.1 48.87 93.66 0
246415368 61603842 float sum -1 5011.3 49.17 94.25 0 5013.5 49.15 94.21 0
247463944 61865986 float sum -1 5048.8 49.01 93.94 0 5047.8 49.02 93.96 0
248512520 62128130 float sum -1 5072.4 48.99 93.90 0 5065.3 49.06 94.04 0
249561096 62390274 float sum -1 5065.3 49.27 94.43 0 5085.6 49.07 94.06 0
250609672 62652418 float sum -1 5084.5 49.29 94.47 0 5084.8 49.29 94.46 0
251658248 62914562 float sum -1 5116.5 49.19 94.27 0 5120.2 49.15 94.20 0
252706824 63176706 float sum -1 5144.3 49.12 94.15 0 5168.5 48.89 93.71 0
253755400 63438850 float sum -1 5166.5 49.12 94.14 0 5193.0 48.86 93.66 0
254803976 63700994 float sum -1 5178.1 49.21 94.31 0 5178.4 49.21 94.31 0
255852552 63963138 float sum -1 5203.3 49.17 94.25 0 5223.6 48.98 93.88 0
256901128 64225282 float sum -1 5229.6 49.12 94.16 0 5223.9 49.18 94.26 0
257949704 64487426 float sum -1 5236.0 49.26 94.42 0 5244.7 49.18 94.27 0
258998280 64749570 float sum -1 5301.3 48.86 93.64 0 5307.5 48.80 93.53 0
260046856 65011714 float sum -1 5340.9 48.69 93.32 0 5366.2 48.46 92.88 0
261095432 65273858 float sum -1 5338.9 48.90 93.73 0 5332.8 48.96 93.84 0
262144008 65536002 float sum -1 5355.2 48.95 93.82 0 5349.1 49.01 93.93 0
263192584 65798146 float sum -1 5385.2 48.87 93.67 0 5397.8 48.76 93.46 0
264241160 66060290 float sum -1 5401.5 48.92 93.76 0 5430.2 48.66 93.27 0
265289736 66322434 float sum -1 5405.0 49.08 94.07 0 5398.6 49.14 94.19 0
266338312 66584578 float sum -1 5418.2 49.16 94.22 0 5463.3 48.75 93.44 0
267386888 66846722 float sum -1 5462.5 48.95 93.82 0 5445.7 49.10 94.11 0
Out of bounds values : 0 OK
Avg bus bandwidth : 89.8902
Is this result normal? or the performance is low?