# \#iterations

**URL:** https://forums.developer.nvidia.com/tag/iterations/603.md

[Latest](https://forums.developer.nvidia.com/latest.md) · [Categories](https://forums.developer.nvidia.com/categories.md) · [Tags](https://forums.developer.nvidia.com/tags.md)

---

## [RDMA not working with ConnectX-6](https://forums.developer.nvidia.com/t/rdma-not-working-with-connectx-6/205913)

<div class="topic-metadata">

**Author:** [@nick.whitlock](https://forums.developer.nvidia.com/u/nick.whitlock)\
**Replies:** 2\
**Last updated:** [January 29, 2022, 12:05am UTC](https://forums.developer.nvidia.com/t/rdma-not-working-with-connectx-6/205913 "2022-01-29T00:05:41Z")

</div>

I am trying to get RoCEv2 working. I have two ConnectX-6 NICs (Mellanox Technologies MT2892 Family) connected to each other with an Ethernet cable. The firewall service is disabled. Using ibv\_devinfo, ifconfig, show\_gid…

---

## [HDR Infiniband and ConnectX-6 VPI interfaces](https://forums.developer.nvidia.com/t/hdr-infiniband-and-connectx-6-vpi-interfaces/205934)

<div class="topic-metadata">

**Author:** [@ITRider](https://forums.developer.nvidia.com/u/ITRider)\
**Replies:** 1\
**Last updated:** [January 11, 2022, 12:41am UTC](https://forums.developer.nvidia.com/t/hdr-infiniband-and-connectx-6-vpi-interfaces/205934 "2022-01-11T00:41:13Z")

</div>

Using infiniband-diags (35), we see ~twice the latency rate is the equipment is rated for: \[root@ne09 ~\]# ib\_send\_lat ne09-ib #bytes #iterations t\_min\[usec\] t\_max\[usec\] t\_typical\[usec\] t\_avg\[usec\] t\_stdev\[usec\] 99% per…

---

## [ib\_write\_bw does not go beyond 20G/s when using average packet size 512](https://forums.developer.nvidia.com/t/ib-write-bw-does-not-go-beyond-20g-s-when-using-average-packet-size-512/206002)

<div class="topic-metadata">

**Author:** [@badr.zarhri](https://forums.developer.nvidia.com/u/badr.zarhri)\
**Replies:** 3\
**Last updated:** [September 30, 2021, 2:33pm UTC](https://forums.developer.nvidia.com/t/ib-write-bw-does-not-go-beyond-20g-s-when-using-average-packet-size-512/206002 "2021-09-30T14:33:29Z")

</div>

Dear community, We tried also some home made tools, with multithreading, and multiple work requests in parallel, and the maximum we could reach is 60G/s. Is this the maximum speed the adapter can reach for this size of…

---

## [We have a few servers with MCX623106AS-CDAT Ethernet 100Gb 2-port QSFP56 cards.
Is there any published performance baseline for these cards? What am I supposed to see if I run a raw\_ethernet\_bw test between 2 of these?](https://forums.developer.nvidia.com/t/we-have-a-few-servers-with-mcx623106as-cdat-ethernet-100gb-2-port-qsfp56-cards-is-there-any-published-performance-baseline-for-these-cards-what-am-i-supposed-to-see-if-i-run-a-raw-ethernet-bw-test-between-2-of-these/206033)

<div class="topic-metadata">

**Author:** [@paolo.prinsecchi](https://forums.developer.nvidia.com/u/paolo.prinsecchi)\
**Replies:** 3\
**Last updated:** [September 14, 2021, 1:30am UTC](https://forums.developer.nvidia.com/t/we-have-a-few-servers-with-mcx623106as-cdat-ethernet-100gb-2-port-qsfp56-cards-is-there-any-published-performance-baseline-for-these-cards-what-am-i-supposed-to-see-if-i-run-a-raw-ethernet-bw-test-between-2-of-these/206033 "2021-09-14T01:30:50Z")

</div>

This is what I see, testing on 2 new HP dl385G10Plus v servers, with latest generation Epyc CPUs. One Arista 7800 switch between them. Ubuntu 20.04.1 , kernel 5.4.0-81-generic server: raw\_ethernet\_bw --server -d mlx5\_0…

---

## [Completion errors](https://forums.developer.nvidia.com/t/completion-errors/206081)

<div class="topic-metadata">

**Author:** [@alexander.a.oganezov](https://forums.developer.nvidia.com/u/alexander.a.oganezov)\
**Replies:** 1\
**Last updated:** [July 1, 2021, 5:21pm UTC](https://forums.developer.nvidia.com/t/completion-errors/206081 "2021-07-01T17:21:36Z")

</div>

During scaled up runs of DAOS stack on the Frontera/TACC cluster, we occasionally experience completion errors coming out of mlnx driver that look like: c121-063.frontera.tacc.utexas.edu ERROR 2021/04/05 17:38:12 daos\_e…

---

## [linux-rdma perftest ib\_read\_bw failure with use\_cuda option](https://forums.developer.nvidia.com/t/linux-rdma-perftest-ib-read-bw-failure-with-use-cuda-option/206087)

<div class="topic-metadata">

**Author:** [@jseong82.kim](https://forums.developer.nvidia.com/u/jseong82.kim)\
**Replies:** 0\
**Last updated:** [June 17, 2021, 8:16am UTC](https://forums.developer.nvidia.com/t/linux-rdma-perftest-ib-read-bw-failure-with-use-cuda-option/206087 "2021-06-17T08:16:41Z")

</div>

I’d like to measure GPU-GPU ib\_read / write bandwidth with varying combination of our PCIe topology within a single node. But, --use\_cuda option doesn’t work properly. Could you give me help for going further step to m…

---

## [RDMA issues "ConnectX-5 Ex 40GbE Dual-Port QSFP28"](https://forums.developer.nvidia.com/t/rdma-issues-connectx-5-ex-40gbe-dual-port-qsfp28/206126)

<div class="topic-metadata">

**Author:** [@elio1](https://forums.developer.nvidia.com/u/elio1)\
**Replies:** 4\
**Last updated:** [May 20, 2021, 8:16pm UTC](https://forums.developer.nvidia.com/t/rdma-issues-connectx-5-ex-40gbe-dual-port-qsfp28/206126 "2021-05-20T20:16:24Z")

</div>

Hey Everyone! I’m facing a strange issue here that I can’t seem to solve :( Long story short, I have around 20 servers to setup. After installing the first 9, I realized that some of them are acting up. The weird thi…

---

## [Is there any configuration that is limiting the virtual function bandwidth on connectx-5-ex adapter ?
We've working with this adapter and noticed that we are limited to 50gb when using the virtual function while the same limit does not occur with the pf](https://forums.developer.nvidia.com/t/is-there-any-configuration-that-is-limiting-the-virtual-function-bandwidth-on-connectx-5-ex-adapter-weve-working-with-this-adapter-and-noticed-that-we-are-limited-to-50gb-when-using-the-virtual-function-while-the-same-limit-does-not-occur-with-the-pf/206203)

<div class="topic-metadata">

**Author:** [@effi.shimon](https://forums.developer.nvidia.com/u/effi.shimon)\
**Replies:** 1\
**Last updated:** [February 8, 2021, 11:30am UTC](https://forums.developer.nvidia.com/t/is-there-any-configuration-that-is-limiting-the-virtual-function-bandwidth-on-connectx-5-ex-adapter-weve-working-with-this-adapter-and-noticed-that-we-are-limited-to-50gb-when-using-the-virtual-function-while-the-same-limit-does-not-occur-with-the-pf/206203 "2021-02-08T11:30:47Z")

</div>

IB\_SEND\_BW result: PF: #bytes #iterations BW peak\[Gb/sec\] BW average\[Gb/sec\] MsgRate\[Mpps\] 65536 344452 0.00 60.19 0.114807 65536 344471 0.00 60.20 0.114814 Single VF: #bytes #iterations BW peak\[Gb/sec\] BW averag…

---

## [Can't use RDMA on Azure NC24rs\_v3](https://forums.developer.nvidia.com/t/cant-use-rdma-on-azure-nc24rs-v3/206525)

<div class="topic-metadata">

**Author:** [@system](https://forums.developer.nvidia.com/u/system)\
**Replies:** 2\
**Last updated:** [January 10, 2021, 7:40pm UTC](https://forums.developer.nvidia.com/t/cant-use-rdma-on-azure-nc24rs-v3/206525 "2021-01-10T19:40:11Z")

</div>

I tried to use infiniband on Standard\_NC24rs\_v3 Ubuntu 16.04 , but failed. I installed https://www.mellanox.com/downloads/ofed/MLNX\_OFED-4.7-3.2.9.0/MLNX\_OFED\_LINUX-4.7-3.2.9.0-ubuntu18.04-x86\_64.tgz from https://docs.m…

---

## [GPUDirect RDMA at the ibverbs level.](https://forums.developer.nvidia.com/t/gpudirect-rdma-at-the-ibverbs-level/206260)

<div class="topic-metadata">

**Author:** [@OmarAlama](https://forums.developer.nvidia.com/u/OmarAlama)\
**Replies:** 4\
**Last updated:** [November 30, 2020, 10:51am UTC](https://forums.developer.nvidia.com/t/gpudirect-rdma-at-the-ibverbs-level/206260 "2020-11-30T10:51:42Z")

</div>

The goal is simple. I want to use GPU Direct RDMA at the ibverbs level. I I do not want to use any cuda aware MPI implementation because I require a greater level of control. I want to perform transfers from a gpu to a …

---

## [Pinging between KVM guests using IPOIB over Virtual Functions.](https://forums.developer.nvidia.com/t/pinging-between-kvm-guests-using-ipoib-over-virtual-functions/206629)

<div class="topic-metadata">

**Author:** [@gurvindernarula1](https://forums.developer.nvidia.com/u/gurvindernarula1)\
**Replies:** 1\
**Last updated:** [January 14, 2020, 12:05am UTC](https://forums.developer.nvidia.com/t/pinging-between-kvm-guests-using-ipoib-over-virtual-functions/206629 "2020-01-14T00:05:19Z")

</div>

Hello, I am trying to setup connectivity over IPoIB between 2 KVM Guests. On the host I have SR-IOV setup on ConnectX-3 VPI card with virtual function configured. On the host root@pve:~# lspci | grep Mell 0a:00.1 In…

---

## [My softroce (on CX4 card) can't work with roce (on CX4 card)](https://forums.developer.nvidia.com/t/my-softroce-on-cx4-card-cant-work-with-roce-on-cx4-card/206621)

<div class="topic-metadata">

**Author:** [@1692492213](https://forums.developer.nvidia.com/u/1692492213)\
**Replies:** 8\
**Last updated:** [November 25, 2019, 9:05am UTC](https://forums.developer.nvidia.com/t/my-softroce-on-cx4-card-cant-work-with-roce-on-cx4-card/206621 "2019-11-25T09:05:29Z")

</div>

The softroce vertion (rdma-core vertion is 26.0 and perftest vertion is 4.8, both are the latest version in https://github.com/linux-rdma/). And the MLNX-OFED version on hard roce is MLNX\_OFED\_LINUX-4.5-1.0.1.0-ubuntu16.…

---

## [Can SOFTROCE on a client send message to ROCE on a server(not mlx5 card, it's a roce card from another company) by using ib\_send\_bw?](https://forums.developer.nvidia.com/t/can-softroce-on-a-client-send-message-to-roce-on-a-server-not-mlx5-card-its-a-roce-card-from-another-company-by-using-ib-send-bw/206663)

<div class="topic-metadata">

**Author:** [@1692492213](https://forums.developer.nvidia.com/u/1692492213)\
**Replies:** 1\
**Last updated:** [November 14, 2019, 1:24am UTC](https://forums.developer.nvidia.com/t/can-softroce-on-a-client-send-message-to-roce-on-a-server-not-mlx5-card-its-a-roce-card-from-another-company-by-using-ib-send-bw/206663 "2019-11-14T01:24:40Z")

</div>

We try to let SOFTROCE on a client send message to ROCE on a server(not mlx5 card, it’s a roce card from another company) by using ib\_send\_bw. But it fails, the error message is: Send BW Test Dual-port : OFF Device : …

---

## [iSer over SR-IOV.](https://forums.developer.nvidia.com/t/iser-over-sr-iov/206752)

<div class="topic-metadata">

**Author:** [@gurvindernarula1](https://forums.developer.nvidia.com/u/gurvindernarula1)\
**Replies:** 2\
**Last updated:** [August 25, 2019, 2:08am UTC](https://forums.developer.nvidia.com/t/iser-over-sr-iov/206752 "2019-08-25T02:08:28Z")

</div>

Hello, I’m running into an issue initiating an iSer connection to a iSer enabled storage server. The connections times out when the initiator tries to log into the target. The iser initiator is a KVM Guest running Ubunt…

---

## [Device Memory MR and GPU RDMA](https://forums.developer.nvidia.com/t/device-memory-mr-and-gpu-rdma/206912)

<div class="topic-metadata">

**Author:** [@saptarshi.mrg](https://forums.developer.nvidia.com/u/saptarshi.mrg)\
**Replies:** 1\
**Last updated:** [February 13, 2019, 3:38pm UTC](https://forums.developer.nvidia.com/t/device-memory-mr-and-gpu-rdma/206912 "2019-02-13T15:38:59Z")

</div>

Is it legal to pass the dma address of a GPU buffer to the Ib verbs for Memory Registration for initiating RDMA. Does MR differentiate between device memory/cpu memory. In my NVME-OF test setup, I do a 128K read over fa…

---

## [ib\_send\_bw ignores '-d 'option on client side,why?](https://forums.developer.nvidia.com/t/ib-send-bw-ignores-d-option-on-client-side-why/206955)

<div class="topic-metadata">

**Author:** [@jianguo72](https://forums.developer.nvidia.com/u/jianguo72)\
**Replies:** 1\
**Last updated:** [December 23, 2018, 1:10pm UTC](https://forums.developer.nvidia.com/t/ib-send-bw-ignores-d-option-on-client-side-why/206955 "2018-12-23T13:10:33Z")

</div>

I used ‘-d’ to indicate the devices to send RDMA traffic to server, but ib\_send\_bw either ignores this option to use another device: for example, here I want to use dev mlx5\_0, but ib\_send\_bw choose mlx5\_1. is it an is…

---

## [Odd, unsymmetric ib\_send\_lat results?](https://forums.developer.nvidia.com/t/odd-unsymmetric-ib-send-lat-results/206979)

<div class="topic-metadata">

**Author:** [@fangchin](https://forums.developer.nvidia.com/u/fangchin)\
**Replies:** 14\
**Last updated:** [December 11, 2018, 1:58pm UTC](https://forums.developer.nvidia.com/t/odd-unsymmetric-ib-send-lat-results/206979 "2018-12-11T13:58:29Z")

</div>

I have two small IB clusters set up for testing: Both have SB7700 IB switch Two servers, each with a MCX455A-ECAT ConnectX-4 VPI adapter, are connected to each switch. Essential system and software info: \[root@fs00 ~…

---

## [ib\_send\_bw test with ConnectX-3 VPI adapter card over XenServer 6.2 fails with message: Failed to modify QP 100 to RTR](https://forums.developer.nvidia.com/t/ib-send-bw-test-with-connectx-3-vpi-adapter-card-over-xenserver-6-2-fails-with-message-failed-to-modify-qp-100-to-rtr/206984)

<div class="topic-metadata">

**Author:** [@javi3388](https://forums.developer.nvidia.com/u/javi3388)\
**Replies:** 2\
**Last updated:** [December 10, 2018, 4:16pm UTC](https://forums.developer.nvidia.com/t/ib-send-bw-test-with-connectx-3-vpi-adapter-card-over-xenserver-6-2-fails-with-message-failed-to-modify-qp-100-to-rtr/206984 "2018-12-10T16:16:44Z")

</div>

Hi all, we are testing a ConnectX-3 card with XenServer 6.2. For that, we have two nodes, the first node has XenServer 6.2 dom0 and the second node has CentOS 6.4. In both nodes we have installed the MLNX\_OFED version 2…

---

## [Can't get full FDR bandwidth with Connect-IB card in PCI 2.0 x16 slot](https://forums.developer.nvidia.com/t/cant-get-full-fdr-bandwidth-with-connect-ib-card-in-pci-2-0-x16-slot/207133)

<div class="topic-metadata">

**Author:** [@system](https://forums.developer.nvidia.com/u/system)\
**Replies:** 3\
**Last updated:** [July 20, 2018, 9:11pm UTC](https://forums.developer.nvidia.com/t/cant-get-full-fdr-bandwidth-with-connect-ib-card-in-pci-2-0-x16-slot/207133 "2018-07-20T21:11:55Z")

</div>

Greetings, We have some nodes (Dell R415s running RHEL 6.8) with Connect-IB cards in a PCI 2.0 x16 slot (the only one available) and can’t seem to get more than 45 Gbit/s using ib\_send\_bw. I have two of the nodes connec…

---

## [Poor bandwidth performance when running with large block size](https://forums.developer.nvidia.com/t/poor-bandwidth-performance-when-running-with-large-block-size/207162)

<div class="topic-metadata">

**Author:** [@wonzhq](https://forums.developer.nvidia.com/u/wonzhq)\
**Replies:** 9\
**Last updated:** [May 28, 2018, 1:50pm UTC](https://forums.developer.nvidia.com/t/poor-bandwidth-performance-when-running-with-large-block-size/207162 "2018-05-28T13:50:45Z")

</div>

Hi all, I have a cluster running ROCE on Mellanox NIC. # lspci | grep Mellanox 03:00.0 Ethernet controller: Mellanox Technologies MT27800 Family \[ConnectX-5\] 03:00.1 Ethernet controller: Mellanox Technologies MT27800…

---

## [mellanox connectX3 adapter failed to reach the rational speed](https://forums.developer.nvidia.com/t/mellanox-connectx3-adapter-failed-to-reach-the-rational-speed/207282)

<div class="topic-metadata">

**Author:** [@system](https://forums.developer.nvidia.com/u/system)\
**Replies:** 1\
**Last updated:** [December 22, 2017, 6:44am UTC](https://forums.developer.nvidia.com/t/mellanox-connectx3-adapter-failed-to-reach-the-rational-speed/207282 "2017-12-22T06:44:53Z")

</div>

Hello, We are using the mellanox connectx3 adapter(Link Speed:40Gps) on windows10 PCs. We also use the proper infiniband cable for this kind of adapter which can provide the rational infiniband speed . However, we stil…

---

## [ib\_read/write\_bw can\`t use the "--run\_infinitely" parameter?](https://forums.developer.nvidia.com/t/ib-read-write-bw-can-t-use-the-run-infinitely-parameter/207300)

<div class="topic-metadata">

**Author:** [@kuangzhuqing](https://forums.developer.nvidia.com/u/kuangzhuqing)\
**Replies:** 8\
**Last updated:** [December 10, 2017, 1:25pm UTC](https://forums.developer.nvidia.com/t/ib-read-write-bw-can-t-use-the-run-infinitely-parameter/207300 "2017-12-10T13:25:47Z")

</div>

When using the ib\_read / write\_bw tool, I found the following problem： All the items are zero. Using the “perfquery” tool almost can not monitor the sending of data packets. \[root@localhost infiniband\]# ib\_read\_…

---

## [Infiniband performance tuning](https://forums.developer.nvidia.com/t/infiniband-performance-tuning/207467)

<div class="topic-metadata">

**Author:** [@teja781](https://forums.developer.nvidia.com/u/teja781)\
**Replies:** 2\
**Last updated:** [June 7, 2017, 2:05pm UTC](https://forums.developer.nvidia.com/t/infiniband-performance-tuning/207467 "2017-06-07T14:05:21Z")

</div>

Hello, I am trying to tune the setup I have to improve the performance of the infiniband network to the maximum possible. The current setup is as follows. 1 Dell T3600 quad core, 3.6GHz node 1 Supermicro dual quad cor…

---

## [Own libibverbs application similar to ib\_send\_bw with low throughput](https://forums.developer.nvidia.com/t/own-libibverbs-application-similar-to-ib-send-bw-with-low-throughput/207520)

<div class="topic-metadata">

**Author:** [@michael.schoettner](https://forums.developer.nvidia.com/u/michael.schoettner)\
**Replies:** 0\
**Last updated:** [April 10, 2017, 7:23am UTC](https://forums.developer.nvidia.com/t/own-libibverbs-application-similar-to-ib-send-bw-with-low-throughput/207520 "2017-04-10T07:23:36Z")

</div>

Dear Mellanox Community, I created a small application that mimics the concept of ib\_send\_bw to get familiar with libibverbs and programming for InfiniBand hardware. With the source code of ib\_send\_bw and resources like…

---

## [CentOS 7 KVM-SR-IOV Performance?](https://forums.developer.nvidia.com/t/centos-7-kvm-sr-iov-performance/207863)

<div class="topic-metadata">

**Author:** [@mkkang01](https://forums.developer.nvidia.com/u/mkkang01)\
**Replies:** 12\
**Last updated:** [January 10, 2017, 3:11pm UTC](https://forums.developer.nvidia.com/t/centos-7-kvm-sr-iov-performance/207863 "2017-01-10T15:11:27Z")

</div>

Hello, I have a big problem to get the right performance for KVM virtualization (SR-IOV) on top of CentOS7 Host. I’m using ConnectX-3 card and MLNX\_OFED\_LINUX-3.0-2.0.1 version. The firmware is up-to-date (2.34.5000). …

---

## [40Gb/s IPoIB only gives 5Gb/s real throughput?!](https://forums.developer.nvidia.com/t/40gb-s-ipoib-only-gives-5gb-s-real-throughput/207645)

<div class="topic-metadata">

**Author:** [@daniel.infuehr](https://forums.developer.nvidia.com/u/daniel.infuehr)\
**Replies:** 9\
**Last updated:** [December 8, 2016, 4:35am UTC](https://forums.developer.nvidia.com/t/40gb-s-ipoib-only-gives-5gb-s-real-throughput/207645 "2016-12-08T04:35:57Z")

</div>

I really need some expertise here: For the current situation see update 2 below. Update: I tested with Windows 2012 Clients to verify and I still get about 5.5 Gbit/s max. Maybe someone has other 40Gbit adapters what …

---

## [SX6036 IPoIB speed issue](https://forums.developer.nvidia.com/t/sx6036-ipoib-speed-issue/207648)

<div class="topic-metadata">

**Author:** [@zbruce-merrie](https://forums.developer.nvidia.com/u/zbruce-merrie)\
**Replies:** 2\
**Last updated:** [November 3, 2016, 10:29am UTC](https://forums.developer.nvidia.com/t/sx6036-ipoib-speed-issue/207648 "2016-11-03T10:29:54Z")

</div>

Hi, I was hoping that someone would be able to help me. I have 15 HP DL380 gen9 servers running Windows Server 2012 R2 each with dual port HP Connect-X3 VPI cards connected to a Mellanox SX6036 switch. The line speed …

---

## [ConnectX-2 IPoIB Performance Help](https://forums.developer.nvidia.com/t/connectx-2-ipoib-performance-help/207647)

<div class="topic-metadata">

**Author:** [@daniel.infuehr](https://forums.developer.nvidia.com/u/daniel.infuehr)\
**Replies:** 1\
**Last updated:** [October 13, 2016, 5:22pm UTC](https://forums.developer.nvidia.com/t/connectx-2-ipoib-performance-help/207647 "2016-10-13T17:22:57Z")

</div>

I really need some expertise here: I have two Windows 10 Machines with two MHQH19B-XTR 40 Gbit Adapters and a QSFP cable in between. The Vlan manager is opensm. The connection should be about 32Gbits Lan. In reality i …

---

## [Is this the best our FDR adapters can do?](https://forums.developer.nvidia.com/t/is-this-the-best-our-fdr-adapters-can-do/207661)

<div class="topic-metadata">

**Author:** [@fangchin](https://forums.developer.nvidia.com/u/fangchin)\
**Replies:** 3\
**Last updated:** [August 18, 2016, 7:32pm UTC](https://forums.developer.nvidia.com/t/is-this-the-best-our-fdr-adapters-can-do/207661 "2016-08-18T19:32:34Z")

</div>

We have a small test setup illustrated below. I have done some ib\_write\_bw tests. Got “decent” numbers, but not as fast as I anticipated. First, some background of the setup: Two 1U storage servers each has a EDR HCA…

---

## [ib\_send\_bw performance puzzle](https://forums.developer.nvidia.com/t/ib-send-bw-performance-puzzle/207732)

<div class="topic-metadata">

**Author:** [@system](https://forums.developer.nvidia.com/u/system)\
**Replies:** 4\
**Last updated:** [April 27, 2016, 10:35pm UTC](https://forums.developer.nvidia.com/t/ib-send-bw-performance-puzzle/207732 "2016-04-27T22:35:27Z")

</div>

I tried play with ib\_send\_bw with different message sizes. I noticed that the best performance can be reached if the size is power of two. I tried different size with 1 byte smaller or 1 byte larger, the results are surp…

[Next page](https://forums.developer.nvidia.com/tag/iterations/603.md?match_all_tags=true&page=1&tags%5B%5D=iterations)
