The core issue with multi-stage pipelining is understanding what the bandwidth truly is. Suppose there are two “memory read units” with constant read speeds. In this case, the bandwidth actually depends on whether these two units are continuously issuing read requests.
If the two units are connected to a single pipeline with a flow rate limit, then no matter how many requests they make, the total bandwidth won’t increase and will be limited by the pipeline’s capacity. Common sense suggests it should be the latter, but I’ve come to realize that, at least with DRAM, it seems to be the former(DRAM has many ports and each has const bandwidth?).
For example, when I measure the L1 bandwidth, if a block occupies an entire SM, the more threads there are within the block, the higher the bandwidth(total data amount/time) linearly increases. This seems to indicate that even with 1024 threads, simple operations of reading values from global memory to shared memory do not fully utilize the L1 bandwidth.