It is intended to convey the number of data bus wires used to implement the electrical interface between the GPU chip itself and the GPU DRAM memory (which is off-chip). It would be approximately correct to say it is the bus width between (off-chip) device memory and the L2 cache.
I’m not aware of any CUDA GPU ever that had a L2 cache line of 128 bytes. To my knowledge it is uniformly 32 bytes. An L2 cache line miss would typically be serviced from (off-chip) DRAM memory. The transactions associated with the DRAM bus are not typically at the width of the DRAM bus. Typically, the DRAM bus is broken into so-called “partitions”. Depending on the mapping of L2 cache lines to physical DRAM memory, the memory controller will issue one or more transactions on one or more partitions, to service L2 cache miss(es).
An L1 cache miss would usually attempt to hit in L2 first. If it missed in L2, see above.
I’m not sure it is documented, but if we were for example to pretend that a partition has a width of 64 bits, then a cache line miss for L2 would require 4 partition transactions. Whether those partition transactions would all occur on the same partition or would be spread across partitions is something that I can’t answer, is generally undocumented, and probably depends on GPU architecture, the exact addresses involved, and possibly other factors. You can imagine that the GPU designers want to make it such that typical data access patterns will tend to fall into a mapping arrangement such that DRAM bus bandwidth will be maximally/optimally utilized.
GPUs have varying DRAM bus widths. I have seen GPUs that have a bus width as low as 64 bits and as high as 4096 bits (I think A100 GPUs are even higher, but I haven’t checked). If you have a GPU where the bus width is 256 bits, then that is because the GPU designers felt that was best for that particular product/SKU. I think there are many factors that go into such a design choice, including cost, DRAM technology, and desired bandwidth for the product segment (i.e. desired performance).