# Bank Conflicts and Serialized Warps

**URL:** <https://forums.developer.nvidia.com/t/bank-conflicts-and-serialized-warps/13603>\
**Category:** CUDA Programming and Performance\
**Created:** [December 3, 2009, 5:18pm UTC](https://forums.developer.nvidia.com/t/bank-conflicts-and-serialized-warps/13603 "2009-12-03T17:18:14Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Basilios](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/basilios/32/451167_2.png) [@Basilios](https://forums.developer.nvidia.com/u/Basilios)\
**Post date:** [December 3, 2009, 5:18pm UTC](https://forums.developer.nvidia.com/t/bank-conflicts-and-serialized-warps/13603/1 "2009-12-03T17:18:14Z")

</div>

I’ve been running a few small kernels through the CUDA Visual Profiler to test my understanding of shared memory, and I’ve run into a bit of a problem.

I’m _almost_ certain that my kernel should be causing 16-way bank conflicts, but according to the “warp\_serialize” field in the profiler, this isn’t the case.

The kernel is listed below:

[codebox] **global** void bank\_conflicts(float\* array) {

```
extern __shared__ float cache[];

float tmp = cache[threadIdx.x * 16];

array[threadIdx.x] = tmp;

```

}[/codebox]

For each thread in a warp, (threadIdx.x \* 16) % 16 will evaluate to 0, which I thought would force all threads to access the same shared memory bank. If this is the case, then all warps should be serialized.

However, if I run this kernel using a single block of 32 threads then the Visual Profiler reports that “warp\_serialize” is equal to 120. I could almost understand if this value was 128 (each of the 32 threads reads 4 bytes) but that it is 120 has me completely baffled!

If anybody could shed light on the situation, it would be much appreciated!

EDIT: Another curious point of note is that the Visual Profiler reports that this kernel contains 5 branches, and that 1 of them is divergent. Does anybody know what’s going on?

---

<div class="post-metadata">

**Author:** ![\_Big\_Mac](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@\_Big\_Mac](https://forums.developer.nvidia.com/u/_Big_Mac)\
**Post date:** [December 3, 2009, 7:33pm UTC](https://forums.developer.nvidia.com/t/bank-conflicts-and-serialized-warps/13603/2 "2009-12-03T19:33:20Z")

</div>

If all threads in a half-warp access a single address, shared mem goes into broadcast mode and delivers all data in a single cycle.

---

<div class="post-metadata">

**Author:** ![Basilios](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/basilios/32/451167_2.png) [@Basilios](https://forums.developer.nvidia.com/u/Basilios)\
**Post date:** [December 3, 2009, 8:18pm UTC](https://forums.developer.nvidia.com/t/bank-conflicts-and-serialized-warps/13603/3 "2009-12-03T20:18:19Z")

</div>

> [@](#):
>
> If all threads in a half-warp access a single address, shared mem goes into broadcast mode and delivers all data in a single cycle.

Maybe I’m being stupid, but how would that happen in this case?

The value of threadIdx.x is different for each thread, so cache[threadIdx.x \* 16] will evaluate to a different memory location on each thread. My understanding is that this would result in them accessing the same bank, but with different addresses.

---

<div class="post-metadata">

**Author:** ![\_Big\_Mac](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@\_Big\_Mac](https://forums.developer.nvidia.com/u/_Big_Mac)\
**Post date:** [December 3, 2009, 9:18pm UTC](https://forums.developer.nvidia.com/t/bank-conflicts-and-serialized-warps/13603/4 "2009-12-03T21:18:03Z")

</div>

Hm, that was a brainfart on my part. Disregard.

---

<div class="post-metadata">

**Author:** ![Gregory\_Diamos](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@Gregory\_Diamos](https://forums.developer.nvidia.com/u/Gregory_Diamos)\
**Post date:** [December 3, 2009, 10:02pm UTC](https://forums.developer.nvidia.com/t/bank-conflicts-and-serialized-warps/13603/5 "2009-12-03T22:02:15Z")

</div>

Perhaps they don’t count the first thread when reporting on warp\_serialize? Otherwise it would be at least 1 for any program.

---

<div class="post-metadata">

**Author:** ![\_PM](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/_pm/32/20210_2.png) [@\_PM](https://forums.developer.nvidia.com/u/_PM)\
**Post date:** [December 3, 2009, 10:24pm UTC](https://forums.developer.nvidia.com/t/bank-conflicts-and-serialized-warps/13603/6 "2009-12-03T22:24:24Z")

</div>

I thought the traditional trick was to allocate a shared array of n\*17 to force offseted access.  
[url=“[http://people.maths.ox.ac.uk/~gilesm/hpc/NVIDIA/CUDA\_Optimization\_Harris.pdf](http://people.maths.ox.ac.uk/~gilesm/hpc/NVIDIA/CUDA_Optimization_Harris.pdf)”][http://people.maths.ox.ac.uk/~gilesm/hpc/N...tion\_Harris.pdf[/url]](http://people.maths.ox.ac.uk/~gilesm/hpc/N...tion_Harris.pdf%5B/url%5D)  
look p.41

EDIT: I read you post too fast. But if you did an allocation of cache with a size n %16 != 0 then it’s the reason you don’t have bank conflict.

---

<div class="post-metadata">

**Author:** ![Basilios](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/basilios/32/451167_2.png) [@Basilios](https://forums.developer.nvidia.com/u/Basilios)\
**Post date:** [December 4, 2009, 9:23am UTC](https://forums.developer.nvidia.com/t/bank-conflicts-and-serialized-warps/13603/7 "2009-12-04T09:23:31Z")

</div>

> [@](#):
>
> EDIT: I read you post too fast. But if you did an allocation of cache with a size n %16 != 0 then it’s the reason you don’t have bank conflict.

I can see why that would happen, but the amount of shared memory I’m allocating is defined as threads \* 16, so size % 16 will always be 0.

(Thanks, by the way, for the tip about padding shared memory; I’m sure that will come in useful later.)
