# atomicAdd support Double Which card?

**URL:** <https://forums.developer.nvidia.com/t/atomicadd-support-double-which-card/19217>\
**Category:** CUDA Programming and Performance\
**Created:** [October 14, 2010, 1:20pm UTC](https://forums.developer.nvidia.com/t/atomicadd-support-double-which-card/19217 "2010-10-14T13:20:16Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![London](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@London](https://forums.developer.nvidia.com/u/London)\
**Post date:** [October 14, 2010, 1:20pm UTC](https://forums.developer.nvidia.com/t/atomicadd-support-double-which-card/19217/1 "2010-10-14T13:20:16Z")

</div>

Hi,

Which card model is going to support/supports atomicAdd working on double types?

Thaks

---

<div class="post-metadata">

**Author:** ![London](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@London](https://forums.developer.nvidia.com/u/London)\
**Post date:** [October 14, 2010, 1:20pm UTC](https://forums.developer.nvidia.com/t/atomicadd-support-double-which-card/19217/2 "2010-10-14T13:20:16Z")

</div>

Hi,

Which card model is going to support/supports atomicAdd working on double types?

Thaks

---

<div class="post-metadata">

**Author:** ![avidday](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/avidday/32/59219_2.png) [@avidday](https://forums.developer.nvidia.com/u/avidday)\
**Post date:** [October 14, 2010, 2:01pm UTC](https://forums.developer.nvidia.com/t/atomicadd-support-double-which-card/19217/3 "2010-10-14T14:01:56Z")

</div>

None of them, as far as I can tell. Fermi adds 32 bit floating point support for AtomicAdd(), but that’s it.

---

<div class="post-metadata">

**Author:** ![avidday](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/avidday/32/59219_2.png) [@avidday](https://forums.developer.nvidia.com/u/avidday)\
**Post date:** [October 14, 2010, 2:01pm UTC](https://forums.developer.nvidia.com/t/atomicadd-support-double-which-card/19217/4 "2010-10-14T14:01:56Z")

</div>

None of them, as far as I can tell. Fermi adds 32 bit floating point support for AtomicAdd(), but that’s it.

---

<div class="post-metadata">

**Author:** ![London](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@London](https://forums.developer.nvidia.com/u/London)\
**Post date:** [October 17, 2010, 9:02am UTC](https://forums.developer.nvidia.com/t/atomicadd-support-double-which-card/19217/5 "2010-10-17T09:02:22Z")

</div>

> [@](#):
>
> None of them, as far as I can tell. Fermi adds 32 bit floating point support for AtomicAdd(), but that’s it.

Does that mean we can only use float at maximum if we need to use atomic operations?

If the values we have is double with 6 decimal places, assign them to float variables, how much accuracy we will lose? If that is okey, then we can use the converted float variables with the atomic operations.

---

<div class="post-metadata">

**Author:** ![London](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@London](https://forums.developer.nvidia.com/u/London)\
**Post date:** [October 17, 2010, 9:02am UTC](https://forums.developer.nvidia.com/t/atomicadd-support-double-which-card/19217/6 "2010-10-17T09:02:22Z")

</div>

> [@](#):
>
> None of them, as far as I can tell. Fermi adds 32 bit floating point support for AtomicAdd(), but that’s it.

Does that mean we can only use float at maximum if we need to use atomic operations?

If the values we have is double with 6 decimal places, assign them to float variables, how much accuracy we will lose? If that is okey, then we can use the converted float variables with the atomic operations.

---

<div class="post-metadata">

**Author:** ![jack](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jack/32/12069_2.png) [@jack](https://forums.developer.nvidia.com/u/jack)\
**Post date:** [October 17, 2010, 3:02pm UTC](https://forums.developer.nvidia.com/t/atomicadd-support-double-which-card/19217/7 "2010-10-17T15:02:01Z")

</div>

> [@](#):
>
> Does that mean we can only use float at maximum if we need to use atomic operations?
> 
> If the values we have is double with 6 decimal places, assign them to float variables, how much accuracy we will lose? If that is okey, then we can use the converted float variables with the atomic operations.

Yes, that means that you can only use single-precision floats with atomic operations.

However, single-precision float is accurate to ~7 decimal places, so in this case, you shouldn’t lose any precision.

---

<div class="post-metadata">

**Author:** ![jack](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jack/32/12069_2.png) [@jack](https://forums.developer.nvidia.com/u/jack)\
**Post date:** [October 17, 2010, 3:02pm UTC](https://forums.developer.nvidia.com/t/atomicadd-support-double-which-card/19217/8 "2010-10-17T15:02:01Z")

</div>

> [@](#):
>
> Does that mean we can only use float at maximum if we need to use atomic operations?
> 
> If the values we have is double with 6 decimal places, assign them to float variables, how much accuracy we will lose? If that is okey, then we can use the converted float variables with the atomic operations.

Yes, that means that you can only use single-precision floats with atomic operations.

However, single-precision float is accurate to ~7 decimal places, so in this case, you shouldn’t lose any precision.

---

<div class="post-metadata">

**Author:** ![London](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@London](https://forums.developer.nvidia.com/u/London)\
**Post date:** [October 17, 2010, 7:49pm UTC](https://forums.developer.nvidia.com/t/atomicadd-support-double-which-card/19217/9 "2010-10-17T19:49:05Z")

</div>

> [@](#):
>
> Yes, that means that you can only use single-precision floats with atomic operations.
> 
> However, single-precision float is accurate to ~7 decimal places, so in this case, you shouldn’t lose any precision.

Thanks for the reply. The card we have now is only support Integer with the Atomic operation. Does that mean when we assign the double value to a integer type variable, we will lose all the digits after the decimal points?

By the way, I am very interested in the [GPU.Net](http://GPU.Net). I will give a it try when the beta is released.

---

<div class="post-metadata">

**Author:** ![London](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@London](https://forums.developer.nvidia.com/u/London)\
**Post date:** [October 17, 2010, 7:49pm UTC](https://forums.developer.nvidia.com/t/atomicadd-support-double-which-card/19217/10 "2010-10-17T19:49:05Z")

</div>

> [@](#):
>
> Yes, that means that you can only use single-precision floats with atomic operations.
> 
> However, single-precision float is accurate to ~7 decimal places, so in this case, you shouldn’t lose any precision.

Thanks for the reply. The card we have now is only support Integer with the Atomic operation. Does that mean when we assign the double value to a integer type variable, we will lose all the digits after the decimal points?

By the way, I am very interested in the [GPU.Net](http://GPU.Net). I will give a it try when the beta is released.

---

<div class="post-metadata">

**Author:** ![Magorath](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@Magorath](https://forums.developer.nvidia.com/u/Magorath)\
**Post date:** [October 18, 2010, 6:42am UTC](https://forums.developer.nvidia.com/t/atomicadd-support-double-which-card/19217/11 "2010-10-18T06:42:31Z")

</div>

You can write your own atomic function using the other ones:

[codebox] **device** inline void myAtomicAdd(double \*address, double value) //See CUDA official forum

{

```
unsigned long long oldval, newval, readback;

```

oldval = \_\_double\_as\_longlong(\*address);

```
newval = __double_as_longlong(__longlong_as_double(oldval) + value);

while ((readback=atomicCAS((unsigned long long *)address, oldval, newval)) != oldval)

{

    oldval = readback;

    newval = __double_as_longlong(__longlong_as_double(oldval) + value);

}

```

}[/codebox]

---

<div class="post-metadata">

**Author:** ![Magorath](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@Magorath](https://forums.developer.nvidia.com/u/Magorath)\
**Post date:** [October 18, 2010, 6:42am UTC](https://forums.developer.nvidia.com/t/atomicadd-support-double-which-card/19217/12 "2010-10-18T06:42:31Z")

</div>

You can write your own atomic function using the other ones:

[codebox] **device** inline void myAtomicAdd(double \*address, double value) //See CUDA official forum

{

```
unsigned long long oldval, newval, readback;

```

oldval = \_\_double\_as\_longlong(\*address);

```
newval = __double_as_longlong(__longlong_as_double(oldval) + value);

while ((readback=atomicCAS((unsigned long long *)address, oldval, newval)) != oldval)

{

    oldval = readback;

    newval = __double_as_longlong(__longlong_as_double(oldval) + value);

}

```

}[/codebox]

---

<div class="post-metadata">

**Author:** ![London](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@London](https://forums.developer.nvidia.com/u/London)\
**Post date:** [October 18, 2010, 8:50am UTC](https://forums.developer.nvidia.com/t/atomicadd-support-double-which-card/19217/13 "2010-10-18T08:50:40Z")

</div>

> [@](#):
>
> You can write your own atomic function using the other ones:
> 
> [codebox] **device** inline void myAtomicAdd(double \*address, double value) //See CUDA official forum
> 
> {
> 
> ```
> unsigned long long oldval, newval, readback;
> 
> ```
> 
> oldval = \_\_double\_as\_longlong(\*address);
> 
> ```
> newval = __double_as_longlong(__longlong_as_double(oldval) + value);
> 
> while ((readback=atomicCAS((unsigned long long *)address, oldval, newval)) != oldval)
> 
> {
> 
> oldval = readback;
> 
> newval = __double_as_longlong(__longlong_as_double(oldval) + value);
> 
> }
> 
> ```
> 
> }[/codebox]

Hi Magorath,

Thanks for the information.

How is the performance of this custom atomicadd function? Which CUDA official forum(in the code comments) is it from?

Why NVIDIA does not build this function with the the current cards?

---

<div class="post-metadata">

**Author:** ![London](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@London](https://forums.developer.nvidia.com/u/London)\
**Post date:** [October 18, 2010, 8:50am UTC](https://forums.developer.nvidia.com/t/atomicadd-support-double-which-card/19217/14 "2010-10-18T08:50:40Z")

</div>

> [@](#):
>
> You can write your own atomic function using the other ones:
> 
> [codebox] **device** inline void myAtomicAdd(double \*address, double value) //See CUDA official forum
> 
> {
> 
> ```
> unsigned long long oldval, newval, readback;
> 
> ```
> 
> oldval = \_\_double\_as\_longlong(\*address);
> 
> ```
> newval = __double_as_longlong(__longlong_as_double(oldval) + value);
> 
> while ((readback=atomicCAS((unsigned long long *)address, oldval, newval)) != oldval)
> 
> {
> 
> oldval = readback;
> 
> newval = __double_as_longlong(__longlong_as_double(oldval) + value);
> 
> }
> 
> ```
> 
> }[/codebox]

Hi Magorath,

Thanks for the information.

How is the performance of this custom atomicadd function? Which CUDA official forum(in the code comments) is it from?

Why NVIDIA does not build this function with the the current cards?

---

<div class="post-metadata">

**Author:** ![avidday](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/avidday/32/59219_2.png) [@avidday](https://forums.developer.nvidia.com/u/avidday)\
**Post date:** [October 18, 2010, 10:47am UTC](https://forums.developer.nvidia.com/t/atomicadd-support-double-which-card/19217/15 "2010-10-18T10:47:14Z")

</div>

> [@](#):
>
> How is the performance of this custom atomicadd function?

Poor. A thread might have to make many attempts before the CAS operation is successful. I haven’t ever fully been convinced that it is formally correct either.

> [@](#):
>
> Why NVIDIA does not build this function with the the current cards?

Because it is very complex to do in hardware. Atomic operations are done in the memory controller and caches. Performing an atomic add of a 64 bit floating point number requires what is effectively a full 64 bit FPU into the memory controller. That is a lot of transistors for little real world application.

My gut feeling is that if you think you need 64 bit atomic floating point operations, you are probably using the wrong algorithmic approach.

---

<div class="post-metadata">

**Author:** ![avidday](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/avidday/32/59219_2.png) [@avidday](https://forums.developer.nvidia.com/u/avidday)\
**Post date:** [October 18, 2010, 10:47am UTC](https://forums.developer.nvidia.com/t/atomicadd-support-double-which-card/19217/16 "2010-10-18T10:47:14Z")

</div>

> [@](#):
>
> How is the performance of this custom atomicadd function?

Poor. A thread might have to make many attempts before the CAS operation is successful. I haven’t ever fully been convinced that it is formally correct either.

> [@](#):
>
> Why NVIDIA does not build this function with the the current cards?

Because it is very complex to do in hardware. Atomic operations are done in the memory controller and caches. Performing an atomic add of a 64 bit floating point number requires what is effectively a full 64 bit FPU into the memory controller. That is a lot of transistors for little real world application.

My gut feeling is that if you think you need 64 bit atomic floating point operations, you are probably using the wrong algorithmic approach.

---

<div class="post-metadata">

**Author:** ![London](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@London](https://forums.developer.nvidia.com/u/London)\
**Post date:** [October 18, 2010, 11:06am UTC](https://forums.developer.nvidia.com/t/atomicadd-support-double-which-card/19217/17 "2010-10-18T11:06:23Z")

</div>

> [@](#):
>
> Poor. A thread might have to make many attempts before the CAS operation is successful. I haven’t ever fully been convinced that it is formally correct either.
> 
> Because it is very complex to do in hardware. Atomic operations are done in the memory controller and caches. Performing an atomic add of a 64 bit floating point number requires what is effectively a full 64 bit FPU into the memory controller. That is a lot of transistors for little real world application.
> 
> My gut feeling is that if you think you need 64 bit atomic floating point operations, you are probably using the wrong algorithmic approach.

Hi avidday, thanks for your reply.

The data we marshalled from C# application are all in type double so the simulation results calculated by each GPU thread is of type double.

We need the atomicadd function so that we can add the results from each thread into the global memory. ( The recution algorithm is not suitable for our model).

What will be the best solution for the scenario? Do we need to convert all the double to float?

If Inter/Float atomic operation is our only choices now, when we do the conversion by assigment from double to float/Integer(so we can use atomic functions), will the truncation cause much problem?

---

<div class="post-metadata">

**Author:** ![London](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@London](https://forums.developer.nvidia.com/u/London)\
**Post date:** [October 18, 2010, 11:06am UTC](https://forums.developer.nvidia.com/t/atomicadd-support-double-which-card/19217/18 "2010-10-18T11:06:23Z")

</div>

> [@](#):
>
> Poor. A thread might have to make many attempts before the CAS operation is successful. I haven’t ever fully been convinced that it is formally correct either.
> 
> Because it is very complex to do in hardware. Atomic operations are done in the memory controller and caches. Performing an atomic add of a 64 bit floating point number requires what is effectively a full 64 bit FPU into the memory controller. That is a lot of transistors for little real world application.
> 
> My gut feeling is that if you think you need 64 bit atomic floating point operations, you are probably using the wrong algorithmic approach.

Hi avidday, thanks for your reply.

The data we marshalled from C# application are all in type double so the simulation results calculated by each GPU thread is of type double.

We need the atomicadd function so that we can add the results from each thread into the global memory. ( The recution algorithm is not suitable for our model).

What will be the best solution for the scenario? Do we need to convert all the double to float?

If Inter/Float atomic operation is our only choices now, when we do the conversion by assigment from double to float/Integer(so we can use atomic functions), will the truncation cause much problem?

---

<div class="post-metadata">

**Author:** ![avidday](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/avidday/32/59219_2.png) [@avidday](https://forums.developer.nvidia.com/u/avidday)\
**Post date:** [October 18, 2010, 11:28am UTC](https://forums.developer.nvidia.com/t/atomicadd-support-double-which-card/19217/19 "2010-10-18T11:28:46Z")

</div>

> [@](#):
>
> We need the atomicadd function so that we can add the results from each thread into the global memory. ( The recution algorithm is not suitable for our model).

I am willing to be that you can (and probably should) be using a parallel reduction or prefix sum for this. You might believe that you need atomic operations, but my experience with what I expect are very comparable simulation applications tells me it is almost never the case.

---

<div class="post-metadata">

**Author:** ![avidday](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/avidday/32/59219_2.png) [@avidday](https://forums.developer.nvidia.com/u/avidday)\
**Post date:** [October 18, 2010, 11:28am UTC](https://forums.developer.nvidia.com/t/atomicadd-support-double-which-card/19217/20 "2010-10-18T11:28:46Z")

</div>

> [@](#):
>
> We need the atomicadd function so that we can add the results from each thread into the global memory. ( The recution algorithm is not suitable for our model).

I am willing to be that you can (and probably should) be using a parallel reduction or prefix sum for this. You might believe that you need atomic operations, but my experience with what I expect are very comparable simulation applications tells me it is almost never the case.

[Next page](https://forums.developer.nvidia.com/t/atomicadd-support-double-which-card/19217.md?page=2)
