# No need to check cudaThreadSynchronize() in release mode?

**URL:** <https://forums.developer.nvidia.com/t/no-need-to-check-cudathreadsynchronize-in-release-mode/8957>\
**Category:** CUDA Programming and Performance\
**Created:** [April 17, 2009, 7:26am UTC](https://forums.developer.nvidia.com/t/no-need-to-check-cudathreadsynchronize-in-release-mode/8957 "2009-04-17T07:26:10Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Manjunath\_Gudisi](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@Manjunath\_Gudisi](https://forums.developer.nvidia.com/u/Manjunath_Gudisi)\
**Post date:** [April 17, 2009, 7:26am UTC](https://forums.developer.nvidia.com/t/no-need-to-check-cudathreadsynchronize-in-release-mode/8957/1 "2009-04-17T07:26:10Z")

</div>

Hello,

After evry global function, we should call CUT\_CHECK\_ERROR(errorMessage) macro.  
This macro calls 2 functions cudaGetLastError() and cudaThreadSynchronize().

It is defined in Cutil.h like this,

#ifdef \_DEBUG

# define CUT\_CHECK\_ERROR(errorMessage) do { \

cudaError\_t err = cudaGetLastError();   
if( cudaSuccess != err) {   
fprintf(stderr, “Cuda error: %s in file ‘%s’ in line %i : %s.\n”,   
errorMessage, **FILE** , **LINE** , cudaGetErrorString( err) );   
exit(EXIT\_FAILURE);   
}   
err = cudaThreadSynchronize();   
if( cudaSuccess != err) {   
fprintf(stderr, “Cuda error: %s in file ‘%s’ in line %i : %s.\n”,   
errorMessage, **FILE** , **LINE** , cudaGetErrorString( err) );   
exit(EXIT\_FAILURE);   
} } while (0)

#else

# define CUT\_CHECK\_ERROR(errorMessage)

#endif

This means, in release mode this macro just do nothing.

So, No need to synchronize the threads in Release mode? or  
GPU will do it automatically?

---

<div class="post-metadata">

**Author:** ![navier-stokes](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/navier-stokes/32/9814_2.png) [@navier-stokes](https://forums.developer.nvidia.com/u/navier-stokes)\
**Post date:** [April 17, 2009, 8:25am UTC](https://forums.developer.nvidia.com/t/no-need-to-check-cudathreadsynchronize-in-release-mode/8957/2 "2009-04-17T08:25:17Z")

</div>

> [@](#):
>
> So, No need to synchronize the threads in Release mode? or
> 
> GPU will do it automatically?

No, synchronization is necessary when using asynchronous operations using streams. See programming guide section 4.5.2.5

---

<div class="post-metadata">

**Author:** ![MisterAnderson42](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@MisterAnderson42](https://forums.developer.nvidia.com/u/MisterAnderson42)\
**Post date:** [April 17, 2009, 11:49am UTC](https://forums.developer.nvidia.com/t/no-need-to-check-cudathreadsynchronize-in-release-mode/8957/3 "2009-04-17T11:49:20Z")

</div>

We can be a bit more general: No, no cudaThreadSynchronize() call is EVER needed to ensure the correctness of your program (as long as you aren’t using the zero-copy feature of CUDA 2.2, or Async memcpy operations, I forgot about that one). Any cudaMemcpy/etc… call will either be queued and executed in order or implicitly synchronize with the GPU.

The only reasons to ever call cudaThreadSynchronize() are

1. Make sure the GPU has finished executing so you can measure a wall-clock time for benchmarking reasons
2. Check error codes after kernel calls
3. In CUDA 2.2: ensure that a kernel has finished so you can read values written to host mapped memory, though events are probably more efficient for that
4. When using memcpy \*Async methods copying to the host

And I would highly recommend not using CUT\_CHECK\_ERROR. There are often many times when you might want to check for error conditions even in a release build based on an option flag.

---

<div class="post-metadata">

**Author:** ![MisterAnderson42](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@MisterAnderson42](https://forums.developer.nvidia.com/u/MisterAnderson42)\
**Post date:** [April 17, 2009, 12:40pm UTC](https://forums.developer.nvidia.com/t/no-need-to-check-cudathreadsynchronize-in-release-mode/8957/4 "2009-04-17T12:40:53Z")

</div>

I guess that sounds really bad, huh. I say that cudaThreadSynchronize() is never needed and then give a huge list of exceptions… oh well. I was just trying to get the point across that in the vast majority of CUDA applications, cudaThreadSynchronize() is not needed to guarantee correctness. The API/driver will automatically sync for you in almost all cases where it is needed.

---

<div class="post-metadata">

**Author:** ![Noel\_Lopes](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@Noel\_Lopes](https://forums.developer.nvidia.com/u/Noel_Lopes)\
**Post date:** [April 17, 2009, 9:21pm UTC](https://forums.developer.nvidia.com/t/no-need-to-check-cudathreadsynchronize-in-release-mode/8957/5 "2009-04-17T21:21:16Z")

</div>

Hello,

I’m a newbie, so let me ask a question:

What if you call a kernel1 which calculates some outputs that are needed as inputs to kernel2. Am I just wasting my time, calling cudaThreadSynchronize() before calling kernel2?

Thanks

---

<div class="post-metadata">

**Author:** ![tmurray](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@tmurray](https://forums.developer.nvidia.com/u/tmurray)\
**Post date:** [April 17, 2009, 9:23pm UTC](https://forums.developer.nvidia.com/t/no-need-to-check-cudathreadsynchronize-in-release-mode/8957/6 "2009-04-17T21:23:44Z")

</div>

> [@](#):
>
> Hello,
> 
> I’m a newbie, so let me ask a question:
> 
> What if you call a kernel1 which calculates some outputs that are needed as inputs to kernel2. Am I just wasting my time, calling cudaThreadSynchronize() before calling kernel2?
> 
> Thanks

Yes. Because only one kernel can run at a time, there’s already an implicit synchronization barrier between kernel1 and kernel2.

---

<div class="post-metadata">

**Author:** ![vivekv80](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@vivekv80](https://forums.developer.nvidia.com/u/vivekv80)\
**Post date:** [April 20, 2009, 3:23pm UTC](https://forums.developer.nvidia.com/t/no-need-to-check-cudathreadsynchronize-in-release-mode/8957/7 "2009-04-20T15:23:30Z")

</div>

Does that also mean you implictly call \_\_syncthreads() inside your kernel only when you are using shared memory? Other wise Cuda sets up the sync barrier for kernel execution? Is that correct?  
(Newbie asking questions)

---

<div class="post-metadata">

**Author:** ![MisterAnderson42](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@MisterAnderson42](https://forums.developer.nvidia.com/u/MisterAnderson42)\
**Post date:** [April 20, 2009, 5:11pm UTC](https://forums.developer.nvidia.com/t/no-need-to-check-cudathreadsynchronize-in-release-mode/8957/8 "2009-04-20T17:11:35Z")

</div>

> [@](#):
>
> Does that also mean you implictly call \_\_syncthreads() inside your kernel only when you are using shared memory? Other wise Cuda sets up the sync barrier for kernel execution? Is that correct?
> 
> (Newbie asking questions)

\_\_syncthreads() and cudaThreadSynchronize() are completely different things.

You never need to call \_\_syncthreads() except to prevent race conditions when writing/reading shared memory.

---

<div class="post-metadata">

**Author:** ![Manjunath\_Gudisi](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@Manjunath\_Gudisi](https://forums.developer.nvidia.com/u/Manjunath_Gudisi)\
**Post date:** [April 21, 2009, 7:48am UTC](https://forums.developer.nvidia.com/t/no-need-to-check-cudathreadsynchronize-in-release-mode/8957/9 "2009-04-21T07:48:09Z")

</div>

> [@](#):
>
> The only reasons to ever call cudaThreadSynchronize() are
> 
> 1. Make sure the GPU has finished executing so you can measure a wall-clock time for benchmarking reasons

is cudaThreadSynchronize() will take 600+ms to execute??

becuase I’m executing below 2 lines code and it is taking 600+ ms

foo\<\<\<32,32\>\>\>( … );

cudaThreadSynchronize() ;

and when I execute only kernal function,

foo\<\<\<32,32\>\>\>( … );

it taking 0.1ms.

Why?

---

<div class="post-metadata">

**Author:** ![eyalhir74](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@eyalhir74](https://forums.developer.nvidia.com/u/eyalhir74)\
**Post date:** [April 21, 2009, 8:17am UTC](https://forums.developer.nvidia.com/t/no-need-to-check-cudathreadsynchronize-in-release-mode/8957/10 "2009-04-21T08:17:10Z")

</div>

> [@](#):
>
> is cudaThreadSynchronize() will take 600+ms to execute??
> 
> becuase I’m executing below 2 lines code and it is taking 600+ ms
> 
> foo\<\<\<32,32\>\>\>( … );
> 
> cudaThreadSynchronize() ;
> 
> and when I execute only kernal function,
> 
> foo\<\<\<32,32\>\>\>( … );
> 
> it taking 0.1ms.
> 
> Why?

The 600+ms is the time your kernel runs.

The kernel call is async and therefore if you don’t put cudaThreadSyncronize (or other kernel call, or other memcpy

as suggested by tmurray and MisterAnderson) you imidiatly go back to CPU host code. And if you measure this you’ll get 0.1ms.

On the other hand putting a cudaThreadSyncronize after the kernel run (or other kernel call or cudamemcpy which will implecitly call cudaThreadSync)

and ONLY then measuring the time, you’ll see the actuall time your kernel ran.

eyal