N-Body Benchmark Crash

Accelerated Computing CUDA CUDA Programming and Performance
1

I just started working on my own n-body simulation, and I wanted to compare it’s performance to the n-body SDK demo. I got the first few data points down (using powers of 2 sizes):

Note: These numbers are from my 8800GT with 512MB memory.

Number of Bodies,Iterations,Compute Time (ms),Interactions Per Second (in Billions),GFLOP/s

1024,100,39.849,2.631,52.628

1024,200,83.801,2.503,50.051

1024,300,120.952,2.601,52.016

1024,400,165.941,2.528,50.552

2048,100,75.599,5.548,110.961

2048,200,151.737,5.528,110.568

2048,300,224.951,5.594,111.872

2048,400,302.937,5.538,110.764

4096,100,278.180,6.031,120.621

4096,200,560.314,5.989,119.770

4096,300,843.045,5.970,119.904

4096,400,1121.440,5.984,119.683

8192,100,823.988,8.144,162.888

8192,200,1641.121,8.178,163.568

8192,300,2459.290,8.186,163.727

8192,400,3280.464,8.183,163.657

16384,100,2709.589,9.907,198.137

16384,200,5402.960,9.937,198.732

16384,300,8110.443,9.929,198.585

16384,400,10812.151,9.931,198.618

32768,100,10779.012,9.961,199.228

32768,200,21559.355,9.961,199.216

32768,300,32348.813,9.958,199.156

32768,400,43112.066,9.962,199.247

65536,100,40979.711,10.481,209.614

65536,200,81941.430,10.483,209.660

65536,300,122912.563,10.483,209.660

65536,400,163883.922,10.483,209.659

131072,100,159203.453,10.791,215.823

262144,100,636327.813,10.799,215.988

However, when I went to run the simulation on the next size (n = 262144), the program immediately crashed, I got a Vista notification saying the video driver had crashed and restarted, and the following error in the console:

C:\ProgramData\NVIDIA Corporation\NVIDIA CUDA SDK\bin\win64\Release>nbody -benchmark -n=262144

Run "nbody -benchmark -n=<numBodies>" to measure perfomance.

cutilCheckMsg cudaThreadSynchronize error: Kernel execution failed in file <bodysystemcuda.cu>, line 155 : unknown error.

Any ideas on what the problem is here? I was planning to benchmark the next few powers-of-two sizes (n = 524288, 1048576, 2097152) so I have an idea about what I should expect from my own code.

I’m running on Vista64 if that matters…

2

Try turning off the TDR timeout and running again. You might be hitting the watchdog timer, everyone’s favorite friend.

3

Are there any instructions on how to do that? I tried searching the forum, but nothing came up (maybe the search is broken? I’m sure it would have been mentioned somewhere…)

This Microsoft article I found had some information: [url=“http://www.microsoft.com/whdc/device/display/wddm_timeout.mspx”]http://www.microsoft.com/whdc/device/displ...dm_timeout.mspx[/url]

I added and set the following registry keys in HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\GraphicsDrivers (per the link):

TdrLevel = 0

The benchmark is still crashing on n=262144 though…

4

Update: It’s working now. I had to do a restart of the machine before the registry settings take effect.

I’ll run some more benchmarks tonight and tomorrow and post them above in case anyone is interested. I’ll need them for my own comparisons anyway.