Memory corruption?

Hello,

after a lot work my code is working… or almost.

I wrote a program that run a few thousands of iterations of runge kutta method (for ODE systems) and print in a text file what got.

Well, when i set a low number of iterations it runs well ( low number of iterations = 14400) . But, when it get bigger (144000 iterations) i have a lot of trouble.

When i make it runs on full load ( 1e13 iterations) i get a lot of “0” and random numbers on exit data. If i try to run it on emulation mode i get this error:

*** glibc detected *** ./tr-raios-emu: double free or corruption (!prev): 0x0000000000ce8550 ***

======= Backtrace: =========

/lib/libc.so.6[0x7f568ae3ddd6]

/lib/libc.so.6(cfree+0x6c)[0x7f568ae4274c]

/usr/local/cuda/lib64/libcudart.so.2[0x7f568b9035fb]

/usr/local/cuda/lib64/libcudart.so.2[0x7f568b90c567]

/usr/local/cuda/lib64/libcudart.so.2[0x7f568b9117a0]

/lib/libc.so.6(__cxa_finalize+0xa0)[0x7f568ae00fe0]

/usr/local/cuda/lib64/libcudart.so.2[0x7f568b8e7e16]

======= Memory map: ========

00400000-00415000 r-xp 00000000 08:04 944997							 /home/pedro/Dropbox/Unicamp/Iniciação/Processamento em GPUs/Códigos/Fonte/Traçamento de Raios/Multicore/tr-raios-emu

00615000-00616000 r--p 00015000 08:04 944997							 /home/pedro/Dropbox/Unicamp/Iniciação/Processamento em GPUs/Códigos/Fonte/Traçamento de Raios/Multicore/tr-raios-emu

00616000-00617000 rw-p 00016000 08:04 944997							 /home/pedro/Dropbox/Unicamp/Iniciação/Processamento em GPUs/Códigos/Fonte/Traçamento de Raios/Multicore/tr-raios-emu

00ce3000-00d04000 rw-p 00000000 00:00 0								  [heap]

7f5680000000-7f5680021000 rw-p 00000000 00:00 0 

7f5680021000-7f5684000000 ---p 00000000 00:00 0 

7f56851d6000-7f56851d7000 ---p 00000000 00:00 0 

7f56851d7000-7f5685217000 rw-p 00000000 00:00 0 

7f5685217000-7f5685218000 ---p 00000000 00:00 0 

7f5685218000-7f5685258000 rw-p 00000000 00:00 0 

7f5685258000-7f5685259000 ---p 00000000 00:00 0 

7f5685259000-7f5685299000 rw-p 00000000 00:00 0 

7f5685299000-7f568529a000 ---p 00000000 00:00 0 

7f568529a000-7f56852da000 rw-p 00000000 00:00 0 

7f56852da000-7f56852db000 ---p 00000000 00:00 0 

7f56852db000-7f568531b000 rw-p 00000000 00:00 0 

7f568531b000-7f568531c000 ---p 00000000 00:00 0 

7f568531c000-7f568535c000 rw-p 00000000 00:00 0 

7f568535c000-7f568535d000 ---p 00000000 00:00 0 

7f568535d000-7f568539d000 rw-p 00000000 00:00 0 

7f568539d000-7f568539e000 ---p 00000000 00:00 0 

7f568539e000-7f56853de000 rw-p 00000000 00:00 0 

7f56853de000-7f56853df000 ---p 00000000 00:00 0 

7f56853df000-7f568541f000 rw-p 00000000 00:00 0 

7f568541f000-7f5685420000 ---p 00000000 00:00 0 

7f5685420000-7f5685460000 rw-p 00000000 00:00 0 

7f5685460000-7f5685461000 ---p 00000000 00:00 0 

7f5685461000-7f56854a1000 rw-p 00000000 00:00 0 

7f5687ac7000-7f5687add000 r-xp 00000000 08:03 531611					 /lib/libz.so.1.2.3.3

7f5687add000-7f5687cdc000 ---p 00016000 08:03 531611					 /lib/libz.so.1.2.3.3

7f5687cdc000-7f5687cdd000 r--p 00015000 08:03 531611					 /lib/libz.so.1.2.3.3

7f5687cdd000-7f5687cde000 rw-p 00016000 08:03 531611					 /lib/libz.so.1.2.3.3

7f5687cde000-7f5688017000 r-xp 00000000 08:03 852066					 /usr/lib/libcuda.so.190.53

7f5688017000-7f5688116000 ---p 00339000 08:03 852066					 /usr/lib/libcuda.so.190.53

7f5688116000-7f5688152000 rw-p 00338000 08:03 852066					 /usr/lib/libcuda.so.190.53

7f5688152000-7f568a7a0000 rw-p 00000000 00:00 0 

7f568a7a0000-7f568a7a7000 r-xp 00000000 08:03 533851					 /lib/librt-2.10.1.so

7f568a7a7000-7f568a9a6000 ---p 00007000 08:03 533851					 /lib/librt-2.10.1.so

7f568a9a6000-7f568a9a7000 r--p 00006000 08:03 533851					 /lib/librt-2.10.1.so

7f568a9a7000-7f568a9a8000 rw-p 00007000 08:03 533851					 /lib/librt-2.10.1.so

7f568a9a8000-7f568a9bf000 r-xp 00000000 08:03 531939					 /lib/libpthread-2.10.1.so

7f568a9bf000-7f568abbe000 ---p 00017000 08:03 531939					 /lib/libpthread-2.10.1.so

7f568abbe000-7f568abbf000 r--p 00016000 08:03 531939					 /lib/libpthread-2.10.1.so

7f568abbf000-7f568abc0000 rw-p 00017000 08:03 531939					 /lib/libpthread-2.10.1.so

7f568abc0000-7f568abc4000 rw-p 00000000 00:00 0 

7f568abc4000-7f568abc6000 r-xp 00000000 08:03 531927					 /lib/libdl-2.10.1.so

7f568abc6000-7f568adc6000 ---p 00002000 08:03 531927					 /lib/libdl-2.10.1.so

7f568adc6000-7f568adc7000 r--p 00002000 08:03 531927					 /lib/libdl-2.10.1.so

7f568adc7000-7f568adc8000 rw-p 00003000 08:03 531927					 /lib/libdl-2.10.1.so

7f568adc8000-7f568af2e000 r-xp 00000000 08:03 531666					 /lib/libc-2.10.1.so

7f568af2e000-7f568b12d000 ---p 00166000 08:03 531666					 /lib/libc-2.10.1.so

7f568b12d000-7f568b131000 r--p 00165000 08:03 531666					 /lib/libc-2.10.1.so

7f568b131000-7f568b132000 rw-p 00169000 08:03 531666					 /lib/libc-2.10.1.so

7f568b132000-7f568b137000 rw-p 00000000 00:00 0 

7f568b137000-7f568b14d000 r-xp 00000000 08:03 531461					 /lib/libgcc_s.so.1

7f568b14d000-7f568b34c000 ---p 00016000 08:03 531461					 /lib/libgcc_s.so.1

7f568b34c000-7f568b34d000 r--p 00015000 08:03 531461					 /lib/libgcc_s.so.1

7f568b34d000-7f568b34e000 rw-p 00016000 08:03 531461					 /lib/libgcc_s.so.1

7f568b34e000-7f568b3d0000 r-xp 00000000 08:03 531928					 /lib/libm-2.10.1.so

7f568b3d0000-7f568b5d0000 ---p 00082000 08:03 531928					 /lib/libm-2.10.1.so

7f568b5d0000-7f568b5d1000 r--p 00082000 08:03 531928					 /lib/libm-2.10.1.so

7f568b5d1000-7f568b5d2000 rw-p 00083000 08:03 531928					 /lib/libm-2.10.1.so

7f568b5d2000-7f568b6c4000 r-xp 00000000 08:03 850434					 /usr/lib/libstdc++.so.6.0.13

7f568b6c4000-7f568b8c4000 ---p 000f2000 08:03 850434					 /usr/lib/libstdc++.so.6.0.13

7f568b8c4000-7f568b8cb000 r--p 000f2000 08:03 850434					 /usr/lib/libstdc++.so.6.0.13

7f568b8cb000-7f568b8cd000 rw-p 000f9000 08:03 850434					 /usr/lib/libstdc++.so.6.0.13

7f568b8cd000-7f568b8e2000 rw-p 00000000 00:00 0 

7f568b8e2000-7f568b920000 r-xp 00000000 08:03 92948					  /usr/local/cuda/lib64/libcudart.so.2.3

7f568b920000-7f568bb20000 ---p 0003e000 08:03 92948					  /usr/local/cuda/lib64/libcudart.so.2.3

7f568bb20000-7f568bb21000 r--p 0003e000 08:03 92948					  /usr/local/cuda/lib64/libcudart.so.2.3

7f568bb21000-7f568bb22000 rw-p 0003f000 08:03 92948					  /usr/local/cuda/lib64/libcudart.so.2.3

7f568bb22000-7f568bb41000 r-xp 00000000 08:03 531663					 /lib/ld-2.10.1.soAbortado

Can someone tell me whats going on? Maybe “cudaMemcpy” isnt doing his job for so too much data?

If you are getting glibc problems like that in emulation, there is a problem with you code, because the “real” cuda runtime is untouched in emulation. Run the program inside gdb and get a backtrace, and see where it is failing, or trying running inside valgrind and see what the invalid memory access is.

I cannot run on cuda-gdb. It doesnt start any kernel (always return “Unspecific Launching Failure”.

I did say gdb, not cuda-gdb. Sound like you have an illegal/out of bounds memory access in your code. Valgrind (or gpu-ocelot) will detect the problem.