Hi all!
We run an operational weather forecasting model on a Cray XT4 (catamount target, crosscompiled on a linux host) and we’ve been getting rather strange segmentation faults with our code lately. Here one example…
----- DEBUG: PCB, CONTEXT, STACK TRACE ---------------------
PROCESSOR [ 0]
log_nid = 276 phys_nid = 0x198 host_id = 8 host_pid = 1948
group_id = 31329 num_procs = 804 rank = 276 local_pid = 3
base_node_index = 0 last_node_index = 401
text_base = 0x00000000200000 text_len = 0x00000001000000
data_base = 0x00000001200000 data_len = 0x00000008a00000
stack_base = 0x0000007ec00000 stack_len = 0x00000001000000
heap_base = 0x00000009e00000 heap_len = 0x00000032600000
ss = 0x000000000000001f fs = 000000000000000000 gs = 0x0000000000000017
rip = 0x00000000003d3547
rdi = 0x000000007ffffe35 rsi = 0x00000000000001cb rbp = 0x000000007fffffff
rsp = 0x000000007fbf7bc0 rbx = 0x0000000080000002 rdx = 0x0000000000000005
rcx = 0x0000000000000071 rax = 0x000000000000001b cs = 0x000000000000001f
R8 = 0x000000007ffffcbf R9 = 0x00000000000001e7 R10 = 0x000000000fe80210
R11 = 0x0000000080000000 R12 = 0x0000000000000005 R13 = 0x000000007ffffe8a
R14 = 0x0000000009490a1c R15 = 0x000000000fe80210
rflg = 0x0000000000010a12 prev_sp = 0x000000007fbf7bc0
error_code = 4
SIGNAL #[11][Segmentation fault] fault_address = 0x000000040fe7f660
Stack Trace: ------------------------------
#0 0x00000000003d3547 in numeric_utilities_interpol_sl_tricubic_()
… the code runs more than 10 times a day and usually without such problem. The routine where the code bugs is a rather complex tricubic interpolation…
wx = - w(i,j,k,1)
wy = - w(i,j,k,2)
wz = - w(i,j,k,3)
ap1z = ap1(wz)
a0z = a0 (wz)
am1z = am1(wz)
am2z = am2(wz)
! tricubic interpolation of the transported field
s_new(i,j,k) = &
& ap1(wx) * ( ap1(wy) * ( ap1z * s_old(ip1, jp1, kp1 ) &
& + a0z * s_old(ip1, jp1, k0 ) &
& + am1z * s_old(ip1, jp1, km1 ) &
& + am2z * s_old(ip1, jp1, km2 ) ) &
& + a0 (wy) * ( ap1z * s_old(ip1, j0 , kp1 ) &
& + a0z * s_old(ip1, j0 , k0 ) &
& + am1z * s_old(ip1, j0 , km1 ) &
& + am2z * s_old(ip1, j0 , km2 ) ) &
& + am1(wy) * ( ap1z * s_old(ip1, jm1, kp1 ) &
& + a0z * s_old(ip1, jm1, k0 ) &
& + am1z * s_old(ip1, jm1, km1 ) &
& + am2z * s_old(ip1, jm1, km2 ) ) &
& + am2(wy) * ( ap1z * s_old(ip1, jm2, kp1 ) &
& + a0z * s_old(ip1, jm2, k0 ) &
& + am1z * s_old(ip1, jm2, km1 ) &
& + am2z * s_old(ip1, jm2, km2 ) ) )
s_new(i,j,k) = s_new(i,j,k) + &
& a0 (wx) * ( ap1(wy) * ( ap1z * s_old(i0 , jp1, kp1 ) &
& + a0z * s_old(i0 , jp1, k0 ) &
& + am1z * s_old(i0 , jp1, km1 ) &
& + am2z * s_old(i0 , jp1, km2 ) ) &
& + a0 (wy) * ( ap1z * s_old(i0 , j0 , kp1 ) &
& + a0z * s_old(i0 , j0 , k0 ) &
& + am1z * s_old(i0 , j0 , km1 ) &
& + am2z * s_old(i0 , j0 , km2 ) ) &
& + am1(wy) * ( ap1z * s_old(i0 , jm1, kp1 ) &
& + a0z * s_old(i0 , jm1, k0 ) &
& + am1z * s_old(i0 , jm1, km1 ) &
& + am2z * s_old(i0 , jm1, km2 ) ) &
& + am2(wy) * ( ap1z * s_old(i0 , jm2, kp1 ) &
& + a0z * s_old(i0 , jm2, k0 ) &
& + am1z * s_old(i0 , jm2, km1 ) &
& + am2z * s_old(i0 , jm2, km2 ) ) )
s_new(i,j,k) = s_new(i,j,k) + &
& am1(wx) * ( ap1(wy) * ( ap1z * s_old(im1, jp1, kp1 ) &
& + a0z * s_old(im1, jp1, k0 ) &
& + am1z * s_old(im1, jp1, km1 ) &
& + am2z * s_old(im1, jp1, km2 ) ) &
& + a0 (wy) * ( ap1z * s_old(im1, j0 , kp1 ) &
& + a0z * s_old(im1, j0 , k0 ) &
& + am1z * s_old(im1, j0 , km1 ) &
& + am2z * s_old(im1, j0 , km2 ) ) &
& + am1(wy) * ( ap1z * s_old(im1, jm1, kp1 ) &
& + a0z * s_old(im1, jm1, k0 ) &
& + am1z * s_old(im1, jm1, km1 ) &
& + am2z * s_old(im1, jm1, km2 ) ) &
& + am2(wy) * ( ap1z * s_old(im1, jm2, kp1 ) &
& + a0z * s_old(im1, jm2, k0 ) &
& + am1z * s_old(im1, jm2, km1 ) &
& + am2z * s_old(im1, jm2, km2 ) ) )
s_new(i,j,k) = s_new(i,j,k) + &
& am2(wx) * ( ap1(wy) * ( ap1z * s_old(im2, jp1, kp1 ) &
& + a0z * s_old(im2, jp1, k0 ) &
& + am1z * s_old(im2, jp1, km1 ) &
& + am2z * s_old(im2, jp1, km2 ) ) &
& + a0 (wy) * ( ap1z * s_old(im2, j0 , kp1 ) &
& + a0z * s_old(im2, j0 , k0 ) &
& + am1z * s_old(im2, j0 , km1 ) &
& + am2z * s_old(im2, j0 , km2 ) ) &
& + am1(wy) * ( ap1z * s_old(im2, jm1, kp1 ) &
& + a0z * s_old(im2, jm1, k0 ) &
& + am1z * s_old(im2, jm1, km1 ) &
& + am2z * s_old(im2, jm1, km2 ) ) &
& + am2(wy) * ( ap1z * s_old(im2, jm2, kp1 ) &
& + a0z * s_old(im2, jm2, k0 ) &
& + am1z * s_old(im2, jm2, km1 ) &
& + am2z * s_old(im2, jm2, km2 ) ) )
! Tendency ds/dt:
! sten(i,j,k) = ( s_new(i,j,k) - s_old(i,j,k) ) / 2.0 / dt
… since the code is rather highly optimized…
FTNOPTS = -Mpreprocess -O3 -Mvect=sse -Mvect=noassoc
-Kieee -Mscalarsse -Mcache_align
-Mflushz -Mlre -Mprefetch -Mbyteswapio -Mipa=fast
-I/opt/xt-mpt/default/mpich2-64/P2/include
…it is not possible to located the exact line of the error. Also, since the code takes 30 minutes to execute on ~800 processors, running it without optimization and debugging options in a debugger is not really a great option. Strangely, the segfault only occurs sporadically and not reproducibly. I’ve tried playing around with stack sizes (in the yod call) but I haven’t really got enough experience to do this knowledgeably and my tries were without success.
I would be glad for any suggestions in how to go about this problem. If you need any additional information, don’t hesitate to ask.
Thanks and kind regards from sunny Switzerland,
Oliver
pgf90 -V
pgf90 7.0-3 64-bit target on x86-64 Linux