Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaMemcpy

Accelerated Computing HPC Compilers nvc, nvc++ and nvfortran
1

Hi,

I run a kernel on GPU and it gave the following error:

0: copyout Memcpy (host=0x2778e40, dev=0x7fdfd0001000, size=24) FAILED: 700(an illegal memory access was encountered)
/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/compilers/lib/libnvf.so(__fort_abort+0x4d) [0x7fe066ee6cfd]

Then after adding the “-lineinfo” into to Makefile, I run the code with cuda-memcheck. It gave the following error for by thread (33,0,0) to (63,0,0)

========= Invalid global read of size 8
========= at 0x00001180 in pseudo_gpu_mod_elcomp_pseudo_d_
========= by thread (33,0,0) in block (365,0,0)
========= Address 0x7f078d9f7b08 is out of bounds
========= Device Frame:pseudo_gpu_mod_elcomp_pseudo_d_ (pseudo_gpu_mod_elcomp_pseudo_d_ : 0x1180)
========= Saved host backtrace up to driver entry point at kernel launch time
========= Host Frame:/usr/lib64/gcc/x86_64-suse-linux/7/…/…/…/…/lib64/libcuda.so.1 [0x20d6ea]
========= Host Frame:/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4/lib64/libcudart.so.11.0 [0x1405c]
========= Host Frame:/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4/lib64/libcudart.so.11.0 (cudaLaunchKernel + 0x1d8) [0x67e88]
========= Host Frame:/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/compilers/lib/libcudafor.so (__pgiLaunchKernel + 0x1a6) [0x11c4c]
========= Host Frame:./qage [0x16083]
========= Host Frame:./qage [0x73ef0]
========= Host Frame:./qage [0xdb232]
========= Host Frame:./qage [0x406ad]
========= Host Frame:./qage [0x54f3]

After using export PGI_TERM=‘trace’ and addr2line -e qage 0x16083 on the terminal, it gave “??:0”. I could not find what the problem is. Do you have any recommendation to debug this problem? How can I solve this issue?

Thank you,
Yunus

2

Appears to be an error when accessing a module variable.

Note that “PGI_TERM” is host only so wont give a backtrace from the GPU.

Do you have any recommendation to debug this problem? How can I solve this issue?

I don’t have enough information to help solve the problem, but you can compile with “-g” and then run the program though the CUDA debugger, cuda-gdb, to see it it offers more information.

-Mat

1 Like