Control pgfortran's optimization level on CUDA kernel

I’m debugging a CUDA fortran code.In emu mode, it works great.
But it’s failed with cuda execution, with error messages:

copyout Memcpy (host=0x7c5c60, dev=0x1e720000, size=1310720) FAILED: 4(unspecified launch failure)
Notice: I use “-Mcuda=emu -Mbounds” parameters, so do not worry about out-bound array problem.

As I/O is not allowed in cuda kernel, the handy debug is really challenge. So I decided to comment some lines to see the problem.
Question 1: Are there any debug tools for pgfortran cuda?

But another important thing for debug, is to stop the optimization of compiler. I try to use “pgfortran -O0”, it do not stop the optimization of cuda code.
Question 2: How to stop the optimization of cuda fortran code?


I have a test on the compiler parameters of :
pgfortran -Mmpi=mpich1 -Mcuda=keepgpu -Mcuda=keepbinn -Mcuda=keepptx -Mcuda=ptxinfo

I can see the following files generated for kernel module compiling kept.
(all global and device subroutines are organized in raycast_GPUkernel module)

  1. raycast_GPUkernel.001.gpu – ANSI C code file, converted from raycast_GPUkernel.F90
  2. raycast_GPUkernel.001.h – header file of file 1, two include file are also required by file 1:“cuda_runtime.h” and “pgi_cuda_runtime.h”
  3. raycast_GPUkernel.002.bin
  4. raycast_GPUkernel.002.ptx – ptx code for cuda 1.4, ASM file
  5. raycast_GPUkernel.003.bin
  6. raycast_GPUkernel.003.ptx – ptx code for cuda 2.1 ?

So for a cuda fortran code, it first convert to cuda c by pgfortran? Then compiled into ptx file using embered nvcc-like tools ?

Notice in file 4 & 6, some text are list below:

     .version 2.1   !--- for .002.ptx, it's .version 1.4 
    .target sm_20          !---                         sm_13
    // compiled with /opt/applications/pgi/linux86-64/2011/cuda/3.1/open64/lib/be


    //-----------------------------------------------------------
    // Compiling /tmp/pbs.15539.service0/pgnvdvLwd-JE4MrdP.nv4 (/tmp/pbs.15539.ss
ervice0/pgnvdLLwdVIRdSi62.nv6)
    //-----------------------------------------------------------

    //-----------------------------------------------------------
    // Options:
    //-----------------------------------------------------------
    //  Target:ptx, ISA:sm_20, Endian:little, Pointer Size:64
    //  -O3 (Optimization level)
    //  -g0 (Debug level)
    //  -m2 (Report advisories)
    //-----------------------------------------------------------

I think this “-O3” is automatically given by pgfortran, do not depending on the parameter “-O” given by user to pgfortran.

Question 3: Is there a way to allow user to compiler the raycast_GPUkernel.001.gpu by nvcc?

Thanks in advance.

nvcc raycast_GPUkernel.001.gpu -cubin -O0

In file included from /usr/local/cuda/bin/…/include/cuda_runtime.h:55,
from raycast_GPUkernel.001.gpu:1:
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:89: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:90: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:91: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:92: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:93: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:101: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:102: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:103: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:104: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:105: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:106: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:107: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:109: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:110: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:111: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:119: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:120: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:121: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:122: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:123: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:124: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:125: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:126: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:127: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:128: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:129: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:137: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:138: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:139: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:140: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:141: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:142: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:143: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:144: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:152: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:153: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:161: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:162: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:170: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:171: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:172: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:173: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:174: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:175: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:176: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:184: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:185: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:186: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:187: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:188: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:189: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:197: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:198: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:199: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:207: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:208: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:216: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:217: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:218: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:226: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:227: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:228: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:229: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:230: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:238: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:239: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:240: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:241: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:249: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:250: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:251: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:252: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:253: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:254: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:255: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:263: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:264: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:272: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:273: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:274: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:275: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:283: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:284: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:285: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:294: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:295: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:296: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:297: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/cuda_runtime_api.h:298: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/cuda_runtime_api.h:299: warning: host’ attribute directive ignored
In file included from /usr/local/cuda/bin/…/include/cuda_runtime.h:56,
from raycast_GPUkernel.001.gpu:1:
/usr/local/cuda/bin/…/include/driver_functions.h:56: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/driver_functions.h:68: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/driver_functions.h:79: warning: host' attribute directive ignored In file included from /usr/local/cuda/bin/../include/cuda_runtime.h:58, from raycast_GPUkernel.001.gpu:1: /usr/local/cuda/bin/../include/vector_functions.h:56: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:61: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/vector_functions.h:66: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:71: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/vector_functions.h:76: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:81: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/vector_functions.h:86: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:91: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/vector_functions.h:96: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:101: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/vector_functions.h:106: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:111: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/vector_functions.h:116: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:121: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/vector_functions.h:126: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:131: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/vector_functions.h:136: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:141: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/vector_functions.h:146: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:151: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/vector_functions.h:156: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:161: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/vector_functions.h:166: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:171: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/vector_functions.h:176: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:181: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/vector_functions.h:186: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:191: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/vector_functions.h:196: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:201: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/vector_functions.h:206: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:211: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/vector_functions.h:216: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:221: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/vector_functions.h:226: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:231: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/vector_functions.h:236: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:241: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/vector_functions.h:246: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:251: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/vector_functions.h:256: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:261: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/vector_functions.h:266: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:271: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/vector_functions.h:276: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:281: warning: host' attribute directive ignored /usr/local/cuda/bin/../include/vector_functions.h:286: warning: host’ attribute directive ignored
/usr/local/cuda/bin/…/include/vector_functions.h:291: warning: `host’ attribute directive ignored
In file included from raycast_GPUkernel.001.gpu:3:
raycast_GPUkernel.001.h:2: error: syntax error before string constant

Hi tlstar,

Question 1: Are there any debug tools for pgfortran cuda?

In emulation mode, you can use the PGI debugger, pgdbg. On device debugging is still in development.

Question 2: How to stop the optimization of cuda fortran code?

“-O0” will disable Fortran optimization. You are correct in that we do not adjust the optimization flags given to the NVIDIA back-end tools, however, it is unlikely that your problem is being caused by an optimization bug in the NVIDIA back end compiler. More likely the error is being caused by a programming error or in the low level CUDA C that gets generated.

Question 3: Is there a way to allow user to compiler the raycast_GPUkernel.001.gpu by nvcc?

We don’t use nvcc, only a few of NVIDIA’s back end tools. While not advisable, you can find the steps necessary to recreate the build using the verbose (-v) flag. Compile with “-Mcuda=keepgpu -v” and look for the “pgnvd” driver commands. There will be multiple pgnvd commands targeting different device compute capabilities. Copy one of these pgnvd commands to your command line and add the “-v” flag. This will show you the exact steps to create the “bin” file.

So for a cuda fortran code, it first convert to cuda c by pgfortran? Then compiled into ptx file using embered nvcc-like tools ?

Yes, we generate very low level highly optimized CUDA C and then use some of NVIDIA’s backend tools to create the device code. We’ll eventually go directly to producing ptx since this is a requirement to support debugging on the device. The process is similar to what is done on the host, but instead of the code-generator targeting assembly code, it targets very low level CUDA C.

  • Mat