Question about openacc autocompare with MPI

leejsera · June 14, 2023, 8:26am

Hi Mat I Have some questions about autocompare flag when i using MPI functions.

My enviroments.
nvhpc/22.11
use Makefile flags
ACC = -fast -acc -Minfo=accel -ta=tesla:autocompare

I know how to i use autocompare function.
but when i using MPI functions it seems malfunction.

This is my code.

    subroutine mpi_subdomain_ghostcell_update(Value_sub)

        use mpi_topology!,   only : comm_1d_x1, comm_1d_x2, comm_1d_x3

        implicit none

        double precision, dimension(0:n1sub, 0:n2sub, 0:n3sub), intent(inout) :: Value_sub

        integer :: ierr
        integer :: request(4),rr(2)

        ! Update the ghostcells in the x-direction using derived datatypes and subcommunicator.
        !$acc host_data use_device(Value_sub)
        call MPI_Isend(Value_sub,1, ddtype_sendto_E  , comm_1d_x1%east_rank, 111, comm_1d_x1%mpi_comm, request(1), ierr)
        call MPI_Irecv(Value_sub,1, ddtype_recvfrom_W, comm_1d_x1%west_rank, 111, comm_1d_x1%mpi_comm, request(2), ierr)
        call MPI_Isend(Value_sub,1, ddtype_sendto_W  , comm_1d_x1%west_rank, 222, comm_1d_x1%mpi_comm, request(3), ierr)
        call MPI_Irecv(Value_sub,1, ddtype_recvfrom_E, comm_1d_x1%east_rank, 222, comm_1d_x1%mpi_comm, request(4), ierr)
        call MPI_Waitall(4, request, MPI_STATUSES_IGNORE, ierr)
        ! Update the ghostcells in the y-direction using derived datatypes and subcommunicator.
        call MPI_Isend(Value_sub,1, ddtype_sendto_N  , comm_1d_x2%east_rank, 111, comm_1d_x2%mpi_comm, request(1), ierr)
        call MPI_Irecv(Value_sub,1, ddtype_recvfrom_S, comm_1d_x2%west_rank, 111, comm_1d_x2%mpi_comm, request(2), ierr)
        call MPI_Isend(Value_sub,1, ddtype_sendto_S  , comm_1d_x2%west_rank, 222, comm_1d_x2%mpi_comm, request(3), ierr)
        call MPI_Irecv(Value_sub,1, ddtype_recvfrom_N, comm_1d_x2%east_rank, 222, comm_1d_x2%mpi_comm, request(4), ierr)
        call MPI_Waitall(4, request, MPI_STATUSES_IGNORE, ierr)
        
        ! Update the ghostcells in the z-direction using derived datatypes and subcommunicator.
        call MPI_Isend(Value_sub,1, ddtype_sendto_F  , comm_1d_x3%east_rank, 111, comm_1d_x3%mpi_comm, request(1), ierr)
        call MPI_Irecv(Value_sub,1, ddtype_recvfrom_B, comm_1d_x3%west_rank, 111, comm_1d_x3%mpi_comm, request(2), ierr)
        call MPI_Isend(Value_sub,1, ddtype_sendto_B  , comm_1d_x3%west_rank, 222, comm_1d_x3%mpi_comm, request(3), ierr)
        call MPI_Irecv(Value_sub,1, ddtype_recvfrom_F, comm_1d_x3%east_rank, 222, comm_1d_x3%mpi_comm, request(4), ierr)
        call MPI_Waitall(4, request, MPI_STATUSES_IGNORE, ierr)
        !$acc end host_data
        
    end subroutine mpi_subdomain_ghostcell_update

..................
calc 'U' array (it is 3dimension array)
..................

line 779 !$acc compare(U)

call mpi_subdomain_ghostcell_update(U)

line 781 !$acc compare(U) 
line 784 !$acc update host(U) 
line 786 !$acc compare(U)

This is terminal output

PCAST Double u in function mod_flowarray__makefld, /home/jsera.lee/apply_openACC/src/mod_flowarray.f90:781
        idx: 258 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 259 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 260 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 261 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 262 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 263 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 264 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 265 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 266 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 267 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
PCAST Double u in function mod_flowarray__makefld, /home/jsera.lee/apply_openACC/src/mod_flowarray.f90:784
        idx: 258 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 259 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 260 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 261 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 262 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 263 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 264 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 265 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 266 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 267 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
PCAST Double u in function mod_flowarray__makefld, /home/jsera.lee/apply_openACC/src/mod_flowarray.f90:786
        idx: 258 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 259 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 260 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 261 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 262 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 263 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 264 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 265 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 266 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01
        idx: 267 FAIL ABS  act: 7.03125000000000000e-01 exp: 0.00000000000000000e+00 dif: 7.03125000000000000e-01

At first i really confused about this output. but i can find some clue about this.

If you have some computation or data movement outside of the control of OpenACC that affects only host or only device memory, you have to account for that when using PCAST. For instance, an MPI data transfer to host memory would have to be copied to the device. If you use CUDA-aware MPI and do an MPI data transfer directly between two GPUs, then the host memory values are stale and must be updated. Similarly, if you have any cuBLAS or cuSolver calls on the device, then the host memory values are stale. This is likely where you have the OpenACC host_data construct, which says that something in device memory is being processed.

https://developer.nvidia.com/blog/detecting-divergence-using-pcast-to-compare-gpu-to-cpu-results/

As you can see i used “!$acc host_data use_device” because for CUDA-aware MPI or GPU-direct.
all information array is located GPU buffer so gpu to gpu communications are more efficient.
but I don’t know is it work properly. (This is not main question this topic)

Q. This is my Question.
line 779 !$acc compare(U) passed → cpu/gpu buffer have same data.
MPI function use host_data use_device.
line 781 !$acc compare(U) detected difference → I understand because gpu communication execution.
line 784 !$acc update host(U) detected difference → I understand it is difference cpu<->gpu data.
but 1. memcpy device to host 2. compare for detect 3. update host data using device data (it is my opinion)
so line 786 !$acc compare(U) detected difference → I can’t understand. line 784 already updated host data so i think it is not possible.

Could i ask what is wrong?

MatColgrove · June 15, 2023, 5:28pm

Hi leejsera,

I don’t use the compare feature myself so am not an expert here.

I’m not sure what your PCAST_COMPARE setting is, but the default is to print up to 50 differences. Here, 10 are getting printed starting at index 258, meaning only part of the array shows differences. What I’m wondering is given only the halos are getting update, this is causing some data to become stale?

Maybe try the following to ensure the data is in sync:

!$acc update device(U)
call mpi_subdomain_ghostcell_update(U)
!$acc update self(U) 
!$acc compare(U)

buffer so gpu to gpu communications are more efficient. but I don’t know is it work properly

Can you elaborate? Depending on the problem, using compare may or may not be the best way to determine if things are working as expected.

-Mat

leejsera · June 16, 2023, 4:20am

There are many things I can’t understand…

when i use at my code. “!$acc compare all”
I can’t compile this code.

NVFORTRAN-S-0034-Syntax error at or near identifier all (mod_passivescalar.f90: 673)
0 inform, 0 warnings, 1 severes, 0 fatal for rhs_ibm_ps
make: *** [Makefile:39: obj/mod_passivescalar.o] Error 2

Q1.
I find this command “!$acc compare all” below references.
Are there references wrong information? or Is there anything wrong at my settings?

github.com

NCAR/GPU_workshop/blob/workshop/08_PCAST/08_PCAST.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "746d1042-9b52-4e9f-bce4-0bcd7eb15d48",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "![NCAR UCAR Logo](../NCAR_CISL_NSF_banner.jpeg)\n",
    "# Verifying Code Correctness with PCAST\n",
    "\n",
    "By: Daniel Howard [dhoward@ucar.edu](mailto:dhoward@ucar.edu), Consulting Services Group, CISL & NCAR \n",
    "\n",
    "Date: May 19th, 2022"
   ]
  },

This file has been truncated. show original

Q2.
When i use “call acc_compare_all()”
I can compile the code. but it print information without the name.
like this.
variable name, function name is printed out (null).
so it is useless…

PCAST Double (null) in function (null), (null):0
        idx: 256 FAIL ABS  act: 0.00000000000000000e+00 exp: -nan dif: nan
PCAST Long (null) in function (null), (null):0
        idx: 0 FAIL act: 0 exp: 75424
PCAST Long (null) in function (null), (null):0
        idx: 0 FAIL act: 0 exp: 192864
PCAST Double (null) in function (null), (null):0
        idx: 0 FAIL ABS  act: 0.00000000000000000e+00 exp: 3.30000000000000000e+01 dif: 3.30000000000000000e+01
        idx: 2 FAIL ABS  act: 0.00000000000000000e+00 exp: 3.40000000000000000e+01 dif: 3.40000000000000000e+01
        idx: 4 FAIL ABS  act: 0.00000000000000000e+00 exp: 3.50000000000000000e+01 dif: 3.50000000000000000e+01
        idx: 6 FAIL ABS  act: 0.00000000000000000e+00 exp: 3.60000000000000000e+01 dif: 3.60000000000000000e+01
PCAST Double (null) in function (null), (null):0
        idx: 0 FAIL ABS  act: 0.00000000000000000e+00 exp: -1.16077616512673330e+03 dif: 1.16077616512673330e+03
        idx: 1 FAIL ABS  act: 0.00000000000000000e+00 exp: -2.32126214528074888e+03 dif: 2.32126214528074888e+03
        idx: 2 FAIL ABS  act: 0.00000000000000000e+00 exp: -2.32126214528074888e+03 dif: 2.32126214528074888e+03
        idx: 3 FAIL ABS  act: 0.00000000000000000e+00 exp: -2.32126214528074888e+03 dif: 2.32126214528074888e+03
        idx: 4 FAIL ABS  act: 0.00000000000000000e+00 exp: -2.32126214528074888e+03 dif: 2.32126214528074888e+03
        idx: 5 FAIL ABS  act: 0.00000000000000000e+00 exp: -2.32126214528074888e+03 dif: 2.32126214528074888e+03
        idx: 6 FAIL ABS  act: 0.00000000000000000e+00 exp: -2.32126214528074888e+03 dif: 2.32126214528074888e+03
        idx: 7 FAIL ABS  act: 0.00000000000000000e+00 exp: -2.32126214528074888e+03 dif: 2.32126214528074888e+03

Meanwhile when i use “!$acc compare(dpsdtr_sub )”
It works!

PCAST Double dpsdtr_sub in function rhs_ibm_ps, /home/jsera.lee/apply_openACC/src/mod_passivescalar.f90:728
        idx: 0 FAIL ABS  act: 0.00000000000000000e+00 exp: -6.48899480404105270e+03 dif: 6.48899480404105270e+03
        idx: 1 FAIL ABS  act: 0.00000000000000000e+00 exp: -3.24449740202052726e+03 dif: 3.24449740202052726e+03
        idx: 2 FAIL ABS  act: 0.00000000000000000e+00 exp: -3.24449740202052726e+03 dif: 3.24449740202052726e+03
        idx: 3 FAIL ABS  act: 0.00000000000000000e+00 exp: -3.24449740202052726e+03 dif: 3.24449740202052726e+03
        idx: 4 FAIL ABS  act: 0.00000000000000000e+00 exp: -3.24449740202052726e+03 dif: 3.24449740202052726e+03
        idx: 5 FAIL ABS  act: 0.00000000000000000e+00 exp: -3.24449740202052726e+03 dif: 3.24449740202052726e+03
        idx: 6 FAIL ABS  act: 0.00000000000000000e+00 exp: -3.24449740202052726e+03 dif: 3.24449740202052726e+03
        idx: 7 FAIL ABS  act: 0.00000000000000000e+00 exp: -3.24449740202052726e+03 dif: 3.24449740202052726e+03

Q3.
Please tell me who should ask about PCAST, autocompare.

leejsera · June 16, 2023, 4:28am

Your suggestion doesn’t work.

!$acc update device(U)
call mpi_subdomain_ghostcell_update(U)
!$acc update self(U)
!$acc compare(U)

Q1.
before MPI function, openacc copute about “U” array so new information is only exist at “gpu buffer U”, “cpu buffer U” have information but it is older values.
So I can’t understand why code’s first line “!$acc update device”.
I think if i want to sync cpu/gpu, I have to use “!$acc update host”

Q2.
At mpi functions i use “!$acc host_data use_device(Value_sub)”
So as i thinking it communicate gpu buffer to gpu buffer.
so gpu buffer’s data is new.
After MPI Function, I have to !$acc update host and then cpu/gpu are sync.
Is there anything wrong at my guess?

In detail.

acc compute region about ‘U’
MPI communication for ghost cell ‘U’
acc compute region about ‘U’

(U is already exist at gpu buffer)
So I think , and MPI communication gpu ↔ gpu will be more efficient.
like cuda-aware MPI

Some example at online, they use below process.
I think this 2 memory copy is unneccessary.

acc compute region about ‘U’
acc update host
MPI communication for ghost cell ‘U’
acc update device
acc compute region about ‘U’

!$acc update host(sigmazz,sigmayz,sigmaxz)
! sigmazz(k+1), left shift
  call MPI_SENDRECV(sigmazz(:,:,1),number_of_values,MPI_DOUBLE_PRECISION, &
         receiver_left_shift,message_tag,sigmazz(:,:,NZ_LOCAL+1), &
         number_of_values,

...

!$acc update device(sigmazz,sigmayz,sigmaxz)

MatColgrove · June 16, 2023, 4:32pm

I think the references are wrong, or for some reason we didn’t implement “acc compare all”. Though you can use the “call acc_compare_all()”.

Please tell me who should ask about PCAST, autocompare.

I’ll need to create a reproducing example and ask engineering for input. Though it won’t be until Tuesday when our office reopens after a U.S. Holiday.

before MPI function, openacc copute about “U” array so new information is only exist at “gpu buffer U”, “cpu buffer U” have information but it is older values.
So I can’t understand why code’s first line “!$acc update device”.
I think if i want to sync cpu/gpu, I have to use “!$acc update host”

The only reason to add the updates is for the compare functions to work since the MPI calls are outside of the autocompare feature’s ability to track. They aren’t needed nor desired without autocompare since the data copies would extraneous.

I’ll ask again, what is the core issue that you’re trying to solve? Depending on the problem, auto-compare may or may not be the best tool to help solve it and spending time working through these issues may not be beneficial.

leejsera · June 20, 2023, 10:56am

Q. I wonder that is it my setting problem? or this is also non-implemented?
when using call acc_compare_all() it printed (null)

The only reason to add the updates is for the compare functions to work since the MPI calls are outside of the autocompare feature’s ability to track. They aren’t needed nor desired without autocompare since the data copies would extraneous.

I understood your words.

I’ll ask again, what is the core issue that you’re trying to solve? Depending on the problem, auto-compare may or may not be the best tool to help solve it and spending time working through these issues may not be beneficial.

The problem I want to solve is comparative verification. (baseline is CPU results.)
Parallelization was applied using OpenACC to a large project.
I was using PCast AutoCompare to compare it with the CPU results.
I was able to find and modify my many mistakes using AutoCompare.
AutoCompare seems to be a very useful and powerful verification tool!

When i asked at first, I tried to gpu_mem ↔ gpu_mem MPI communication. Using !$acc host_data use_device()

However, due to the problem of AutoCompare can not notice GPU-MPI communication, now I try to 2 steps.

Verification of OpenACC Calculation Results using AutoCompare.
Change CPU-MPI communication to GPU-MPI communication.

Current my code status.

!$acc compare(U)
!$acc update host(U)

call mpi_subdomain_ghostcell_update(U)

!$acc update device(U) 
!$acc compare(U)

and no use !$acc host_data use_device() in mpi_subdomain_ghostcell_update subroutine.

It is Inefficient because of device ↔ host memory copy 2times, but verification is first!
I will first verify the calculation results, after will apply CUDA-aware MPI or GPU RDMA.

I am always grateful for the fast, accurate and friendly answer.
Thnks Mat.

MatColgrove · June 20, 2023, 7:01pm

Sounds like the best strategy. I’m not 100% sure PCAST is supposed to work with host_data, but the PCAST article implies that it does provided you add the extra updates so the memories are in sync. Michael, who lead this effort, has since retired so I can’t ask him, and Adam, the co-author, hasn’t gotten back to me yet.

It is Inefficient because of device ↔ host memory copy 2times, but verification is first!
I will first verify the calculation results, after will apply CUDA-aware MPI or GPU RDMA.

Autocompare itself is very slow given the loops get run on both the host and device. I’d only use it for verification and disable it for production or performance testing.

-Mat

system · July 4, 2023, 7:02pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
OpenACC usage inside OpenMP constructs Legacy PGI Compilers	6	3855	August 26, 2019
Benchmarking CUDA-Aware MPI Technical Blog	16	1322	August 20, 2019
Dealing with allocatable arrays with OpenACC Legacy PGI Compilers	8	1826	November 30, 2020
Request support/help for PBS with OpenMPI Legacy PGI Compilers	21	14839	August 9, 2022
MPI send + OpenACC + acc_malloc fail with NVFortran, but work with C nvc, nvc++ and nvfortran	10	64	September 6, 2024
Segfault with MPI_Send + acc_malloc Legacy PGI Compilers	3	2581	April 8, 2020
NV 24.1 Default MPI seg faulting on derived type host_data MPI calls - sometimes nvc, nvc++ and nvfortran	15	739	June 6, 2024
accelerate a single loop with mpi and gpu Legacy PGI Compilers	21	15832	July 19, 2013
matrix reduction using cuda fortran and GPU Legacy PGI Compilers	33	13511	December 21, 2012
OpenACC: Best way to parallelize nested DO loops (continued) nvc, nvc++ and nvfortran	22	1545	March 28, 2023

Question about openacc autocompare with MPI

Related topics