FFTW3 MPI freezes during execution

Accelerated Computing HPC Compilers Legacy PGI Compilers
1

Have used FFTW3 enabled with MPI support in past versions of the compiler. Now I am using OpenMPI 3.1.3 that is packaged with PGI 19.4. And have tried installing OpenMPI 3.1.4 with PGI 19.4 from scratch.

On all machines I have tested (which include Intel and AMD CPUs), I run into a problem where FFTW MPI calls stop indefinitely. The code does not error out, it seems like a deadlock situation.

The code worked as expected with PGI 18.10 and using the OpenMPI packaged with it (2.1.2 I believe).

Is this a known problem, and are there any recommended solutions?

2

Hi Eric,

Sorry but we’ve not had any similar reports or encountered the issues here so don’t know what could be wrong.

Do you have a reproducing example you could provide so I can investigate? Plus details on how you build FFTW and the version you’re using? If you’re unable to post, please either email myself or PGI Customer Service (trs@pgroup.com)

Thanks,
Mat

3

I have pulled the minimum lines of code needed to reproduce the error. Basically, when I run this, “Hello world” is never printed and the code never terminates.

When working with compiler version 18.10 we never had any problems with our fftw routines. After making the jump to 19.4, it just stopped working.

To build fftw, I have the PGI bin directory, and PGI/OpenMPI 3.1.3 bin directory in my path. Then I build with:

$ ./configure --prefix=/home/username/fftw CC=pgcc --enable-mpi
$ make
$ make install

Building the code with:
$ mpic++ -o fftw_bug main.cpp -I$(FFTWHOME)/include -L$(FFTWHOME)/lib -lfftw3_mpi -lfftw3

#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
#include <fftw3-mpi.h>

int main(int argc, char** argv)
{
  MPI_Init(&argc, &argv);

  const int buf_sz = 128;
  ptrdiff_t N0, N1;
  fftw_complex *data_in, *data_out;
  fftw_plan plan_fwd;

  N0 = 128;
  N1 = 128;

  data_in  = fftw_alloc_complex(buf_sz);
  data_out = fftw_alloc_complex(buf_sz);

  plan_fwd = fftw_mpi_plan_dft_2d(N0, N1, data_in, data_out, MPI_COMM_WORLD, FFTW_FORWARD,  FFTW_PATIENT);

  printf("Hello world\n");

  MPI_Finalize();
}

I do not currently have access to the 18.10 compiler to rerun, but this has been a problem since all of our systems were upgraded to 19.4

4

Hi Eric,

I don’t see a hang but rather you’re program seg faults for me since I think your data arrays are too small. Using the example from 2d MPI example (FFTW 3.3.10), I updated your program to use a call to “fftw_mpi_local_size_2d” to get the correct size. After this, the program ran to completion.

% cat test_fftw.cpp
#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
#include <fftw3-mpi.h>

int main(int argc, char** argv)
{
  MPI_Init(&argc, &argv);

  const int buf_sz = 128;
  ptrdiff_t N0, N1;
  ptrdiff_t alloc_local, local_n0, local_0_start;
  fftw_complex *data_in, *data_out;
  fftw_plan plan_fwd;

  N0 = 128;
  N1 = 128;

  alloc_local = fftw_mpi_local_size_2d(N0, N1, MPI_COMM_WORLD,
                                      &local_n0, &local_0_start);
//  data_in  = fftw_alloc_complex(buf_sz);
//  data_out = fftw_alloc_complex(buf_sz);
  data_in  = fftw_alloc_complex(alloc_local);
  data_out = fftw_alloc_complex(alloc_local);

  plan_fwd = fftw_mpi_plan_dft_2d(N0, N1, data_in, data_out, MPI_COMM_WORLD, FFTW_FORWARD,  FFTW_PATIENT);

  printf("Hello world\n");

  MPI_Finalize();
}
% mpicxx -I/proj/pgi/linux86-64-llvm/2019/fftw/fftw-3.3.8/openmpi-3.1.3/include/ -L/proj/pgi/linux86-64-llvm/2019/fftw/fftw-3.3.8/openmpi-3.1.3/lib -lfftw3_mpi -lfftw3 test_fftw.cpp -fast -w
% mpirun -np 1 a.out                                                                                                                   Hello world

Alternatively, you can use “fftw_alloc_complex(buf_sz*buf_sz);” to get the correct size.

Not sure this will fix your issue, but let’s start there. Otherwise, I’m not sure size I can’t reproduce the error.

-Mat