Large file support on MPI?

Accelerated Computing HPC Compilers Legacy PGI Compilers
1

Hi,

Currently we have using PGI5.1-3 Server version with MPICH 1.2.6 on Athlon XP system.
Everything works fine except large file support on mpi.
I have choose -Mlfs during compiling the codes.
However, the executing always stop and error happen when output file larger than 2GB.
If only with pgf90(or 77) -Mlfs, it works fine.
Is this trouble causing by installing MPICH without -Mlfs configuration?
Currently MPICH are installed with your suggestion option (list in FAQ).
Any suggestions?

2

Hi Danish,

Although I have not tried this myself, I believe you need to reconfigure and rebuild your MPICH library using the following define flags in order to support large files. Configure your installation as normal, but instead of setting CC to pgcc set it to:

setenv CC "pgcc -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE"

Everything I’ve read says that this should do it, but please let us know how it does.

Thanks,
Mat

3

Thanks for your reply!

According your suggestion, I change the configuration option and rebuild again, the command is like below:

env CC=pgcc CFLAGS=‘-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Msignextend -tp px’ FC=pgf77 FFLAGS=‘-tp px’ CXX=pgCC CCFLAGS=‘-tp px’ F90=pgf90 F90FLAGS=‘-tp px’ MAKE=gmake ./configure --enable-debug -opt=-fast -prefix=/usr/local/mpich
gmake -f Makefile
gmake install

However, the problem still happen when the file size exceed 2GB.
The following is error message:

p1_28959: p4_error: net_recv read: probable EOF on socket: 1
p3_2338: p4_error: net_recv read: probable EOF on socket: 1
p2_4167: p4_error: net_recv read: probable EOF on socket: 1
/usr/local/mpich/bin/mpirun: line 1: 15803 File size limit exceeded(core dumped) /home/danish/run/ECS/shelf -p4pg /home/danish/run/ECS/PI15677 -p4wd /home/danish/run/ECS
Broken pipe
Broken pipe
Broken pipe

I also use -Mlfs during compilation, like
mpif90 -Mlfs -tp athlonxp -r8 -O2 ***.f90

Still not work. :(
Any ideas?

4

Hi Danish,

I did a bit more digging and per the ROMIO user’s guide in section 4.1 the correct define flags are “-D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE” not “-D_LARGEFILE_SOURCE” as I saw posted on the web. These are glibc definitions included in the 2.4 and newer kernels, so make sure your kernal supports this.

Try again using the new define flags. Although pgcc/pgCC should work, if it doesn’t try using gcc/g++ instead. There may be some GNUisms I’m unaware of. If it still doesn’t work, try contacting the ROMIO team (See: ROMIO) . Let us know how it goes!

Thanks,
Mat

5

Hi mkcolg,

I have try the new flags with PGI and read ROMIO user guide.
Seems still not working with this.
However, I found there is a large file test in ROMIO directory.
This test is not working on my system.

There are some questions came out after try this and read the manual.
Althrough ROMIO is combine in MPICH(1.2.6), I am not very sure ROMIO was installed properly with MPICH standard install procedure.

Second, ROMIO seems only work with MPICH2.
I am not very sure about this.
If so, MPICH2 need to be installed.
I found MPI-2-C++ diretory in mpich install directory, also ROMIO.
Is this means these will be installed properly?
I have check the ROMIO directory and found "romioinstal"l script in “util”.
The default install path is /usr/local/romio.
If ROMIO is install properly, seems /usr/local/romio should be there.
But I can’t find this directory in /usr/local, even in /usr/local/mpich(my mpich install prefix).
So I wondering ROMIO and MPICH2 are not installed, this cause the CFLAGS won’t take any effects.

Sorry to ask you some questions not directly relating to PGI.
And thanks for your help.