Compiling WRF3.1 on Rocks 5.1 cluster

Accelerated Computing HPC Compilers Legacy PGI Compilers
1

Greetings folks,

I’m trying to compile the latest version of WRF for use on a dual quad-core Rocks 5.1 cluster. I use mpi and gcc version 4.1.2 on a x86_64-redhat-linux using PGF90 7.2-5.

I’ve been able to successfuly compile em_real using option #1 (serial). Using option #1 I’ve also been able to execute the model successfully for a sizeable domain for a 6 week simulation (but I need to speed up the process).

I’ve been able to compile em_real using option #2 (smpar), though the results are vastly different from the results using #1 (much beyond what should be acceptable - I trust the serial results whereas I do not trust the smpar results). I’ve used 4, 6, and 7 processors when running using OpenMP, but in each case, though the speed differs (much faster), the results are not trustworthy.

I would like to compile using MPI (option #3: dmpar), but am not able to do so. The error I receive is reproduced below (the symbolic links are created but not any of the executables):

make[3]: Entering directory `/home/matt1/WRF31/WRFV3/external/RSL_LITE’
mpicc -cc=gcc -DMPI2_SUPPORT -DFSEEKO64_OK -w -O3 -DDM_PARALLEL -c
c_code.c
gcc: unrecognized option ‘-cc=gcc’
mpicc -cc=gcc -DMPI2_SUPPORT -DFSEEKO64_OK -w -O3 -DDM_PARALLEL -c
buf_for_proc.c
gcc: unrecognized option ‘-cc=gcc’
mpicc -cc=gcc -DMPI2_SUPPORT -DFSEEKO64_OK -w -O3 -DDM_PARALLEL -c
rsl_malloc.c
gcc: unrecognized option ‘-cc=gcc’
mpicc -cc=gcc -DMPI2_SUPPORT -DFSEEKO64_OK -w -O3 -DDM_PARALLEL -c
rsl_bcast.c
gcc: unrecognized option ‘-cc=gcc’
mpicc -cc=gcc -DMPI2_SUPPORT -DFSEEKO64_OK -w -O3 -DDM_PARALLEL -c
task_for_point.c
gcc: unrecognized option ‘-cc=gcc’
mpicc -cc=gcc -DMPI2_SUPPORT -DFSEEKO64_OK -w -O3 -DDM_PARALLEL -c
period.c
gcc: unrecognized option ‘-cc=gcc’
mpicc -cc=gcc -DMPI2_SUPPORT -DFSEEKO64_OK -w -O3 -DDM_PARALLEL -c
swap.c
gcc: unrecognized option ‘-cc=gcc’
mpicc -cc=gcc -DMPI2_SUPPORT -DFSEEKO64_OK -w -O3 -DDM_PARALLEL -c
cycle.c
gcc: unrecognized option ‘-cc=gcc’
mpif90 -f90=pgf90 -fastsse -Mvect=noaltcode -Msmartalloc
-Mprefetch=distance:8 -Mfprelaxed -w -Mfree -byteswapio -r4 -i4
-byteswapio -o f_pack.o -c f_pack.F90
gfortran: unrecognized option ‘-Mvect=noaltcode’
gfortran: unrecognized option ‘-Msmartalloc’
gfortran: unrecognized option ‘-Mprefetch=distance:8’
gfortran: unrecognized option ‘-Mfprelaxed’
gfortran: unrecognized option ‘-Mfree’
gfortran: unrecognized option ‘-byteswapio’
gfortran: unrecognized option ‘-r4’
gfortran: unrecognized option ‘-byteswapio’
cc1: error: unrecognized command line option “-i4”
cc1: error: unrecognized command line option “-f90=pgf90”
cc1: error: unrecognized command line option “-fastsse”
make[3]: [f_pack.o] Error 1 (ignored)
/lib/cpp -C -P -I. -DEM_CORE=1 -DNMM_CORE=0 -DNMM_MAX_DIM=2600
-DCOAMPS_CORE=0 -DDA_CORE=0 -DEXP_CORE=0 -DIWORDSIZE=4 -DDWORDSIZE=8
-DRWORDSIZE=4 -DLWORDSIZE=4 -DNONSTANDARD_SYSTEM_SUBR -DDM_PARALLEL
-DNETCDF -DGRIB1 -DINTIO -DLIMIT_ARGS -DCONFIG_BUF_LEN=32768
-DMAX_DOMAINS_F=21 -DNMM_NEST=0 -traditional f_xpose.F90 > f_xpose.f
mpif90 -f90=pgf90 -fastsse -Mvect=noaltcode -Msmartalloc
-Mprefetch=distance:8 -Mfprelaxed -w -Mfree -byteswapio -r4 -i4
-byteswapio -o f_xpose.o -c f_xpose.f
gfortran: unrecognized option ‘-Mvect=noaltcode’
gfortran: unrecognized option ‘-Msmartalloc’
gfortran: unrecognized option ‘-Mprefetch=distance:8’
gfortran: unrecognized option ‘-Mfprelaxed’
gfortran: unrecognized option ‘-Mfree’
gfortran: unrecognized option ‘-byteswapio’
gfortran: unrecognized option ‘-r4’
gfortran: unrecognized option ‘-byteswapio’
f951: error: unrecognized command line option “-f90=pgf90”
f951: error: unrecognized command line option “-fastsse”
make[3]: [f_xpose.o] Error 1 (ignored)

And on and on … I’m not using gfortran, but for some reason the redirect of the error message spews this out.

I’m pasting the configure.wrf below. Any help would be greatly
appreciated.

Architecture specific settings

To enable over-decompostion with dmpar and dm+sm compiles,

add -DALLOW_OVERDECOMP to ARCH_LOCAL, below. (not recommended)

Settings for Linux x86_64, PGI compiler with gcc (dmpar)

DMPARALLEL = 1
OMPCPP = # -D_OPENMP
OMP = # -mp -Minfo=mp
SFC = pgf90
SCC = gcc
DM_FC = mpif90 -f90=$(SFC)
DM_CC = mpicc -cc=$(SCC) -DMPI2_SUPPORT
FC = $(DM_FC)
CC = $(DM_CC) -DFSEEKO64_OK
LD = $(FC)
RWORDSIZE = $(NATIVE_RWORDSIZE)
PROMOTION = -r$(RWORDSIZE) -i4
ARCH_LOCAL = -DNONSTANDARD_SYSTEM_SUBR
CFLAGS_LOCAL = -w -O3
LDFLAGS_LOCAL =
CPLUSPLUSLIB =
ESMF_LDFLAG = $(CPLUSPLUSLIB)
FCOPTIM = -fastsse -Mvect=noaltcode -Msmartalloc -Mprefetch=distance:8 -Mfprelaxed # -Minfo=all =Mneginfo=all
FCREDUCEDOPT = $(FCOPTIM)
FCNOOPT = -O0
FCDEBUG = # -g $(FCNOOPT)
FORMAT_FIXED = -Mfixed
FORMAT_FREE = -Mfree
FCSUFFIX =
BYTESWAPIO = -byteswapio
FCBASEOPTS = -w $(FCDEBUG) $(FORMAT_FREE) $(BYTESWAPIO)
MODULE_SRCH_FLAG = -module $(WRF_SRC_ROOT_DIR)/main
TRADFLAG = -traditional
CPP = /lib/cpp -C -P
AR = ar
ARFLAGS = ru
M4 = m4 -B 14000
RANLIB = ranlib
CC_TOOLS = $(SCC)

Would greatly appreciate any assistance.

2

Hi Matei,

‘mpif90’ is just a wrapper script and your’s appears to have been configured to use gfortran. Some MPI implementation allow you to switch compilers via the “-f90” or “-cc” flags, but others does not.

To fix, you’ll need to use a MPI installation that has been configured to use PGI compilers. We do ship a pre-build MPICH with out products which you can use or you can build one yourself. Note we have a few ‘Tips and Technique’ page on building OpenMPI and MPICH that might be helpful. (See: Porting and Tuning Guides | PGI)

  • Mat
3

I am having the same exact problem. I have OpenMPI 1.25 with PGI compilers on a cluster using RHEL Enterprise 5 with Infiniband interconnect. WRFV2.2 compiles correctly and runs like a dream, but using the same compiling setup for WRFV3 produces errors related to gfortran and unrecognized options from mpif90 that I did not see with the WRFV2.2 compile.

4

Hi cr09,

Unfortunately, I don’t know what would account for this change in behavior in WRF’s build process. Try posting the question on WRF’s user forum . Hopefully someone there would have a better answer.

  • Mat
5

Thanks for the reply Mat. I’ve posted for help on the WRF Forum and with wrfhelp; let me be more specific on my build and the error I receive:

I am attempting to compile WRF Version 3.1 on a Linux system running Red Hat Enterprise Linux 5 using PGI 8.0.6, OpenMPI, and netCDF 3.6.2. The system currently employs multiple copies of WRF Version 2.2 throughout and we never had major issues compiling.

However, using the dmpar option 3 in ./configure in WRFV3.1 has produced nothing but problems. I diligently compared the correct architecture settings between WRFV2.2 and WRF3.1 to ensure consistency in key compiler-specific settings such as FCOPTIM, FCBASEOPTS, etc. The compiler’s first complaint was the “-f90=$(SFC)” and “-cc=$(SCC)” calls on DM_FC and DM_CC. I removed these flags and errors related to those tags went away and quite a bit more code started to compile…

Commands to compile using pgf90, gcc, and mpicc appear to be ok, but the first attempt at compiling using mpif90 yield the following error:

mpif90 -fastsse -w -byteswapio -Mfree -r4 -i4 -byteswapio -o f_pack.o -c f_pack.F90

gfortran: unrecognized option ‘-byteswapio’
gfortran: unrecognized option ‘-Mfree’
gfortran: unrecognized option ‘-r4’
gfortran: unrecognized option ‘-byteswapio’
cc1: error: unrecognized command line option “-i4”
cc1: error: unrecognized command line option “-fastsse”
make[3]: [f_pack.o] Error 1 (ignored)

These errors appear for all attempts at compiling using mpif90. I am unsure why it the program is calling gfortran, as the mpif90 wrapper script should call pgf90.

6

Hi,

My guess is that wrf configure to use MPICH2, which support mpif90 -f90=…

First, I would check if mpif90 use pgf90 for a compiler by typing:

%mpif90 -v x.f

If you don’t see it uses pgf* compiler, then your mpif90 is not configured to use PGI compiler.

If it uses PGI compiler, then I would remove -f90= in configure.wrf.

You can also try to check if your mpif90 support option -f90 by:
% mpif90 -f90=pgf90

If it gives you an error of unknown switch, then you should remove -f90 from your configure.wrf.

If your mpif90 does not support -f90 option or it is not configured with PGI compilers, You will need to build your own OpenMPI with pgf90.

Hongyon

7

Thanks for the response Hongyon. This source of the error was related; for whatever reason, the mpif90 was pointing to gfortran despite being told to point to pgf90. I have redirected it with clean installs of OpenMPI and PGI and appear to have gotten around this compiling error (though others are likely awaiting me!)

Thanks!

8

I’ve gotten a good compile by removing the -fc= and -cc= flags in configure.wrf for DM_FC and DM_CC, and adding -DMPI2_SUPPORT to DM_CC.

9

I’m also using openmpi for this and there appear several issues with openmpi as well as with pgi. I’m going to spend some quality time editing (or creating) a section of the Architecture default to support a pure PGI compile. With luck this will work for getting things running, and if so, then NCAR will accept it as an addition to the code.

That said, editing the linux/pgi section strikes me as the best way to get things right.

gerry