wrf2.2 with pgi7.0-5 on 64 bit linux cluster and Openmp

Hi,

I’m trying to compile wrf2.2 with pgi7.0 on a 64 bit linux cluster, using openmp (I have a dual-quad xeon).
Well i have two dual-quad xeon machines, the second step would be an hybrid openmp-mpich2 configuration.

In the standard configure options coming with wrf2.2 package I can’t find my configuration. I tried to use a configure option found on PGI website for pgi5.2 but it fails to compile, even using pgcc instead og gcc.

Anyone solved this problem?

thanks a lot,

aldo

Hi Aldo,

Unfortunately, the 5.2 configuration for WRF 2.0 won’t work with WRF 2.2 due to several changes in the WRF build process. Below is an architecture configuration for PGI and OpenMP which you can append to the file arch/configure.defaults. When you rerun “./configure”, this option will appear. While, I don’t have a hybrid OpenMP/MPI configuration, you might be able to use the following configuration and blend it into the existing PGI MPICH configuration. I have not tried it, nor know it will actually work, but it might.

Hope this helps,
Mat


###########################################################
#ARCH    PC Linux x86_64 (Opteron), PGI compiler 5.2 or higher  (OpenMP)
#        Note that for 5.1.x comment out -Mpia=fast
#
OMP             =       -mp
OMPCPP          =       -D_OPENMP
FC              =       pgf90
LD              =       pgf90 $(OMP)
CC              =       gcc
SCC             =       $(CC)
SFC             =       $(FC)
RWORDSIZE       =       $(NATIVE_RWORDSIZE)
PROMOTION       =       -r$(RWORDSIZE) -i4
CFLAGS          =       -DLANDREAD_STUB
FCOPTIM         =       -fastsse #-Mipa=fast
FCDEBUG         =       #-g
#FCBASEOPTS      =       -w -byteswapio -Ktrap=fp -Mfree $(FCDEBUG)
FCBASEOPTS      =       -w -byteswapio -Mfree $(OMP) $(FCDEBUG)
FCFLAGS         =       $(FCOPTIM) $(FCBASEOPTS)
ARCHFLAGS       =       -DIWORDSIZE=4 -DDWORDSIZE=8 -DRWORDSIZE=$(RWORDSIZE) -DLWORDSIZE=4 \
                         CONFIGURE_NETCDF_FLAG -DGRIB1 CONFIGURE_GRIB2_FLAG \
                        -DLIMIT_ARGS -DLANDREAD_STUB
# machine-specific flags needed to link in ESMF library (C++ run-time-library, etc.)
ESMF_LIB_FLAGS  =
ESMF_IO_LIB     =       ESMFIOLIB
ESMF_IO_LIB_EXT =       ESMFIOEXTLIB
INCLUDE_MODULES =       -module ../main -I../external/io_netcdf -I../external/io_int -I../external/esmf_time_f90 \
                        -I../frame -I../share -I../phys -I../inc -I../chem
PERL            =       perl
REGISTRY        =       Registry
LIB             =       CONFIGURE_NETCDF_LIB_PATH \
                        -L../external/io_grib1 -lio_grib1 \
                        CONFIGURE_GRIB2_LIB \
                        -L../external/io_grib_share -lio_grib_share \
                        ../frame/module_internal_header_util.o ../frame/pack_utils.o -L../external/esmf_time_f90 -lesmf_time
LDFLAGS         =       CONFIGURE_LDFLAGS
ENVCOMPDEFS     =       CONFIGURE_COMPILEFLAGS
CPP             =       /lib/cpp -C -P -traditional
POUND_DEF       =       -traditional $(COREDEFS) -DNONSTANDARD_SYSTEM -DCONFIG_BUF_LEN=$(CONFIG_BUF_LEN) -DMAX_DOMAINS_F=$(MAX_DOMAINS)
CPPFLAGS        =       -I$(LIBINCLUDE) -C -P $(ARCHFLAGS) $(ENVCOMPDEFS) $(OMPCPP) $(POUND_DEF)
AR              =       ar ru
M4              =       m4 -B 14000
RANLIB          =       ranlib
NETCDFPATH      =       CONFIGURE_NETCDF_PATH
CC_TOOLS        =       $(CC)

externals : wrf_ioapi_includes CONFIGURE_WRFIO_NF wrfio_grib_share wrfio_grib1 CONFIGURE_WRFIO_GRIB2 wrfio_int module_dm.F esmf_time

module_dm.F :
        ( /bin/cp module_dm_warning module_dm.F ; cat module_dm_stubs.F >> module_dm.F )

wrfio_nf :
        ( cd ../external/io_netcdf ; \
          make NETCDFPATH=CONFIGURE_NETCDF_PATH RANLIB="$(RANLIB)" CPP="$(CPP)" FC="$(FC) $(PROMOTION) $(FCFLAGS) -w" TRADFLAG="-traditional" )

wrfio_grib_share :
        ( cd ../external/io_grib_share ; \
          make CC="$(SCC)" CFLAGS="$(CFLAGS)" RM="$(RM)" RANLIB="$(RANLIB)" CPP="$(CPP)" FC="$(SFC) $(PROMOTION) -I. $(FCDEBUG) $(FCBASEOPTS) -w" TRADFLAG="-traditional" archive)

wrfio_grib1 :
        ( cd ../external/io_grib1 ; \
          make CC="$(SCC)" CFLAGS="$(CFLAGS)" RANLIB="$(RANLIB)" CPP="$(CPP)" FC="$(SFC) $(PROMOTION) -I. $(FCDEBUG) $(FCBASEOPTS) -w" TRADFLAG="-traditional" archive)

wrfio_grib2 :
        ( cd ../external/io_grib2 ; \
          make CC="$(SCC)" CFLAGS="$(CFLAGS) CONFIGURE_GRIB2_INC" RM="$(RM)" RANLIB="$(RANLIB)" CPP="$(CPP)" FC="$(SFC) $(PROMOTION) -I. $(FCDEBUG) $(FCBASEOPTS) -w" TRADFLAG="-traditional" FIXED="-Mfixed" archive)

wrfio_int :
        ( cd ../external/io_int ; \
          make CC="$(CC)" RANLIB="$(RANLIB)" CPP="$(CPP)" SFC="$(SFC) $(PROMOTION) $(FCDEBUG) $(FCBASEOPTS)" FC="$(SFC) $(PROMOTION) $(FCDEBUG) $(FCBASEOPTS) -w" \
          TRADFLAG="-traditional" all )

esmf_time :
        ( cd ../external/esmf_time_f90 ; \
          make FC="$(FC) $(PROMOTION) $(FCDEBUG) $(FCBASEOPTS)" RANLIB="$(RANLIB)" CPP="$(CPP) -I../../inc -I. $(POUND_DEF)" )

# compile these without high optimization to speed compile
solve_interface.o : solve_interface.F
shift_domain_em.o : shift_domain_em.F
module_io_mm5.o : module_io_mm5.F
module_si_io.o : module_si_io.F
module_io_wrf.o : module_io_wrf.F
module_domain.o : module_domain.F
module_start.o : module_start.F
module_initialize.o : module_initialize.F
module_initialize_b_wave.o : module_initialize_b_wave.F
module_initialize_hill2d_x.o : module_initialize_hill2d_x.F
module_initialize_quarter_ss.o : module_initialize_quarter_ss.F
module_initialize_squall2d_x.o : module_initialize_squall2d_x.F
module_initialize_squall2d_y.o : module_initialize_squall2d_y.F
module_initialize_real.o : module_initialize_real.F
module_dm.o : module_dm.F
start_domain.o : start_domain.F
mediation_integrate.o : mediation_integrate.F
module_configure.o : module_configure.F

solve_interface.o                                              \
shift_domain_em.o                                              \
module_io_mm5.o module_si_io.o module_io_wrf.o module_domain.o \
module_start.o module_initialize.o module_initialize_b_wave.o  \
module_initialize_hill2d_x.o module_initialize_quarter_ss.o    \
module_initialize_squall2d_x.o module_initialize_squall2d_y.o  \
mediation_integrate.o \
module_configure.o                                             \
module_initialize_real.o module_dm.o start_domain.o :
        $(RM) $@
        $(SED_FTN) $*.F > $*.b
        $(CPP) -I../inc $(CPPFLAGS) $*.b  > $*.f90
        $(RM) $*.b
        $(FC) -c $(PROMOTION) $(FCBASEOPTS) $(MODULE_DIRS) $*.f90

Hi Mat,

thanks for you answer.
If I try the configuration you suggested, compilation fails.
I solved the problem using the single threaded config option (without open mp support) , then I partially clean (with ./clean and not ./clean -a) and rebuild using your options, and it works.

Maybe I cannot compile only a few libraries with Openmp support, and compiling them without it works.

The problem is that doing so the wrf performance is very low, probably affected by the trick i used.

In fact running the model you can see a strange behaviour: one step fast and one step low, like this (even using MPICH on the same shared memory machine performs better…strange, it shouldn’t):

Timing for Writing wrfout_d01_2000-01-24_15:00:00 for domain 1:
0.04400 elapsed seconds.
Timing for main: time 2000-01-24_15:03:00 on domain 1: 1.80500 elapsed
seconds.
Timing for main: time 2000-01-24_15:06:00 on domain 1: 2.34000 elapsed
seconds.
Timing for main: time 2000-01-24_15:09:00 on domain 1: 0.32200 elapsed
seconds.
Timing for main: time 2000-01-24_15:12:00 on domain 1: 2.29700 elapsed
seconds.
Timing for main: time 2000-01-24_15:15:00 on domain 1: 0.30300 elapsed
seconds.
Timing for main: time 2000-01-24_15:18:00 on domain 1: 2.27700 elapsed
seconds.
Timing for main: time 2000-01-24_15:21:00 on domain 1: 0.30000 elapsed
seconds.
Timing for main: time 2000-01-24_15:24:00 on domain 1: 2.29400 elapsed
seconds.
Timing for main: time 2000-01-24_15:27:00 on domain 1: 0.32300 elapsed
seconds.
Timing for main: time 2000-01-24_15:30:00 on domain 1: 2.27300 elapsed
seconds.
Timing for main: time 2000-01-24_15:33:00 on domain 1: 1.72300 elapsed
seconds.
Timing for main: time 2000-01-24_15:36:00 on domain 1: 2.28900 elapsed
seconds.
Timing for main: time 2000-01-24_15:39:00 on domain 1: 0.30000 elapsed
seconds.
Timing for main: time 2000-01-24_15:42:00 on domain 1: 2.38200 sed
onds.

Very strange…does anybody ever experienced such problem?
I just submitted this post even to wrf user forum, maybe it’more appropriate…

thanks for you help


aldo