MM5 performance on AMD64 Opteron

I lately compared the cpu time between Xenon and AMD64 in running MM5 codes (compiled on PGI 5.2), and found that the speed up was merely 1.4 times. Although the Xenon chip has a clock rate 2.8GHz faster than the AMD64 at 1.8GHz, I would expect the benefit of using 64 bits will still triumph a faster 32bit chips by a large margin.

I wonder if there’s any problem with my optimization setting or the environment? I am using the configure.user file provided in the “tips and resource” for compiling the AMD64/MM5 on Federa Linux. Do I need to use other Linux favor such as SuSE?

Any input will be greatly appreciated!

PS I also have other problems with compiling other modules of MM5 on AMD64/PGI fortran. May post my questions later …

Hi Jesse,


The only thing a 64-bit system can guarantee you is that it can handle larger problem sizes. A 64-bit system should at the very least have the same performance as a 32-bit executable running on the same system. Some codes will see a performance benefit, and while rare, others actually degrade.

I just ran MM5 on a 1600 Mhz Dual CPU AMD Opteron running SuSE9.1 using both a 64-bit and 32-bit executable. The 64-bit executable took 144 seconds while the 32-bit took 148 seconds. Some performance benefit, but not much.

To better compare the Xeon and the AMD64, I’d run the Xeon’s MM5 executable on the AMD64. Comparing this result to the 64-bit executable’s result should give you a better idea of what amount of the performance increase can be attributed to 64-bits. Comparing the two 32-bit results will show what portion of the speed-up is due to differences between the two systems.

Please feel free to ask questions about the problems your having with the MM5 modules. I’m not an expert in building these modules, but we should be able to work through each problem. I’m sure if your encountering a problem, others are as well.

  • Mat

I tried your suggestions but failed to run the codes. For example, I compiled MM5 on Xenon and transported them to opteron for a trial run, which failed. The Xenon machine runs on redHat Linux 9 whilst the opteron sits on federa linux. Both have pgi 5.2-4 fortran.

While it is interesting to compare xenon and amd, but I am most concerned with compiling and running mm5 codes. I encountered quite a bit of problems. First, I have some problem with NCAR Graphics. Then I tried to compile the MM5/TERRAIN module without ncargraphics. The compilation apparently succeeded but the codes always fail with segmentation fault. Can you help?

Best regards

Hi Jesse,


Granted it’s not about TERRAIN, but we have created a guide to building and running MM5 (see http://www.pgroup.com/resources/isv.htm#MM5) which might help.

MM5 uses a lot of stack space, so usually when MM5 seg faults it’s because of a stack overflow. To correct, set your stack size to unlimited (check your shell docs for the specific command to increase your stack space. On csh its ‘unlimit’).

The second cause is forgetting to add “-Mbyteswapio -Mnosgimp” and in 64-bits “-DDEC_ALPHA”. Adding “-DDEC_ALPHA” in 32-bits will also cause a seg fault.

I’ve only built the NCAR Graphics library and TERRAIN once but don’t remember the issues (its been awhile). I’ll try it again and log what I do.

  • Mat

Yes, please post your log after coming back from testing ncarg and terrain. In particular, I want to know if one should compile these modules as 32bit or 64bit on opteron?

My compilation problems are with the other MM5 modules, not MM5 itself. Since there are so many modules, I want to resolve them one at a time. Your experience with ncarg and terrain will be helpful.

Great thanks for your prompt response!

Hi Jesse,

I was able to work on the NCAR Graphic library and was able to get the test case to work in 32-bits (on a P4 and AMD64). I did have to make some modifications to the configuration, but not much. To get it to work in 64-bits will require some porting which I have not done. I may have the time later but for now the 32-bit version should work fine. I’ll try and fit in TERRAIN today or tommorrow.

In both cases I used pgf90 and pgcc release 5.2-4 and ncag release 4.3.1. Disclamier: I’ve only done a minimal amount of testing. The accompanying test case worked but I have not tested beyond that. Also, I have not tried increasing the optimization.

Notes on building and installing NCAR can be found at http://ngwww.ucar.edu/ng/installsrc/html.

Steps on a P4:

  1. Download and untar the package
  2. set NCARG to the “nacag-4.3.1” directory.
  3. Modify the file config/LINUX.pgi
  • Add “-lpgf90 -lpgf90_rpm1 -lpgf902” to the “CtoFLibraries”
  1. copy config/LINUX to config/LINUX.org
  2. copy config/LINUX.pgi to config/LINUX
  3. Edit config/Template by either removing or commenting out
    lines 833 and 835. Change line 834 from “.f” to “.F”.
    The resulting preprocesser “.f” files and not formatted properly, so I
    removed the preprocessing since pgf90 can handle “.F” files directly.
  4. Run “./Configure -v” and follow the steps. Be sure to build the
    F90 libraries and set the f90 compiler to ‘pgf90’. I did not use
    the HDF libraries.
  5. Run “make Everything >& make.log”. Inspect the log for errors.
  6. set your PATH to the location you installed the ncar executables.
    Also set NCARG_ROOT.
  7. Test the configuration by running
    % ncargex cpex08
    % ctrans -d X11 cpex08.ncgm

32-bits on AMD64

  1. Download and untar the package
  2. set NCARG to the “nacag-4.3.1” directory.
  3. Set your compiler path to the 32-bit compilers. These
    are located in “$PGI/linux86/5.2/bin”. The 64-bit compilers
    are located in “$PGI/linux86-64/5.2/bin”.
  4. Modify the file config/LINUX.pgi
  • Add “-lpgf90 -lpgf90_rpm1 -lpgf902” to the “CtoFLibraries”
  • Add the line “#define LdCommand ld -m elf_i386”
  1. copy config/LINUX to config/LINUX.org
  2. copy config/LINUX.pgi to config/LINUX
  3. Add “x86_64:” to line 357 of config/ymake. This will allow the
    configure script to recognize the system.
  4. Edit config/Template by either removing or commenting out
    lines 833 and 835. Change line 834 from “.f” to “.F”.
    The resulting preprocesser “.f” files and not formatted properly, so I
    removed the preprocessing since pgf90 can handle “.F” files directly.
  5. Run “./Configure -v” and follow the steps. Be sure to build the
    F90 libraries and set the f90 compiler to ‘pgf90’. I did not use
    the HDF libraries.
  6. Run “make Everything >& make.log”. Instpect the log errors.
  7. set your PATH to the location you installed the ncar executables.
    Also set NCARG_ROOT
  8. Test the configuration by running
    % ncargex cpex08
    % ctrans -d X11 cpex08.ncgm


    Let me know if you have problems or if something is unclear.
  • Mat

I have successfully compiled TERRAIN in 32bit on AMD64. As said, modified the PGI path pointing to 32bit pgi version. Then, edit TERRAIN/Makefile. Add “-L/usr/X11R6/lib” to LOCAL_LIBRARIES; else, the linker cannot find libX11.

May you kindly take a loot at REGRID/pregrid… As before I used 32bit pgi fortran to compile the codes. What happens is that a c program: uti/cio.c is compiled by gnu c which produces 64bit output. What should I do to get around this problem? Thanks!

Hi Jesse,


Either add “-m32” to the CCFLAGS or add a line that sets CC to pgcc. “-m32” is gcc’s flag to tell the compiler to create a 32-bit object.

FYI, I’ve been able to compile the 32-bit TERRAIN with the NCAR Libraries and in 64-bits without the libraries. To compile in 64-bits change the “LOCAL_LIBRARIES” to “-L/usr/X11R6/lib64 -lX11 -L/usr/lib64 -g77libs” and set “NCARGRAPHICS” to “NONCARG”. On 64-bit systems 32-bit libraries are still in “lib” while 64-bit libraries are in “lib64”. However, I’m not sure if it’s really worth it to compile this 64-bits. But that’s one of the great thing about AMD64, if can’t or don’t want to port your app to 64-bits, you don’t have to!

I’m still downloading data so I haven’t tested it yet. I’ll update you if I see the seg fault you reported.

  • Mat

Hi Jesse,


The seg fault your seeing with the 64-bit terrain.exe is caused by a subtle coding error in the file src/terdrv.F. The subroutine TERDRV calls WATERCHECK using 10 actual arguments while the WATERCHECK subroutine is expecting 11. Adding a 11th argument “0” at line 477 and 481 fixes the problem.

Change:

       CALL WATERCHECK(IX,JX,NVC,FORCETER,VEGEC,LVEGE,IWATER,
    -                  49.0,RAD,DIS(N))

to

       CALL WATERCHECK(IX,JX,NVC,FORCETER,VEGEC,LVEGE,IWATER,
    -                  49.0,RAD,DIS(N),0)

Why does this code ever work? Because the program gets lucky when the stack happens to contain a valid value.

  • Mat

Hi,

can anyone post a list of Makefile options that work on amd opteron using pgf90 (suse 9.1 if relevant).

I have TERRAIN working and INTERPRF - but am stuggling with REGRID - the error I get when I run the Makefile is

/usr/bin/cpp -I. -C -P -traditional gribcode.F > gribcode.f
pgf90 -c -Mfreeform -byteswapio -I…/util gribcode.f
PGF90-S-0038-Symbol, mwsize, has not been explicitly declared (gribcode.f)
0 inform, 0 warnings, 1 severes, 0 fatal for findgrib
PGF90-S-0038-Symbol, mwsize, has not been explicitly declared (gribcode.f)
0 inform, 0 warnings, 1 severes, 0 fatal for gribget
PGF90-S-0038-Symbol, mwsize, has not been explicitly declared (gribcode.f)
0 inform, 0 warnings, 1 severes, 0 fatal for gribheader
make[2]: [gribcode.o] Error 2 (ignored)

thanks,

julie

Hi Julie,

The Makefile that comes with the REGRID package from UCAR has a “Linux” section in it which is configured for use with pgf90. The selection of the “Linux” section is done automatically in the Makefile by grepping the output of the “uname -a” command.

Most likely, your Makefile has been modified and the “-DDEC -DBIT32” define flags were removed from the “CPPFLAGS” and “CCFLAGS” variables. To fix, you can add these define flags back in to both the regridder and pregridder sections, or re-download the REGRID.tar.gz package.

Hope this helps,
Mat

thanks - that does help.

I am also having problems getting ictrans working on opteron - I was trying to follow your instructions given previously - but can’t work out which lines to edit because my files don’t have the same line numbers as yours.

Alternatively - are there any binaries out there for ictrans and idt ?

thanks,

Julie

Hi Julie,

I downloaded the latest NCAR graphics library (4.4.1) this morning and was able to get “ictrans” and “idt” to compile. I first moved the “config/LINUX.64.PGI” to “config/LINUX” and then ran “./Configure” as normal. I think this config file is new as of the 4.4.0 release.

As for binaries, I did see a binary available for Red Hat 9.0 using the PGI compilers, but I don’t know if these were for 32 or 64-bit.

  • Mat

Hi,

I’m trying to compile ncarg-4.4.1 using pgf90 (version 6.0) and gcc (3.3.5). The ncarg package comes with LINUX.GNU and LINUX.PGI config files but no config file is available to work with pgf90 and gcc.

Trying to create a $NCARG_INSTALL/config/LINUX file that uses pgf90 and gcc I always get unresoveld symbols.

Does anybody have a “…/config/LINUX” file that works with pgf90 and gcc ?

Thanks,
Víctor.

PS: By the way, I’m running debian 32bits on an AMD64 system.

Hi Victor,

The problem is that they use gcc to link, but the PGI runtimes are not being included on the link line. I tried changing the “CtoFLibraries” in the LINUX file to include these libraries, but for some reason, Configure just ignores them.

Instead, either edit the Makefile’s “CTOFLIBS” variable to include the following libraries, or simply pass them via the command line:

make Everything CTOFLIBS=" -L/usr/pgi/linux86/6.0/lib -lpgf90  -lpgf90_rpm1  -lpgf902 -lpgf90rtl -lpgftnrtl -lpgc -lg2c -lgcc -lm" >& make-output &

Please change the PGI library path if different. I was able to successfully compile all objects using pgf90 and gcc when I added the correct runtime libraries.

Hope this helps,
Mat