unexpected performance behaviour on AMD64


I am testing the performances of geophysical dynamics codes
on a cluster of Opteron 2p boxes with PGI CDK 6.
Until few weeks ago the environment was based
on 2.4.21 Linux kernel.
A migration to the new kernel 2.6.9 was done to the system
since most people reports it is quite beneficial for
the NUMA-ness of the Opteron architecture.
However, I observe a drop in performance of 10-20%.
Any hints about possible sources of this unexpected behaviour ?

PGI is 6.0-2 64-bit target on x86-64 Linux
Linux is 2.6.9-5.0.5.ELsmp
the code is Fortran 77/90 + MPI with optimization flags:
-byteswapio -r8 -Mnodclchk -Mextend -Ktrap=fp
-tp k8-64 -pc=64 -fastsse -O3 -Msmart -Kieee -Mvect=cachesize:1048576,transform

Many thanks for the help


From the release notes I noticed that PGI 6.0 supports NUMAness
only under SuSE and SLES 9.x while I am testing a clone of RHEL 4.
Could this explain performances ?


Hi cmn,

It’s my understanding (from second-hand knowledge) that Red Hat’s NUMA support has problems with how it handles memory. It’s suppose to be fixed with their 2.6.12 build of the kernel. (SuSE 9.3 is fine) So, yes it is possible that this is the problem, but unless you are explicitly linking with the NUMA libraries, you’re most likely getting the NPTL which is the default. With the 6.0-5 version of the compiler, we’ve added a flag “-mp=numa” which will link in the NUMA libraries if available.

Unfortunately, this is a very difficult problem to diagnose. I’d first start by gathering a profile by compiling and linking with “-Mprof=mpi”. If you happen to have an old profile, then you can compare differences to pinpoint the problem. If not, then the profile might at least highlight some bottlenecks.

  • Mat

With the 6.0-5 version …
is it easy to upgrade from the existing 6.0-2 to the newer one?
I imagine it is possibile to download the new SW and reinstall
on top of the existing directory tree, keeping the same license.dat.


Hi Carlo,

It is easy to upgrade the compilers from 6.0-2 to 6.0-5. Simply download (See here) and install the latest PGI Workstation product in the same directory as the CDK. The compilers are the same between the CDK and Workstation products. The main difference is the license and that the CDK comes with more support libraries and applications for cluster developement. Your license and the CDK specific files will be uneffected when you install the Workstation.

  • Mat