OpenMP threads problem on Intel CPU

Hi,

My system is CentOS 5.11 equipped with two Intel® Xeon® CPU E5-2650 with HyperThreading enabled, so the machine shows there are 32 threads I can use.
I try to run an OpenMP program and set OMP_NUM_THREADS to 8 or more the application does not run correctly. But when I run on a system equipped with AMD CPU, it worked fine.
The program showed error message on Intel CPU:
“OMP_NUM_THREADS or NCPUS value (8) is invalid”
I can only use 4 processors and can not be greater than 4, but I want to use 8 and more processors to fasten my forecast model.My pgf90 version is 6.2 which is a bit old version.

Best regards,
Yi-Chuan

Here is a program, check2.f, that determines if your system
can run OpenMP across multiple threads.


pgf90 check2.f dclock_64.s -o check2 -mp
Here is check2.f

        program test
        integer j,k
        integer omp_get_num_procs, omp_get_max_threads
        integer omp_get_num_threads
        integer thread(20)
        integer,parameter:: max_thrd=4
        real*8 dclock, time1, time2
        thread(1)=1
        do i=1,max_thrd
           thread(i+1)=2*thread(i)
        end do
        print *,"number of cores  =", omp_get_num_procs()
        print *,"max threads =", omp_get_max_threads()
        print *,"current num threads =", omp_get_num_threads()
!       call system("uname -a")
        j = 200    ! may want to change this
        do ii=1,max_thrd+1

                call omp_set_num_threads(thread(ii))
                time1 = dclock()
!$omp parallel
!$omp do
                do k = 1, thread(max_thrd+1)
                        call delay(j)
                enddo
!$omp end parallel
                time2 = dclock() - time1
                print *, thread(ii)," core test - delay value =",
     *         j*thread(max_thrd+1),
     *          " time =", time2, " seconds"
        end do
        end
       subroutine delay(n)
       integer n
       integer i
       do i=1,n
          call abc()
       end do
       return
       end
       subroutine abc()
       integer i
       do i=1,1000000
           call def()
       end do
       return
       end
       subroutine def()
         return
       end



Here is the timing routine dclock_64.s


        .file   "dclock-hammer.s"
        .align    8
        .data
# .clock:  .double 0.000000001           # 1.0 GHz
# .clock:  .double 0.000000000750187     # 1.33GHz
# .clock:  .double 0.000000000714        # 1.4 GHz
# .clock:  .double 0.000000000666        # 1.5 GHz
# .clock:  .double 0.000000000625        # 1.6 GHz
# .clock:  .double 0.00000000059         # 1.7 GHz
# .clock:  .double 0.0000000005556       # 1.8 GHz
# .clock:  .double 0.0000000005          # 2.0 GHz
# .clock:  .double 0.000000000455        # 2.2 GHz
# .clock:  .double 0.000000000417        # 2.4 GHz
 .clock:  .double 0.000000000376        # 2.66 GHz
# .clock:  .double 0.000000000357        # 2.8 GHz
# .clock:  .double 0.0000000003333       # 3.0 GHz
# .clock:  .double 0.0000000003125       # 3.2 GHz
# .clock:  .double 0.0000000002777       # 3.6 GHz

.low:   .long 0x00000000
.high:  .long 0x00000000
        .text

        .globl   _DCLOCK, dclock, _dclock, _dclock_, dclock_
_DCLOCK:
dclock:
_dclock:
_dclock_:
dclock_:
        .byte   0x0f, 0x31

        movl    %eax, .low(%RIP)
        movl    %edx, .high(%RIP)

        fildll  .low(%RIP)
        fmull   .clock(%RIP)
        fstpl   -24(%rsp)
        movsd   -24(%rsp), %xmm0
        ret

You should see 1, 2, 4, 8, 16 threads run the same
(non-memory using) computations, and time the
work. Each time the threads double, the processing time should
cut in half from before.
If you extend to 32 threads(max_thrd=5), the hyper-threads, which are not on different cores, should speed nothing up. OpenMP should first assign real cores before assigning hyper-thread cores (which are a second set of registers for the same CPU core).

pgf90 -V
on your system should indicate what type CPUs the compiler thinks you have.

If this works and your code still fails, send your failing program
to trs@pgroup.com so we can take a look.

dave

I just reread the forum entry and realized you had 6.2 release. Back then
we did limit the number of threads Open MP created. Probably 8 or 16.

If you want to use more threads, download the Community Edition 17.4
and use that. A better experience.

dave

Hi, Dave

I tried Community Edition 17.4, and compiled the program successfully, but when I implemented the program, I saw it just took one thread to run. I think it may be the compiler flag problems, but I don’t really understand what should be changed in Makefile.

The following lines are the flags written in Makefile.
LDFLAGS = -L $(PGI)/linux86-64/$(ver)/libso -L $(PGI)/linux86-64/$(ver)/lib -L $(RAP_SHARED_LIB_DIR) -L $(RAP_LIB_DIR)
LOC_PGFFLAGS= -mcmodel=medium -fast -g77libs -mp -Minform=inform -Wl,-Bstatic
GFFLAGS= -mcmodel=medium -fast -g77libs -mp -Minform=inform -Wl,-Bstatic
FFLAGS = -mcmodel=medium -Msave -fast -g77libs -mp -Minform=inform
F90FLAGS = -mcmodel=medium -Msave -fast -g77libs -mp -Minform=inform
LOC_CPPC_CFLAGS = -DPGI_IN_USE -mcmodel=medium

LOC_LDFLAGS = -L $(PGI)/linux86-64/$(ver)/libso -L $(PGI)/linux86-64/$(ver)/lib
-L $(RAP_SHARED_LIB_DIR) -L $(RAP_LIB_DIR)
LOC_LIBS=-lSpdb -lMdv -ldsserver -ldidss -leispack
-lrapformats -ltoolsa -lrapmath -ldataport -ltdrp
-lpgmp -lpgf90 -lpgf90_rpm1 -lpgf902
-lpgf90rtl -lpgftnrtl $(PGI)/linux86-64/$(ver)/lib/nonuma.o
-lnspgc -lpgc -lm -lgcc -lc -lgcc
-lpthread -lrt

And, I examined the check2.f code, it could use up to 16 threads to run no matter which pgf90 version I use. I have tested 6.2, 8.0, 8.0-5, 12.10 and 17.4, all of them are OK. However, my program cannot use threads greater than 4 on Intel CPU.

Many thanks,
Yi-Chuan

Make sure you use the -mp switch during compilation so that
OpenMP directives are processed.

Make sure you use the -mp switch during LINKING so that multi-threaded openmp routines will be linked, rather than dummy single threaded
routines.

I would drop

-mcmodel=medium -g77libs
and see if things still run.

I would not try to list all the PGI libs normally linked because
that is the driver’s job, and the linked list and order changes.

Just add libs not put in normally.

dave

Hi, Dave

I tried your suggestion which is drop -g77libs and add -mp during linking, but still got only one thread to run even though omp_num_threads was greater than one.

Actually, the library during linking have to be listed, or it would report errors of “undefined reference to XXX”. I don’t fully understand what driver do during linking, and what kind of driver will do this. I need to learn more about it.

I list the messages output while compiling the program below. Maybe it can help you to diagnose where goes wrong.

make _CC="gcc34" _CPPC="pgc++ -mp" _FC="pgf90" _F90C="pgf90" \
	DBUG_OPT_FLAGS="" target
make[1]: Entering directory `/work3/myclo/VDRAS/VDRAS_ice_ter'
echo Making C++ program ... ; \
	make _CC="gcc34" _CPPC="pgc++ -mp" _FC="pgf90" DBUG_OPT_FLAGS="" DEBUG_CFLAGS="" DEBUG_LIBS="" DEBUG_LDFLAGS="" SYS_LIBS="" SYS_CFLAGS=" -DLINUX_IL6 -D_BSD_TYPES -DF_UNDERSCORE2" adjoint_moist;
Making C++ program ...
make[2]: Entering directory `/work3/myclo/VDRAS/VDRAS_ice_ter'
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   adjointmain.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   adv.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   adv2.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   adj.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   advect.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   bdya1.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   bdya2.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   coneva.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   coneva1.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   calc_RF_Factor.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   cmic.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   diffa.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   diffp.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   diffus.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   direct.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   dtdqc.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   ecalc.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   eigenx.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   eigeny.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   eigenz.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   fltall.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   force.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   func.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   funcgr.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   grd.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   hpder.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   init.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   linrg.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   rev2.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   setg.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   setup.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   grdchk.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   xder.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   yder.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   zder.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   lbfgs.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   grdchkp.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   bdycd2.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   fill3d.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   ssmo.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   miccon.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   buoy.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   pgfa.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   pgfd.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   buoya.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   convert_time.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   varia.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   surface_interpolation.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   pints.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   lnblnks.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   get_ascii_metars.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   vad.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   calc_background.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   ranf.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   fill2d.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   calcbkg.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   pot2d.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   sounding_barnes.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   sounding_barnes_tq.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   surface_error.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   filt1d.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   read_profiler.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   readradarN.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   rf.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   rfilter_1d.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   rfilter_1d_adj.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   rfilter_3d.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   pre_analyze.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   dualdoppler.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   sounding_vertical_interp.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   tguess.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   TransferTDRPints.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   TransferTDRPreals.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   TransferTDRPstrings.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   model_sounding_vertical_interp.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   readRAOB.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   grvctr.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   grvctra.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   sounding_barnes_1.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   vad_barnes.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   findppi.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   taiwansd.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   taiwanradar.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   taiwansurf.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   grads_write.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   grads_fcst_write.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   grads_sdint.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   grads_write_pu.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   cmic_ice.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   coneva_ice.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   dtdqc_ice.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   miccon_ice.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   buoya_ice.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   miccon_ice_ta.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   coneva_ice_ta.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   tguess_ice.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   mkter.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   terrain.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   grads_fcst_write_terrain.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   grads_write_au.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform   -c  -I/usr/local/rap/include -I/usr/local/rap/include   grads_write_ausfc.f
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium    -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE  -c  -I/usr/local/rap/include -I/usr/local/rap/include   Params.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium    -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE  -c  -I/usr/local/rap/include -I/usr/local/rap/include   f_system.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium    -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE  -c  -I/usr/local/rap/include -I/usr/local/rap/include   get_spdb_surface.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium    -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE  -c  -I/usr/local/rap/include -I/usr/local/rap/include   get_spdb_sounding.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium    -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE  -c  -I/usr/local/rap/include -I/usr/local/rap/include   adjoint.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium    -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE  -c  -I/usr/local/rap/include -I/usr/local/rap/include   Main.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium    -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE  -c  -I/usr/local/rap/include -I/usr/local/rap/include   find_latest_data.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium    -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE  -c  -I/usr/local/rap/include -I/usr/local/rap/include   read_mdv.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium    -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE  -c  -I/usr/local/rap/include -I/usr/local/rap/include   mdv_write.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium    -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE  -c  -I/usr/local/rap/include -I/usr/local/rap/include   mdv_write_fcst.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium    -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE  -c  -I/usr/local/rap/include -I/usr/local/rap/include   find_mdv_in_interval.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium    -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE  -c  -I/usr/local/rap/include -I/usr/local/rap/include   find_mdv_in_interval_forecast.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium    -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE  -c  -I/usr/local/rap/include -I/usr/local/rap/include   procmap_register.cc
echo Linking C++ program ...
Linking C++ program ...
LD_LIBRARY_PATH=/work3/myclo/pgi/linux86-64/17.4/lib::/opt/pgi8/linux86-64/8.0/lib:/opt/pgi8/linux86-64/8.0/libso:/work3/myclo/vapor-2.6.0/lib::
/bin/rm -f adjoint_moist
pgc++ -mp  -o adjoint_moist \
	 	  adjointmain.o adv.o adv2.o adj.o advect.o bdya1.o bdya2.o coneva.o coneva1.o calc_RF_Factor.o cmic.o diffa.o diffp.o diffus.o direct.o dtdqc.o ecalc.o eigenx.o eigeny.o eigenz.o fltall.o force.o func.o funcgr.o grd.o hpder.o init.o linrg.o rev2.o setg.o setup.o grdchk.o xder.o yder.o zder.o lbfgs.o grdchkp.o bdycd2.o fill3d.o ssmo.o miccon.o buoy.o pgfa.o pgfd.o buoya.o convert_time.o varia.o surface_interpolation.o pints.o lnblnks.o get_ascii_metars.o vad.o calc_background.o ranf.o fill2d.o calcbkg.o pot2d.o sounding_barnes.o sounding_barnes_tq.o surface_error.o filt1d.o read_profiler.o readradarN.o rf.o rfilter_1d.o rfilter_1d_adj.o rfilter_3d.o pre_analyze.o dualdoppler.o sounding_vertical_interp.o tguess.o TransferTDRPints.o TransferTDRPreals.o TransferTDRPstrings.o model_sounding_vertical_interp.o readRAOB.o grvctr.o grvctra.o sounding_barnes_1.o vad_barnes.o findppi.o taiwansd.o taiwanradar.o taiwansurf.o grads_write.o grads_fcst_write.o grads_sdint.o grads_write_pu.o cmic_ice.o coneva_ice.o dtdqc_ice.o miccon_ice.o buoya_ice.o miccon_ice_ta.o coneva_ice_ta.o tguess_ice.o mkter.o terrain.o grads_fcst_write_terrain.o grads_write_au.o grads_write_ausfc.o      Params.o f_system.o get_spdb_surface.o get_spdb_sounding.o adjoint.o Main.o find_latest_data.o read_mdv.o mdv_write.o mdv_write_fcst.o find_mdv_in_interval.o find_mdv_in_interval_forecast.o procmap_register.o    -L /work3/myclo/pgi/linux86-64/17.4/libso -L /work3/myclo/pgi/linux86-64/17.4/lib -L /usr/local/rap/lib   -L /usr/local/rap/lib -lSpdb -lMdv -ldsserver -ldidss -leispack -lrapformats -ltoolsa -lrapmath -ldataport -ltdrp -lpgmp -lpgf90 -lpgf90_rpm1 -lpgf902 -lpgf90rtl -pgf90libs -lpgftnrtl -lnuma -lnspgc -lpgc -lm -lgcc -lc -lgcc  
make[2]: Leaving directory `/work3/myclo/VDRAS/VDRAS_ice_ter'
make[1]: Leaving directory `/work3/myclo/VDRAS/VDRAS_ice_ter'

Many thanks again.
Yi-Chuan[/url]

You are linking PGI 8.0 libs with PGI 17.4? Risky.


Instead of -L/path/to/pgi/f90/libs -lpgi1lib -lpgi2lib etc
just use

-pgf90libs

so the current pgf90 libs are linked with pgc++ in the correct
versions and order.

Your use of -Msave and -mcmodel=medium suggests you are declaring large arrays statically, and you wish them all to be zero upon program entry. This makes for very long execution times as these huge
arrays inside the executable are loaded and executed.

Better to dynamically create and initialize the big arrays - smaller and faster.



dave