# OpenMP threads problem on Intel CPU

**URL:** <https://forums.developer.nvidia.com/t/openmp-threads-problem-on-intel-cpu/135418>\
**Category:** Legacy PGI Compilers (archived)\
**Created:** [September 21, 2017, 4:05am UTC](https://forums.developer.nvidia.com/t/openmp-threads-problem-on-intel-cpu/135418 "2017-09-21T04:05:17Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![fa72228565](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@fa72228565](https://forums.developer.nvidia.com/u/fa72228565)\
**Post date:** [September 21, 2017, 4:05am UTC](https://forums.developer.nvidia.com/t/openmp-threads-problem-on-intel-cpu/135418/1 "2017-09-21T04:05:17Z")

</div>

Hi,

My system is CentOS 5.11 equipped with two Intel(R) Xeon(R) CPU E5-2650 with HyperThreading enabled, so the machine shows there are 32 threads I can use.  
I try to run an OpenMP program and set OMP\_NUM\_THREADS to 8 or more the application does not run correctly. But when I run on a system equipped with AMD CPU, it worked fine.  
The program showed error message on Intel CPU:  
“OMP\_NUM\_THREADS or NCPUS value (8) is invalid”  
I can only use 4 processors and can not be greater than 4, but I want to use 8 and more processors to fasten my forecast model.My pgf90 version is 6.2 which is a bit old version.

Best regards,  
Yi-Chuan

---

<div class="post-metadata">

**Author:** ![tull](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@tull](https://forums.developer.nvidia.com/u/tull)\
**Post date:** [September 21, 2017, 5:15pm UTC](https://forums.developer.nvidia.com/t/openmp-threads-problem-on-intel-cpu/135418/2 "2017-09-21T17:15:03Z")

</div>

Here is a program, check2.f, that determines if your system  
can run OpenMP across multiple threads.  
  
  
pgf90 check2.f dclock\_64.s -o check2 -mp  
Here is check2.f

```plaintext
        program test
        integer j,k
        integer omp_get_num_procs, omp_get_max_threads
        integer omp_get_num_threads
        integer thread(20)
        integer,parameter:: max_thrd=4
        real*8 dclock, time1, time2
        thread(1)=1
        do i=1,max_thrd
           thread(i+1)=2*thread(i)
        end do
        print *,"number of cores =", omp_get_num_procs()
        print *,"max threads =", omp_get_max_threads()
        print *,"current num threads =", omp_get_num_threads()
! call system("uname -a")
        j = 200 ! may want to change this
        do ii=1,max_thrd+1

                call omp_set_num_threads(thread(ii))
                time1 = dclock()
!$omp parallel
!$omp do
                do k = 1, thread(max_thrd+1)
                        call delay(j)
                enddo
!$omp end parallel
                time2 = dclock() - time1
                print *, thread(ii)," core test - delay value =",
     * j*thread(max_thrd+1),
     * " time =", time2, " seconds"
        end do
        end
       subroutine delay(n)
       integer n
       integer i
       do i=1,n
          call abc()
       end do
       return
       end
       subroutine abc()
       integer i
       do i=1,1000000
           call def()
       end do
       return
       end
       subroutine def()
         return
       end

```

> Here is the timing routine dclock\_64.s

```plaintext
        .file "dclock-hammer.s"
        .align 8
        .data
# .clock: .double 0.000000001 # 1.0 GHz
# .clock: .double 0.000000000750187 # 1.33GHz
# .clock: .double 0.000000000714 # 1.4 GHz
# .clock: .double 0.000000000666 # 1.5 GHz
# .clock: .double 0.000000000625 # 1.6 GHz
# .clock: .double 0.00000000059 # 1.7 GHz
# .clock: .double 0.0000000005556 # 1.8 GHz
# .clock: .double 0.0000000005 # 2.0 GHz
# .clock: .double 0.000000000455 # 2.2 GHz
# .clock: .double 0.000000000417 # 2.4 GHz
 .clock: .double 0.000000000376 # 2.66 GHz
# .clock: .double 0.000000000357 # 2.8 GHz
# .clock: .double 0.0000000003333 # 3.0 GHz
# .clock: .double 0.0000000003125 # 3.2 GHz
# .clock: .double 0.0000000002777 # 3.6 GHz

.low: .long 0x00000000
.high: .long 0x00000000
        .text

        .globl _DCLOCK, dclock, _dclock, _dclock_, dclock_
_DCLOCK:
dclock:
_dclock:
_dclock_:
dclock_:
        .byte 0x0f, 0x31

        movl %eax, .low(%RIP)
        movl %edx, .high(%RIP)

        fildll .low(%RIP)
        fmull .clock(%RIP)
        fstpl -24(%rsp)
        movsd -24(%rsp), %xmm0
        ret

```

You should see 1, 2, 4, 8, 16 threads run the same  
(non-memory using) computations, and time the  
work. Each time the threads double, the processing time should  
cut in half from before.  
If you extend to 32 threads(max\_thrd=5), the hyper-threads, which are not on different cores, should speed nothing up. OpenMP should first assign real cores before assigning hyper-thread cores (which are a second set of registers for the same CPU core).

pgf90 -V  
on your system should indicate what type CPUs the compiler thinks you have.

If this works and your code still fails, send your failing program  
to [trs@pgroup.com](mailto:trs@pgroup.com) so we can take a look.

dave

---

<div class="post-metadata">

**Author:** ![tull](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@tull](https://forums.developer.nvidia.com/u/tull)\
**Post date:** [September 21, 2017, 8:15pm UTC](https://forums.developer.nvidia.com/t/openmp-threads-problem-on-intel-cpu/135418/3 "2017-09-21T20:15:46Z")

</div>

I just reread the forum entry and realized you had 6.2 release. Back then  
we did limit the number of threads Open MP created. Probably 8 or 16.

If you want to use more threads, download the Community Edition 17.4  
and use that. A better experience.

dave

---

<div class="post-metadata">

**Author:** ![fa72228565](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@fa72228565](https://forums.developer.nvidia.com/u/fa72228565)\
**Post date:** [September 22, 2017, 3:10am UTC](https://forums.developer.nvidia.com/t/openmp-threads-problem-on-intel-cpu/135418/4 "2017-09-22T03:10:16Z")

</div>

Hi, Dave

I tried Community Edition 17.4, and compiled the program successfully, but when I implemented the program, I saw it just took one thread to run. I think it may be the compiler flag problems, but I don’t really understand what should be changed in Makefile.

The following lines are the flags written in Makefile.  
_LDFLAGS = -L $(PGI)/linux86-64/$(ver)/libso -L $(PGI)/linux86-64/$(ver)/lib -L $(RAP\_SHARED\_LIB\_DIR) -L $(RAP\_LIB\_DIR)  
LOC\_PGFFLAGS= -mcmodel=medium -fast -g77libs -mp -Minform=inform -Wl,-Bstatic  
GFFLAGS= -mcmodel=medium -fast -g77libs -mp -Minform=inform -Wl,-Bstatic  
FFLAGS = -mcmodel=medium -Msave -fast -g77libs -mp -Minform=inform  
F90FLAGS = -mcmodel=medium -Msave -fast -g77libs -mp -Minform=inform  
LOC\_CPPC\_CFLAGS = -DPGI\_IN\_USE -mcmodel=medium_  
   
_LOC\_LDFLAGS = -L $(PGI)/linux86-64/$(ver)/libso -L $(PGI)/linux86-64/$(ver)/lib   
-L $(RAP\_SHARED\_LIB\_DIR) -L $(RAP\_LIB\_DIR)  
LOC\_LIBS=-lSpdb -lMdv -ldsserver -ldidss -leispack   
-lrapformats -ltoolsa -lrapmath -ldataport -ltdrp   
-lpgmp -lpgf90 -lpgf90\_rpm1 -lpgf902   
-lpgf90rtl -lpgftnrtl $(PGI)/linux86-64/$(ver)/lib/nonuma.o   
-lnspgc -lpgc -lm -lgcc -lc -lgcc   
-lpthread -lrt_

And, I examined the check2.f code, it could use up to 16 threads to run no matter which pgf90 version I use. I have tested 6.2, 8.0, 8.0-5, 12.10 and 17.4, all of them are OK. However, my program cannot use threads greater than 4 on Intel CPU.

Many thanks,  
Yi-Chuan

---

<div class="post-metadata">

**Author:** ![tull](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@tull](https://forums.developer.nvidia.com/u/tull)\
**Post date:** [September 22, 2017, 4:59pm UTC](https://forums.developer.nvidia.com/t/openmp-threads-problem-on-intel-cpu/135418/5 "2017-09-22T16:59:28Z")

</div>

Make sure you use the -mp switch during compilation so that  
OpenMP directives are processed.

Make sure you use the -mp switch during LINKING so that multi-threaded openmp routines will be linked, rather than dummy single threaded  
routines.

I would drop

-mcmodel=medium -g77libs  
and see if things still run.

I would not try to list all the PGI libs normally linked because  
that is the driver’s job, and the linked list and order changes.

Just add libs not put in normally.

dave

---

<div class="post-metadata">

**Author:** ![fa72228565](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@fa72228565](https://forums.developer.nvidia.com/u/fa72228565)\
**Post date:** [September 25, 2017, 1:28pm UTC](https://forums.developer.nvidia.com/t/openmp-threads-problem-on-intel-cpu/135418/6 "2017-09-25T13:28:04Z")

</div>

Hi, Dave

I tried your suggestion which is drop -g77libs and add -mp during linking, but still got only one thread to run even though omp\_num\_threads was greater than one.

Actually, the library during linking have to be listed, or it would report errors of “undefined reference to XXX”. I don’t fully understand what driver do during linking, and what kind of driver will do this. I need to learn more about it.

I list the messages output while compiling the program below. Maybe it can help you to diagnose where goes wrong.

```nohighlight
make _CC="gcc34" _CPPC="pgc++ -mp" _FC="pgf90" _F90C="pgf90" \
	DBUG_OPT_FLAGS="" target
make[1]: Entering directory `/work3/myclo/VDRAS/VDRAS_ice_ter'
echo Making C++ program ... ; \
	make _CC="gcc34" _CPPC="pgc++ -mp" _FC="pgf90" DBUG_OPT_FLAGS="" DEBUG_CFLAGS="" DEBUG_LIBS="" DEBUG_LDFLAGS="" SYS_LIBS="" SYS_CFLAGS=" -DLINUX_IL6 -D_BSD_TYPES -DF_UNDERSCORE2" adjoint_moist;
Making C++ program ...
make[2]: Entering directory `/work3/myclo/VDRAS/VDRAS_ice_ter'
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include adjointmain.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include adv.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include adv2.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include adj.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include advect.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include bdya1.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include bdya2.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include coneva.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include coneva1.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include calc_RF_Factor.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include cmic.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include diffa.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include diffp.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include diffus.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include direct.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include dtdqc.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include ecalc.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include eigenx.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include eigeny.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include eigenz.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include fltall.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include force.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include func.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include funcgr.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include grd.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include hpder.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include init.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include linrg.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include rev2.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include setg.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include setup.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include grdchk.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include xder.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include yder.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include zder.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include lbfgs.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include grdchkp.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include bdycd2.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include fill3d.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include ssmo.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include miccon.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include buoy.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include pgfa.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include pgfd.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include buoya.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include convert_time.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include varia.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include surface_interpolation.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include pints.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include lnblnks.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include get_ascii_metars.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include vad.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include calc_background.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include ranf.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include fill2d.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include calcbkg.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include pot2d.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include sounding_barnes.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include sounding_barnes_tq.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include surface_error.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include filt1d.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include read_profiler.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include readradarN.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include rf.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include rfilter_1d.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include rfilter_1d_adj.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include rfilter_3d.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include pre_analyze.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include dualdoppler.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include sounding_vertical_interp.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include tguess.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include TransferTDRPints.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include TransferTDRPreals.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include TransferTDRPstrings.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include model_sounding_vertical_interp.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include readRAOB.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include grvctr.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include grvctra.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include sounding_barnes_1.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include vad_barnes.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include findppi.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include taiwansd.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include taiwanradar.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include taiwansurf.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include grads_write.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include grads_fcst_write.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include grads_sdint.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include grads_write_pu.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include cmic_ice.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include coneva_ice.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include dtdqc_ice.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include miccon_ice.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include buoya_ice.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include miccon_ice_ta.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include coneva_ice_ta.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include tguess_ice.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include mkter.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include terrain.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include grads_fcst_write_terrain.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include grads_write_au.f
pgf90 -mcmodel=medium -Msave -fast -mp -Minform=inform -c -I/usr/local/rap/include -I/usr/local/rap/include grads_write_ausfc.f
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE -c -I/usr/local/rap/include -I/usr/local/rap/include Params.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE -c -I/usr/local/rap/include -I/usr/local/rap/include f_system.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE -c -I/usr/local/rap/include -I/usr/local/rap/include get_spdb_surface.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE -c -I/usr/local/rap/include -I/usr/local/rap/include get_spdb_sounding.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE -c -I/usr/local/rap/include -I/usr/local/rap/include adjoint.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE -c -I/usr/local/rap/include -I/usr/local/rap/include Main.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE -c -I/usr/local/rap/include -I/usr/local/rap/include find_latest_data.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE -c -I/usr/local/rap/include -I/usr/local/rap/include read_mdv.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE -c -I/usr/local/rap/include -I/usr/local/rap/include mdv_write.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE -c -I/usr/local/rap/include -I/usr/local/rap/include mdv_write_fcst.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE -c -I/usr/local/rap/include -I/usr/local/rap/include find_mdv_in_interval.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE -c -I/usr/local/rap/include -I/usr/local/rap/include find_mdv_in_interval_forecast.cc
pgc++ -mp -mcmodel=medium -DPGI_IN_USE -mcmodel=medium -DLINUX_IL6 -DBSD -D_BSD_SOURCE -D_SVID_SOURCE -c -I/usr/local/rap/include -I/usr/local/rap/include procmap_register.cc
echo Linking C++ program ...
Linking C++ program ...
LD_LIBRARY_PATH=/work3/myclo/pgi/linux86-64/17.4/lib::/opt/pgi8/linux86-64/8.0/lib:/opt/pgi8/linux86-64/8.0/libso:/work3/myclo/vapor-2.6.0/lib::
/bin/rm -f adjoint_moist
pgc++ -mp -o adjoint_moist \
	 	  adjointmain.o adv.o adv2.o adj.o advect.o bdya1.o bdya2.o coneva.o coneva1.o calc_RF_Factor.o cmic.o diffa.o diffp.o diffus.o direct.o dtdqc.o ecalc.o eigenx.o eigeny.o eigenz.o fltall.o force.o func.o funcgr.o grd.o hpder.o init.o linrg.o rev2.o setg.o setup.o grdchk.o xder.o yder.o zder.o lbfgs.o grdchkp.o bdycd2.o fill3d.o ssmo.o miccon.o buoy.o pgfa.o pgfd.o buoya.o convert_time.o varia.o surface_interpolation.o pints.o lnblnks.o get_ascii_metars.o vad.o calc_background.o ranf.o fill2d.o calcbkg.o pot2d.o sounding_barnes.o sounding_barnes_tq.o surface_error.o filt1d.o read_profiler.o readradarN.o rf.o rfilter_1d.o rfilter_1d_adj.o rfilter_3d.o pre_analyze.o dualdoppler.o sounding_vertical_interp.o tguess.o TransferTDRPints.o TransferTDRPreals.o TransferTDRPstrings.o model_sounding_vertical_interp.o readRAOB.o grvctr.o grvctra.o sounding_barnes_1.o vad_barnes.o findppi.o taiwansd.o taiwanradar.o taiwansurf.o grads_write.o grads_fcst_write.o grads_sdint.o grads_write_pu.o cmic_ice.o coneva_ice.o dtdqc_ice.o miccon_ice.o buoya_ice.o miccon_ice_ta.o coneva_ice_ta.o tguess_ice.o mkter.o terrain.o grads_fcst_write_terrain.o grads_write_au.o grads_write_ausfc.o Params.o f_system.o get_spdb_surface.o get_spdb_sounding.o adjoint.o Main.o find_latest_data.o read_mdv.o mdv_write.o mdv_write_fcst.o find_mdv_in_interval.o find_mdv_in_interval_forecast.o procmap_register.o -L /work3/myclo/pgi/linux86-64/17.4/libso -L /work3/myclo/pgi/linux86-64/17.4/lib -L /usr/local/rap/lib -L /usr/local/rap/lib -lSpdb -lMdv -ldsserver -ldidss -leispack -lrapformats -ltoolsa -lrapmath -ldataport -ltdrp -lpgmp -lpgf90 -lpgf90_rpm1 -lpgf902 -lpgf90rtl -pgf90libs -lpgftnrtl -lnuma -lnspgc -lpgc -lm -lgcc -lc -lgcc  
make[2]: Leaving directory `/work3/myclo/VDRAS/VDRAS_ice_ter'
make[1]: Leaving directory `/work3/myclo/VDRAS/VDRAS_ice_ter'

```

Many thanks again.  
Yi-Chuan[/url]

---

<div class="post-metadata">

**Author:** ![tull](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@tull](https://forums.developer.nvidia.com/u/tull)\
**Post date:** [September 25, 2017, 4:33pm UTC](https://forums.developer.nvidia.com/t/openmp-threads-problem-on-intel-cpu/135418/7 "2017-09-25T16:33:32Z")

</div>

You are linking PGI 8.0 libs with PGI 17.4? Risky.  
  
  
Instead of -L/path/to/pgi/f90/libs -lpgi1lib -lpgi2lib etc  
just use

-pgf90libs

so the current pgf90 libs are linked with pgc++ in the correct  
versions and order.

Your use of -Msave and -mcmodel=medium suggests you are declaring large arrays statically, and you wish them all to be zero upon program entry. This makes for very long execution times as these huge  
arrays inside the executable are loaded and executed.

Better to dynamically create and initialize the big arrays - smaller and faster.  
  
  
  
dave
