problem with compiling OpenMP on pgf90 13.5-0

Accelerated Computing HPC Compilers Legacy PGI Compilers
1

Hi All,

I have a rather lengthy FORTRAN-77 code that compiles just fine using pgf90, using this version:

pgf90 13.5-0 64-bit target on x86-64 Linux -tp piledriver
Copyright 1989-2000, The Portland Group, Inc. All Rights Reserved.
Copyright 2000-2013, STMicroelectronics, Inc. All Rights Reserved.

I recently added some OpenMP directives to parallelize a loop, now I can’t compile the code:

pgf90 -Mextend -O2 -mcmodel=medium -tp piledriver -mp -o fred fred.for

The output is from this process is

/tmp/pgf902x2kIv4wst-z.s: Assembler messages:
/tmp/pgf902x2kIv4wst-z.s:29455: Error: junk $-2411111144' after expression /tmp/pgf902x2kIv4wst-z.s:29455: Error: number of operands mismatch for vfmsubsd’
/tmp/pgf902x2kIv4wst-z.s:29456: Error: junk at end of line, first unrecognized character is %' /tmp/pgf902x2kIv4wst-z.s:29461: Error: junk $-2411111136’ after expression
/tmp/pgf902x2kIv4wst-z.s:29461: Error: number of operands mismatch for vfmaddsd' /tmp/pgf902x2kIv4wst-z.s:29462: Error: junk at end of line, first unrecognized character is %’
/tmp/pgf902x2kIv4wst-z.s:30069: Error: junk $-2411111064' after expression /tmp/pgf902x2kIv4wst-z.s:30069: Error: number of operands mismatch for vfmsubsd’
/tmp/pgf902x2kIv4wst-z.s:30070: Error: junk at end of line, first unrecognized character is %' /tmp/pgf902x2kIv4wst-z.s:30075: Error: junk $-2411111056’ after expression
/tmp/pgf902x2kIv4wst-z.s:30075: Error: number of operands mismatch for vfmaddsd' /tmp/pgf902x2kIv4wst-z.s:30076: Error: junk at end of line, first unrecognized character is %’
/tmp/pgf902x2kIv4wst-z.s:33020: Error: junk $-2373976184' after expression /tmp/pgf902x2kIv4wst-z.s:33020: Error: number of operands mismatch for vfmaddsd’
/tmp/pgf902x2kIv4wst-z.s:33021: Error: junk at end of line, first unrecognized character is (' /tmp/pgf902x2kIv4wst-z.s:33026: Error: junk $-2373976184’ after expression
/tmp/pgf902x2kIv4wst-z.s:33026: Error: number of operands mismatch for vfmaddsd' /tmp/pgf902x2kIv4wst-z.s:33027: Error: junk at end of line, first unrecognized character is (’
/tmp/pgf902x2kIv4wst-z.s:33038: Error: junk $-2373976184' after expression /tmp/pgf902x2kIv4wst-z.s:33038: Error: number of operands mismatch for vfmaddsd’
/tmp/pgf902x2kIv4wst-z.s:33039: Error: junk at end of line, first unrecognized character is (' /tmp/pgf902x2kIv4wst-z.s:33456: Error: junk $-2373976120’ after expression
/tmp/pgf902x2kIv4wst-z.s:33456: Error: number of operands mismatch for vfmsubsd' /tmp/pgf902x2kIv4wst-z.s:33457: Error: junk at end of line, first unrecognized character is %’
/tmp/pgf902x2kIv4wst-z.s:33462: Error: junk $-2373976112' after expression /tmp/pgf902x2kIv4wst-z.s:33462: Error: number of operands mismatch for vfmaddsd’
/tmp/pgf902x2kIv4wst-z.s:33463: Error: junk at end of line, first unrecognized character is `%’

The code actually compiles and runs (in parallel!) using gfortran

gfortran -O2 -mcmodel=medium -ffixed-line-length-132 -fopenmp -o fred fred.for

The code is a MonteCarlo fitter, but the first iteration has no random numbers. The output from that first loop agrees between the pgf90 serial version, the gfortran serial version, and the gfortran parallel version.

We have some older linux boxes with older PGI versions. I can compile the above code using the older compilers with the -mp flag, but then the program hangs. These older boxes have gfortran versions that are out of date, and gfortan won’t work on those boxes.

Any ideas?

2

As an addendum to the above post, I have PGI version 14.6 installed on a similar system, and I get similar errors:

pgf90 --version

pgf90 14.6-0 64-bit target on x86-64 Linux -tp piledriver
The Portland Group - PGI Compilers and Tools
Copyright (c) 2014, NVIDIA CORPORATION. All rights reserved.



pgf90 -Mextend -O2 -mcmodel=medium -tp piledriver -mp=noalign -o fred fred.for
/tmp/pgf907nFnXSfClHOJ.s: Assembler messages:
/tmp/pgf907nFnXSfClHOJ.s:27721: Error: junk $-2411111144' after expression /tmp/pgf907nFnXSfClHOJ.s:27721: Error: number of operands mismatch for vfmsubsd’
/tmp/pgf907nFnXSfClHOJ.s:27722: Error: junk at end of line, first unrecognized character is %' /tmp/pgf907nFnXSfClHOJ.s:27725: Error: junk $-2411111136’ after expression
/tmp/pgf907nFnXSfClHOJ.s:27725: Error: number of operands mismatch for vfmaddsd' /tmp/pgf907nFnXSfClHOJ.s:27726: Error: junk at end of line, first unrecognized character is %’
/tmp/pgf907nFnXSfClHOJ.s:28185: Error: junk $-2411111064' after expression /tmp/pgf907nFnXSfClHOJ.s:28185: Error: number of operands mismatch for vfmsubsd’
/tmp/pgf907nFnXSfClHOJ.s:28186: Error: junk at end of line, first unrecognized character is %' /tmp/pgf907nFnXSfClHOJ.s:28189: Error: junk $-2411111056’ after expression
/tmp/pgf907nFnXSfClHOJ.s:28189: Error: number of operands mismatch for vfmaddsd' /tmp/pgf907nFnXSfClHOJ.s:28190: Error: junk at end of line, first unrecognized character is %’
/tmp/pgf907nFnXSfClHOJ.s:31194: Error: junk $-2373976120' after expression /tmp/pgf907nFnXSfClHOJ.s:31194: Error: number of operands mismatch for vfmsubsd’
/tmp/pgf907nFnXSfClHOJ.s:31195: Error: junk at end of line, first unrecognized character is %' /tmp/pgf907nFnXSfClHOJ.s:31198: Error: junk $-2373976112’ after expression
/tmp/pgf907nFnXSfClHOJ.s:31198: Error: number of operands mismatch for vfmaddsd' /tmp/pgf907nFnXSfClHOJ.s:31199: Error: junk at end of line, first unrecognized character is %’

I had tried the few options available with the -mp flag, and the end result is always the same.

3

Hi JeromeWinchell71787,

Most likely you have an older assembler which doesn’t recognize AVX instructions. (Use “as --version” to see what version you have.) We generate AVX by default for Piledriver architectures.

You’ll need to upgrade your binutils or target an older architecture, ex. “-tp=barcelona”

Hope this helps,
Mat

4

Hi Mat,

Here is the assembler version:

as --version
GNU assembler (GNU Binutils; openSUSE 12.3) 2.23.1
Copyright 2012 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or later.
This program has absolutely no warranty.
This assembler was configured for a target of `x86_64-suse-linux’.

I tried to update the binutils, but they are apparently up to date.

I was able to compile using the -tp barcelona-64 flag, but the code just hangs at run time.

By the way, I posted both messages, but my profile name on the second post was swapped out for some other profile.

5

Hmm, I’m not sure then. Can you post or send to PGI Customer Service (trs@pgroup.com) a reproducing example?

Thanks,
Mat

6

Hi Mat,

As I noted earlier, we have a similar system that is a bit newer. It has the same type of CPUs but fewer of them. I was able to compile using the -tp barcelona flag, but not the pilerdriver flag (the compiler version is 14.6). This code runs in parallel on the older system, although it was necessary to update the PGI libraries as libpgmp.so was missing. The PGI-compiled executable is about 1.5 times faster than the gfortran version, with both running on 42 cores. This is why I am interested in getting the PGI version working…

I would like to sort this out, so I will make the code available to the Customer Service.

Thanks,

Jerry

7

Hi Jerry,

Looks like we’re generating bad FMA instructions, hence another work around is to disable FMA via the “-Mnofma” flag. I’ve added an issue report (FS#20632 - User code gets asm error due to bad FMA code generation) and sent it on engineering.

Thanks,
Mat

8

Hi Mat,

I have made some modest changes to the code. Along the way I moved a few variables from “first private” directives to “shared” directives, and sacrificed a few goats. I can now compile the code on our older piledriver system using PGI 13.5:

pgf90 -tp piledriver -mp -fast -Mextend -mcmodel=medium -o fred fred.for

It compiles without comment or complaint, and runs almost twice as fast (in parallel!) as the gfortran version.


Jerry