OpenMP thread affinity

Accelerated Computing HPC Compilers Legacy PGI Compilers
1

Hi.

When executing PGI compiled OpenMP programs, the n’th thread gets its affinity set to the n’th cpu. How do I disable this?

Thanks,

Casper Kirkegaard

2

Hi Casper,

The PGI run time does not set any thread affinity unless the user sets the environment variable “MP_BIND” to “yes”. When MP_BIND is enabled, a second environment variable “MP_BLIST” sets the affinity map. For example “MP_BLIST=1,3,5,7,0,2,4,6” will map thread 0 to core 1, thread 2 to core 3, etc.

Hope this helps,
Mat

3

Thanks Mat.

After I got your reply I spent quite some time looking through the documentation and google, but I can’t figure out how to set these variables from within my code? (PVF 9)

Casper

4

Hi Casper,

If you are running you application from the DOS command line, you would set these environment variables using the “SET” command, example: “set MP_BIND=yes”.

From within PVF, you can set environment variables by selecting the projects property page’s 'Debugging" option and look for the “environment” field. In this field, add each variable separated by a semi-colon. Example: “OMP_NUM_THREADS=4;MP_BIND=yes;MP_BLIST=1,3,0,2”.

  • Mat
5

BTW, Annemarie asked if 9.0 solved your debugging issue? HERE

If not, can you send PGI customer service (trs@prgoup.com) an example code that illustrates the the problem?

Thanks,
Mat

6

Hi again Mat

I postponed solving the bind issue since my code was still in the test phase , however, im getting there and now I would like to disable the MP_bind feature … I have the issue with both version 8 and 9 of PVF.

I tested several different example codes and executables on at least 5 different Windows machines and my conclusion is that each thread has its affinity set to one cpu by default. Setting MP_BIND=NO does not change anything, the affinity mask is still set. I tested on machines running XP 32 bit, XP 64 bit and server 2003.

Here is what I have done:

Compiled a very simple version of my code with NUM_THREADS=1.

Opened two commands prompts

“set MP_BIND=NO” and then executed my application in each prompt.

In taskmanager I can now see that both instances are running on cpu 1 only - ie. only at half speed. If I use task manager to set the affinity for the processes to use all available cpu’s it works just fine …

It seems like a very general problem to me, but I can e-mail you the code if you like? EDIT: I e-mailed the code to trs@pgroup.com

Regards,

Casper

7

Thanks Casper. Either Dave or I will take a look.

  • Mat
8

Hi again

Do you have an update on this issue?

The problem persists in PVF 10. It also still makes no difference if I change the MP_BIND environment variable …

Regards,

Casper

9

UPDATE:

The issue is partially solved in PVF 10.6. I have primarily tested it running 2 instances of two single thread openMP compiled programs, which is why I did not notice this earlier.

It is only the master thread that has affinity set, other threads does not. When i run two identical instances of my code at the same time it will run at half speed during the sequential part, because both instances have their master threads tied to the same cpu. During the parallel parts the code runs at full speed for all threads, except for the workload handled by the two masterthreads. Is this something you can fix in a future release?

Regards,

Casper

10

Hi Casper,

I glad to see that there has been progress. As for the new issue, what is MP_BLIST set to? When MP_BIND is set to ‘yes’, the runtime will default to binding to 0,1,2,3,… etc. So if two executable are run each using the default binding, I can see how they would get bound to the same CPU. The variable MP_BLIST overrides the default binding order. Try setting MP_BLIST=0,1,2,3 for one exe and MP_BLIST=4,5,6,7 for the other to see it if fixes the problem.

Hope this helps,
Mat

11

Hi Matt

MP_BIND is set to no, but the affinity for the master thread still seems to be set. Manually setting MP_BLIST is not really an option, since many instances of the exe file is likely to be running at the same time on a multi user server. Do you have any other suggestions?

Regards,

Casper

12

Do you have an update on this issue?

It is still present in PVF 10.9 …

Regards,

Casper

13

Hi CasperK,

Just an FYI that I am looking it this but don’t have a definitive answer as to what’s wrong. Unfortunately, the engineer who’s in charge of the OpenMP runtime is out on vacation for a few weeks so I’m needing rely on another engineer who doesn’t know the code as well. Hence why it’s taking so long.

Thanks for your patience,
Mat