Setting MPI Processor/Core Affinity

Accelerated Computing HPC Compilers Legacy PGI Compilers
1

Is there a way to set core/processor affinity with PGI’s default MPICH setup?

That is, I have a dual-Nehalem system so two, quad-core chips. If I request ‘-np 4’ I’d like to know if I’m getting 4 cores on one processor or 2 cores on 2 processors and how I can select that.

Thanks,
Matt

2

Hi Matt,

The order is determined by the order in which the machines appear in the machines.LINUX file.

This would round-robin:

node01
node02
node01
node02
node01
node02
node01
node02

This would set the first two on node01, and the second two on node2

node01:2
node02:2
node01:2
node02:2

This would set the first four on node01, and the next four on node02:

node01:4
node02:4
  • Mat
3

Okay. Couple of questions. First, let’s say my machine’s short hostname is “teslabox”. My /opt/pgi/linux86-64/2010/mpi/mpich/share/machines.LINUX file is:

teslabox

Just one word. What does this mean? Round-robin?

And, is there anyway to dynamically change the order? I’m pretty sure Open MPI and MVAPICH have ways to set this via an environment variable or commandline switch?

4

Hi Matt,

Sorry, I mis-read your OP. You have a single 2-socket system, not 2 1 socket systems.

Setting single system core affinity is a bit more difficult since this is done by the OS not mpirun. What I have done in the past is write a wrapper script which sets the affinity (via taskset or numactl) then run the application. mpirun then runs the wrapper instead of the application.

The problem is knowing the rank of each individual process. In MPICH2, mpiexec sets the environment variable “PMI_RANK” which can be used, but I don’t think MPICH has an equivalent.

Here’s a simple Perl example. Note that PMI_SIZE is set to the total number of MPI processes and can be used to create more complex affinity bindings.

% cat penv.pl
#!/usr/local/bin/perl -w

use strict;

my $cmd = "numactl --physcpubind=$ENV{PMI_RANK} ";
my $arg = "";
while ($arg = shift) {
  $cmd .= $arg . " ";
}
exec($cmd);

% mpiexec -np 4 penv.pl date
Tue Jul 13 11:07:10 PDT 2010
Tue Jul 13 11:07:10 PDT 2010
Tue Jul 13 11:07:10 PDT 2010
Tue Jul 13 11:07:10 PDT 2010



I’m pretty sure Open MPI and MVAPICH have ways to set this via an environment variable or commandline switch?

I think so as well, but I’m not sure of the details.

  • Mat