lustre, mpi and qos

Hello all

I want to setup qos for lustre and MPI.

How I can setup it ?

Lustre

As I understand, you can configure in two ways:

  1. The link http://www.spinics.net/lists/linux-rdma/msg02150.html http://www.spinics.net/lists/linux-rdma/msg02150.html

recommended use like this:

in /etc/opensm/qos-policy.conf

qos-ulps

default :0 # default SL (OPENMPI or MPI)

lustre, service-id 0x : 1 # Lustre

end-qos-ulps

write to /etc/modprobe.d/lustre.conf

options ko2iblnd port_space=339

Where I can get service-id or how I can refer to lustre traffic in qos-policy.conf and setup it your own SL ?

I do not want to use “targetportguid”. It so direct.

What is port_space=339 ? Why 339 or where i can get it ? How port_space connected to lustre and qos ?

  1. Can I direct setup lustre (OSS and MDS) your own SL ?

May be it possibly in Environment variable or may be exist other ways ?

MPI

If I using MVAPICH2 How I can setup SL for MPI traffic ? Is I can do it by Environment variable or may be exist other ways ?

Sorry for my english

important note: for lustre accept config in qos_policy.conf you will need umount lustre, unload lustre drivers, configure qos_policy, restart opensm and mount lustre.

Hi Serge,

you can find an example for implementing what you need on the Mellanox OFED user guide: http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_User_Manual_v4.0.pdf http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_User_Manual_v4.0.pdf , QoS chapter 9.7

I found answers

MPI - only for MVAPICH2 [mvapich-discuss] MPI with service level http://mailman.cse.ohio-state.edu/pipermail/mvapich-discuss/2010-June/002882.html . We can use some enviroments

mpirun_rsh -np <num_procs> <host_list>

MV2_DEFAULT_SERVICE_LEVEL=

Lustre: How I can understand work only qos_policy reference with target GUID.

If anybody know other method for lustre then ping me.

Thanks all

I read this manual. Sorry but I cant found answer in this manual and I cant found in this manual the chapter “QoS chapter 9.7”. Manual have only six chapters.

In 3.2.2.7 page 154 QoS Configuration Examples

"MPI is assigned an SL via the command line

host1# mpirun –sl 0". I cant found this option in my mvapich2 2.1.

In 3.2.2.7 page 154 QoS Configuration Examples recommendation to setup qos-ulps that

"qos-ulps

default

:0 # default SL (for

MPI)

any, target-port-guid OST1,OST2,OST3,OST4:1 # SL for Lustre OST

any, target-port-guid MDS1,MDS2 :2 # SL for Lustre

MDS

end-qos-ulps"

But it setup by target-port-guid. It so direct issues. I want to be use service-ID or some like this. More flexible.

I also cant found about Lustre and Service-ID or some other ways to setup Lustre your SL in this manual.

But still thanks for answer