Mellanox OFED QoS and Rate-limit

Hi,

I’m working on a testing environment where I want to test the QoS functionality in Mofed.

I do think that I have a good comprehension of Basic InfiniBand QoS (SL and VL), and I try to understand Advanced QoS by defining policies (/etc/opensm/qos-policy.conf), and especially the rate-limit functionality.

I basically have 2 groups (One EDR, one FDR) and I want to restrict the bandwidth when one node of one group is communicating with a node of the other group.

Here is my qos-policy.conf :

port-groups

port-group

name: nodeFDR

port-guid: 0x0002c9030032ac41, 0x0002c9030032ac70, 0x0002c9030032acb0, 0x0002c9030032acc0, 0x0002c9030032af00

end-port-group

port-group

name: nodeEDR

port-guid: 0xe41d2d0300f29d70, 0xe41d2d0300f29eac, 0xe41d2d0300f29fb4

end-port-group

end-port-groups

qos-levels

qos-level

name: DEFAULT

use: default QoS Level

sl: 5

rate-limit: 16

end-qos-level

qos-level

name: SDR1X

mtu-limit: 4

rate-limit: 2

sl: 2

end-qos-level

qos-level

name: QDR4X

mtu-limit: 4

rate-limit: 6

sl: 4

end-qos-level

end-qos-levels

qos-match-rules

qos-match-rule

source: nodeEDR

destination: nodeFDR

qos-level-name: SDR1X

end-qos-match-rule

qos-match-rule

source: nodeFDR

destination: nodeEDR

qos-level-name: QDR4X

end-qos-match-rule

end-qos-match-rules

So far so good, opensm seems to be okay with all that. Also When I do an saquery pathrecord, everything seems to be ok (I’m using ibtool from python-rdma) :

ibtool saquery PathRecord --sgid=fe80::e41d:2d03:f2:9fb4 --dgid=fe80::2:c903:32:ac41

PathRecord dump:

service_id…0x0000000000000000

dgid…fe80::2:c903:32:ac41

sgid…fe80::e41d:2d03:f2:9fb4

dlid…6

slid…15

RawTraffic…0

Reserved_353…0

FlowLabel…0

HopLimit…0

tclass…0x00

Reversible…1

NumbPath…0

pkey…0xffff

qos_class…0x000

sl…2

MTUSelector…2

mtu…4

RateSelector…2

rate…2

PacketLifeTimeSelector…2

pkt_life…18

preference…0

ReversePathPKeyMemberBit.0

Reserved_466…0

Reserved_480…0

As expected, I don’t have the same result for the reverse path :

ibtool saquery PathRecord --dgid=fe80::e41d:2d03:f2:9fb4 --sgid=fe80::2:c903:32:ac41

PathRecord dump:

service_id…0x0000000000000000

dgid…fe80::e41d:2d03:f2:9fb4

sgid…fe80::2:c903:32:ac41

dlid…15

slid…6

RawTraffic…0

Reserved_353…0

FlowLabel…0

HopLimit…0

tclass…0x00

Reversible…1

NumbPath…0

pkey…0xffff

qos_class…0x000

sl…4

MTUSelector…2

mtu…4

RateSelector…2

rate…6

PacketLifeTimeSelector…2

pkt_life…18

preference…0

ReversePathPKeyMemberBit.0

Reserved_466…0

Reserved_480…0

Starting from here, I’m running benchmarks (both ib_send_bw and ib_write_bw) but I’m not speed limited. I’m running wirespeed.

Am I missing something here ?

Thank you,

Jérôme