Windows RDMA QoS and WinOF-2 1.80 issues

Hi,

so I’m setting up Scale-Out file server on Windows Server 2016 (fully patched for this date). When I use WinOF-2 newer then 1.70 (tried 1.80.50 and 1.80.51) The QoS is not working properly.

I’m using MXC416A-BCAT with firmware 12.21.2010.

I get RDMA to work but “WinOF-2 Port QoS” indicate that incorrect priorities are used and traffic is marked with wrong DSCP values. Also Get-NetAdapterQos indicates some kind of a problem.

My configuration:

Set-NetAdapterAdvancedProperty -Name SMB1 -RegistryKeyword "*JumboPacket" -RegistryValue 4200 Set-NetAdapterAdvancedProperty -Name SMB1 -RegistryKeyword VlanID -RegistryValue "4029" Set-NetQosDcbxSetting -Willing 0 -Confirm:$false Enable-NetQosFlowControl -Priority 4,5 Disable-NetQosFlowControl -Priority 0,1,2,3,6,7 New-NetQosTrafficClass "SMB" -Priority 5 -BandwidthPercentage 95 -Algorithm ETS New-NetQosPolicy "SMBD" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 5 -Confirm:$false -Precedence 255 -NetworkProfile All New-NetQosPolicy "SMB" -SMB -PriorityValue8021Action 5 -DSCPAction 42 -Confirm:$false -Precedence 255 -NetworkProfile All New-NetQosPolicy "DEFAULT" -Default -PriorityValue8021Action 5 -DSCPAction 42 -Confirm:$false -Precedence 240 -NetworkProfile All New-NetQosPolicy "CLUSTER" -Cluster -PriorityValue8021Action 7 -DSCPAction 56 -Confirm:$false -Precedence 240 -NetworkProfile All New-NetQosPolicy "TCP" -IPProtocolMatchCondition TCP -PriorityValue8021Action 0 -DSCPAction 0 -Confirm:$false -Precedence 10 -NetworkProfile All New-NetQosPolicy "UDP" -IPProtocolMatchCondition UDP -PriorityValue8021Action 0 -DSCPAction 0 -Confirm:$false -Precedence 10 -NetworkProfile All Enable-NetAdapterQos -Name SMB1 Set-NetAdapterQos -Name SMB1 -Enabled $true $m=HKLM:\SYSTEM\CurrentControlSet\Control\Class\{4d36e972-e325-11ce-bfc1-08002be10318}\0001 new-itemProperty -Path $m -Name "RxTrustedState" -PropertyType "String" -Value "2" -Force new-itemProperty -Path $m -Name "TxUntagPriorityTag" -PropertyType "String" -Value "1" -Force new-itemProperty -Path $m -Name "PriorityToDscpMappingTable_7" -PropertyType "String" -Value "56" -Force new-itemProperty -Path $m -Name "PriorityToDscpMappingTable_5" -PropertyType "String" -Value "42" -Force new-itemProperty -Path $m -Name "PriorityToDscpMappingTable_4" -PropertyType "String" -Value "34" -Force Mlx5Cmd.exe -Qosconfig -Dcqcn -Name SMB1 -Set -DcqcnCnpDscp 58 -DcqcnCnp802pPrio 7 Mlx5Cmd.exe -Qosconfig -Dcqcn -Name SMB1 -Disable Mlx5Cmd.exe -Qosconfig -Dcqcn -Name SMB1 -Enable 4 Mlx5Cmd.exe -Qosconfig -Dcqcn -Name SMB1 -Enable 5 #Mlx5Cmd.exe -Qosconfig -Dcqcn -Name "SMB1" -Set –DcqcnCnpPrioMode 0 #btw. this is not working - Invalid command or flag, Restart-NetAdapter SMB Enable-NetAdapterRDMA SMB1

So I want to see SMB Direct on priority 5 with DSCP 42. However counters indicate priority 3 is used instead of 5.

So I’m checking:

PS C:\Windows\system32> Get-NetAdapterqos SMB1 Name : SMB1 Enabled : True Capabilities : Hardware Current -------- ------- MacSecBypass : NotSupported NotSupported DcbxSupport : IEEE IEEE NumTCs(Max/ETS/PFC) : 1/1/1 1/1/1

This is clearly wrong.

Then I reboot the server. And get the same result.

Then I check QoSPolicy:

PS C:\Windows\system32> Get-NetQosPolicy -PolicyStore "ActiveStore" | select elementname,name,owner,NetworkProfile,Precedence,IPProtocol,Template,PriorityValue,DSCPValue,DSCPAction,NetDirectPort | sort -Descending Precedence,elelementname | ft elementname name owner NetworkProfile Precedence IPProtocol Template PriorityValue DSCPValue DSCPAction ----------- ---- ----- -------------- ---------- ---------- -------- ------------- --------- ---------- livemigration livemigration Group Policy (Machine) All 255 None LiveMigration 4 34 34 smb smb Group Policy (Machine) All 255 None SMB 5 42 42 smbd smbd Group Policy (Machine) All 255 None None 5 -1 -1 cluster cluster Group Policy (Machine) All 240 None Cluster 7 56 56 default default Group Policy (Machine) All 240 None Default 5 42 42 tcp tcp Group Policy (Machine) All 10 TCP None 0 0 0 udp udp Group Policy (Machine) All 10 UDP None 0 0 0

So policy is ok. But traffic is using priority 3 and DSCP 26 - and these values are not configured anywhere!

When I use driver 1.70 priorites are used correctly.

Any one has any ideas what can I do to get priorities working correctly with current driver version ?

Hi,

  1. I do not have such issues with 1.70…, haven’t tried 1.90 yet, I will in a month.

  2. TxUntagPriorityTag - line 20 of my output.

  3. RxUntaggedMapToLossless - I’m using only vlan tagged traffic so no need to set it in my case.

  4. PriorityToDscpMappingTable - lines 21-23.

  5. DscpBasedEtsEnabled - I missed that one. However in 1.70 it works fine without it set.

  6. DscpForGlobalFlowControl - missed that also, but that is DSCP only for flow control, not for entire traffic.

  7. RxTrustedState - line 19

Hi,

Look at the User Manual for WinOF-2, page 69. You also need to configure some additional registry keys:

TxUntagPriorityTag If 0x1, do not add 802.1Q tag to transmitted packets which are assigned 802.1p priority, but are not assigned a non-zero VLAN ID (i.e. priority-tagged). Default 0x0, for DSCP based PFC set to 0x1. Note: These packets will count on the original priority, even if the registry is on.

RxUntaggedMapToLossless If 0x1, all untagged traffic is mapped to the lossless receive queue. Default 0x0, for DSCP based PFC set to 0x1.

PriorityToDscpMappingTable_ A value to mark DSCP for RoCE packets assigned to CoS=ID, when priority flow control is enabled. The valid values range is from 0 to 63, Default is ID value, e.g. PriorityToDscpMappingTable_3 is 3. ID values range from 0 to 7.

DscpBasedEtsEnabled If 0x1 - all DSCP based ETS feature is enabled, if 0x0 - disabled. Default 0x0.

DscpForGlobalFlowControl Default DSCP value for flow control. Default 0x1a.

RxTrustedState Default using host priority (PCP) is 1 Default using DSCP value is 2

Hi,

As I know DSCP configuration don’t require vlan tagged traffic.

Maybe this post can help you.

Recommended Network Configuration Examples for RoCE Deployment - new

Hi,

vlan tagged traffic is not required - but I do need it in a vlan

Sorry for late reply but it took me longer then expected to get back to this.

Anyway I think I solved this. Apparently after earlier learning experience I ended up with mismatch DCBX config between card and Windows. I had it enabled in firmware:

LLDP_NB_DCBX_P1

LLDP_NB_RX_MODE_P1

LLDP_NB_TX_MODE_P1

and disabled in windows.

After resetting card firmware settings to default (DCBX not listening to switch) it’s ok now even with 1.80 driver.

For resetting my card: mlxconfig.exe -d mt4115_pciconf0 reset

Thanks for suggestions.