Q3400_RA doesn't work hm_ports_health_policy_file

Infrastructure & Networking Switches and Gateways InfiniBand/VPI Switch Systems
1

Hello!

I’ve run into an issue with the hm_ports_health_policy_file option on a Q3400_RA InfiniBand switch. I’m using OpenSM 5.25.1.

I use the hm_ports_health_policy_file option to drain unhealthy ports:

root@srv1:~$ cat /etc/opensm/opensm.conf | grep opensm-health-policy.conf
hm_ports_health_policy_file /etc/opensm/opensm-health-policy.conf

The file /etc/opensm/opensm-health-policy.conf looks like this:

root@srv1:~$ cat /etc/opensm/opensm-health-policy.conf
0x1c34da03007fd911 50 unhealthy

This works fine for NDR InfiniBand switches. However, it doesn’t seem to work for the Q3400_RA. I can see that OpenSM reads the file /etc/opensm/opensm-health-policy.conf during the heavy sweep process, but the ports listed in the file don’t change their state to Init as they should.

Could you please clarify whether this behaviour is expected? If not, what could be wrong? Do I perhaps need to change some options in the opensm.conf file?

Thanks in advance,

Roman

2

This looks like expected behavior if you’re trying to use manual unhealthy-port policy to force the port to Init.

For manual entries in hm_ports_health_policy_file, the relevant action for switch ports is controlled by hm_sw_manual_action, and the documented default is ‘no_discover’, not ‘isolate’. That means manually marked switch ports are handled as “do not discover” ports, rather than being driven into the same Initialize behavior you may be seeing on other platforms.

hm_ports_health_policy_file is just the file pointer for the health policy.
Manual switch-port behavior is governed by hm_sw_manual_action.
The change is applied after sending OpenSM a HUP / reload signal.

What’s happening is that this line is being parsed, but for a switch port the manual action is likely still the default hm_sw_manual_action no_discover, so OpenSM is not putting that port into the Init state:

0x1c34da03007fd911 50 unhealthy

In opensm.conf, verify these:

hm_unhealthy_ports_checks TRUE
hm_ports_health_policy_file /etc/opensm/opensm-health-policy.conf
hm_sw_manual_action no_discover

If hm_sw_manual_action is unset, the default is no_discover per NVIDIA docs.

Also check the effective value of hm_sw_manual_action in the running config/logs. If you want behavior closer to drain/isolation, set the following and then reload OpenSM (pkill -HUP opensm):

hm_sw_manual_action isolate

3

Thank you for your response!

I have followed your recommendations and conducted a series of tests. Based on the results, I have concluded that the issue is not related to Q3400_RA, but rather appears to be caused by a problem with the OpenSM version.

Test 1: HDR cluster with OpenSM 5.17.2

I tested a legacy HDR cluster where manual isolation of links functions correctly. This cluster uses OpenSM version 5.17.2. The following configuration settings were verified:

root@srv1:~$ cat /etc/opensm/opensm.conf | grep -E 'hm_unhealthy_ports_checks|hm_ports_health_policy_file|hm_sw_manual_action'
hm_unhealthy_ports_checks TRUE
hm_ports_health_policy_file /etc/opensm/opensm-health-policy.conf
hm_sw_manual_action isolate

It is worth noting that manual drain functionality operates even when the hm_sw_manual_action parameter is set to no_discover, although the reason for this behaviour remains unclear.

The health policy configuration file contains the following entries:

root@srv1:~$ cat /etc/opensm/opensm-health-policy.conf
0x08c0ge0300e4eb01 22 unhealthy
0x08c0me0300eze626 4 unhealthy

Link status verification confirms proper operation:

root@srv1:~$ sudo iblinkinfo | grep Init
2 4[ ] ==( 4X 53.125 Gbps Initialize/ LinkUp)==> 7 22[ ] "sw1" ( )
7 22[ ] ==( 4X 53.125 Gbps Initialize/ LinkUp)==> 2 4[ ] "sw2" ( )

OFED version:

root@srv1~$ ofed_info | head -1
MLNX_OFED_LINUX-23.10-3.2.2.0 (OFED-23.10-3.2.2):

Test 2: Server with OpenSM 5.26.1

On another server within the same cluster, I installed a newer version of DOCA, which includes OpenSM 5.26.1. Despite using an identical configuration, the manual isolation functionality does not work as expected.

OFED version on the second server:

root@srv2:~$ ofed_info | head -1
OFED-internal-26.01-1.0.0:

Configuration settings (identical to Test 1):

root@srv2:~$ cat /etc/opensm/opensm.conf | grep -E 'hm_unhealthy_ports_checks|hm_ports_health_policy_file|hm_sw_manual_action'
hm_unhealthy_ports_checks TRUE
hm_ports_health_policy_file /etc/opensm/opensm-health-policy.conf
hm_sw_manual_action isolate

Health policy file (identical to Test 1):

root@srv2:~$ cat /etc/opensm/opensm-health-policy.conf
0x08c0ge0300e4eb01 22 unhealthy
0x08c0me0300eze626 4 unhealthy

During testing, it was observed that the OpenSM process reads the /etc/opensm/opensm-health-policy.conf file when the command pkill -HUP opensm is executed. However, no operational effect is observed despite successful file reading.

The findings suggest that the issue stems from a defect in OpenSM version 5.26.1. We encountered this problem when working with an XDR switch, as the legacy OpenSM version is incompatible with it. However, it appears that the same issue persists with HDR and NDR generations of InfiniBand when using the newer OpenSM version.

I think this might be a software bug — could NVIDIA run some tests in their lab to help us confirm it? Any help here would be awesome.