Q3400_RA Duplicated LID errors in OpenSM log

Infrastructure & Networking Switches and Gateways InfiniBand/VPI Switch Systems
1

Hello,

During the deployment of Infiniband XDR Q3400_RA switches, I noticed the following messages in the opensm.log file:

...
Jul 2115:18:27630245 [DF2F7700] 0x01-> hca_add_port: ERR AB32: Duplicated LID forCA GUID: 0xac543098000e09cf
Jul 2115:18:27630248 [DF2F7700] 0x01-> hca_add_port: ERR AB32: Duplicated LID forCA GUID: 0xac3ae203000e09cf
Jul 2115:18:27630250 [DF2F7700] 0x01-> hca_add_port: ERR AB32: Duplicated LID forCA GUID: 0xac3ae203000e09cf
Jul 2115:18:27630254 [DF2F7700] 0x01-> hca_add_port: ERR AB32: Duplicated LID forCA GUID: 0xac9871230003f7cf
Jul 2115:18:27630257 [DF2F7700] 0x01-> hca_add_port: ERR AB32: Duplicated LID forCA GUID: 0xac9871230003f7cf
Jul 2115:18:27630260 [DF2F7700] 0x01-> hca_add_port: ERR AB32: Duplicated LID forCA GUID: 0xac9871230003f7cf
...

As I can see, similar log entries appear for all the nodes in the cluster. Each node has the same LID (57 in this case) for all four planes within a single switch, but with different port numbers (1–4):

kitaro@nrg-gpuib17-1-osm1:~$ sudo iblinkinfo | grep 0xac3ae2030003f7ff
0xac9871230003f7cf 57 1[ ] ==( 1X 212.5 Gbps Active/ LinkUp)==> 90 128[ ] "MF0;leaf1:Q3400_RA/U1" ( )
0xac9871230003f7cf 57 2[ ] ==( 1X 212.5 Gbps Active/ LinkUp)==> 91 128[ ] "MF0;leaf1:Q3400_RA/U2" ( )
0xac9871230003f7cf 57 3[ ] ==( 1X 212.5 Gbps Active/ LinkUp)==> 92 128[ ] "MF0;leaf1:Q3400_RA/U3" ( )
0xac9871230003f7cf 57 4[ ] ==( 1X 212.5 Gbps Active/ LinkUp)==> 93 128[ ] "MF0;leaf1:Q3400_RA/U4" ( )

I am using the prism_an_lmc 2 option in opensm.conf. If I understand correctly, this should result in four different LIDs for the prism switch. However, this does not seem to be happening.

The other options related to LMC are set to their default values:

lmc 0
lids_per_rtr 0
lmc_esp0 FALSE

The version of the OpenSM is 5.25.1

My main question is: is it acceptable to have such log entries? If not, I would appreciate any suggestions on how to fix this issue.

Thanks in advance,

Roman

2

In a Q3400-RA XDR fabric, a single endpoint can appear across multiple switch planes, so seeing related connectivity through plane labels such as U1 through U4 is expected behavior for this platform.

If OpenSM reports messages such as hca_add_port: ERR AB32: Duplicated LID for CA GUID: 0x…, those messages should not be ignored automatically. At the same time, based on the currently available information, they should be treated as a prompt to check XDR-specific plane-aware behavior rather than as proof of a hardware fault or a definite fabric failure.

In the output provided, the same endpoint GUID was observed on four plane-facing ports connected to Q3400_RA/U1 through Q3400_RA/U4, while the visible base LID remained the same across those plane representations.

That observation is consistent with the fact that Q3400-RA is a multi-plane XDR system. XDR documentation material describes plane-aware behavior, including plane binding, APort selection, HCA plane filtering, and Prism Aggregation Node LID handling across planes.

Important to note:
prism_an_lmc is an XDR/Prism-specific capability used for Prism Aggregation Node LID assignment across planes, and it is distinct from the standard OpenSM lmc setting. Because of that distinction, standard tools may not always present plane-aware addressing in the same way that ordinary lmc-based LID expansion would appear. For this reason, the key validation point is not simply whether iblinkinfo shows four different visible LIDs, but whether the XDR fabric is behaving correctly across all planes and whether the relevant plane-aware state is being programmed as expected.

So a multi-plane topology itself is normal for Q3400-RA. A single endpoint being represented across U1 through U4 is aligned with expected XDR architecture showing plane-related representations of the same connected entities.

The duplicate-LID log entries should be treated as something to validate, but the available data does not necessarily support that they are always an abnormal final state in every Q3400-RA XDR deployment. A safer assumption is that OpenSM’s generic duplicate-LID logic may be encountering planarized representations and that XDR-aware validation is required before concluding there is a defect.

Check the active OpenSM configuration and confirm that the intended XDR-related settings are present at runtime. This check is recommended because XDR validation coverage explicitly includes Prism AN LMC, HCA plane filtering, APort checks, and SMDB validation for XDR fabrics.

For example:
grep -E ‘^(lmc|prism_an_lmc|reassign_lids|honor_guid2lid_file)’ /path/to/opensm.conf

It is also recommended to validate the overall fabric using XDR-aware diagnostics, including OpenSM logs and database outputs, ibdiagnet, and APort or plane-consistency checks where available.

If the duplicate-LID messages continue, please open an Nvidia support case for a deeper look. Useful data to collect includes the following, which can confirm whether the issue is limited to log interpretation or reflects a broader plane-consistency or routing problem in the XDR fabric:
UFM dump if available
OpenSM startup log through fabric bring-up
Active opensm.conf
SMDB output
ibnetdiscover output
ibdiagnet logs and reports
Output showing the affected endpoint across all visible planes

3

Thanks a lot for your answer.

I use the following parameters in the opensm.conf file:

grep -E '^(lmc|prism_an_lmc|reassign_lids|honor_guid2lid_file)' /etc/opensm/opensm.conf
lmc 0
lmc_esp0 FALSE
reassign_lids FALSE
honor_guid2lid_file FALSE
prism_an_lmc 2

Also, you mentioned XDR documentation materials — did you mean a dedicated guide? What is its name?