Orin and PCIe multicast

[mruijter2]

I have been trying to get PCIe multicast to work with 3 Orins.

The setup looks like this:
Orin as Root Complex => |=> Orin Endpoint1
=>Orin Endpoint2

I also tested the exact same thing with Intel hosts as root:
Intel PC as RC => | => Orin Endpoint1
\ => Orin Endpoint2

When I use an Intel system (multiple tested) PCIe multicast works.
When we replace the Intel with an Orin multicast fails.

I can share the details and even the tools used to enable PCIe multicasting should someone be interested.

The question for the PCIe Tegra driver developer(s) is if they are aware of any problem or not yet implemented feature that prevents PCIe MC from working.

Something I noticed that is different is that Intel PCs show a single PCIe domain while the Orin will place the BCM bridges of the PCIe expander cards in a new domain: 0005:00:00.0
I don’t know for sure if this causes the problem but I think it’s unlikely.

Why PCIe multicast works with any PC and not with Orin is the question.

[kayccc]

The Orin HW doesn’t support multicast and P2P. So, your setup doesn’t work.

[mruijter2]

What we have working is this:

Intel PC -> One Stop Systems 580 PCIe expander <-> OSS 580 expander <-> Orin1 EP
                                               <-> OSS 580 expander <-> Orin2 EP

The Orin endpoints have a multicast overlay bar assigned.

For example, when the Orin endpoint looks like this:

lspci -PP -vv -s 0a:00.0 | head -11
0000:00:00.0/01:00.0/02:00.0/03:00.0/04:18.0/06:00.0/07:00.0/08:00.0/09:00.0/0a:00.0 RAM memory: NVIDIA Corporation Device 0001
	Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 255
	Region 0: Memory at 472000000 (32-bit, non-prefetchable) [size=16M]
	Region 2: Memory at 400000000 (64-bit, prefetchable) [size=128K]
	Region 4: Memory at 473000000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+

Bridge 09:00 which is the last bridge that connects the Orin has multicast configured with an overlay bar:

lspci -PP -vv -s 09:00.0 | grep Mcast
		McastCap: MaxGroups 64, ECRCRegen+
		McastCtl: NumGroups 1, Enable+
		McastBAR: IndexPos 12, BaseAddr 0000000470000000
		McastReceiveVec:      0000000000000001
		McastBlockAllVec:     0000000000000000
		McastBlockUntransVec: 0000000000000000
		McastOverlayBAR: OverlaySize 16 (65536 bytes), BaseAddr 0000000472000000

Multicast traffic that is written to 0x470000000 will be sent to all bridges that have multicast enabled after which it arrives on Orin due to it automatically being copied to the overlay. This works well, as long as the root complex is using an Intel based system.

We noticed yesterday that any ARM64 based system used as root complex fails. So it seems to be related to the ARM64 platform. We are still investigating why this doesn’t seem to work with ARM64 systems. Theoretically it should work just like it works with an Intel based RC since most of the heavy lifting is done by the bridge chips themselves.