MSI over PCIe

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

Hi,
trying to get an interrupt from a FPGA vie PCIe.

But first I have tried to look at the configuration of the NVidia bridge:

sudo lspci -vvv

00:02.0 PCI bridge: NVIDIA Corporation Device 10e6 (rev a1) (prog-if 00 [Normal decode])
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 388
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: 51000000-51ffffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: NVIDIA Corporation Device 0000
	Capabilities: [48] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	<b>Capabilities: [50] MSI: Enable- Count=1/2 Maskable- 64bit+</b>
		Address: 0000000000000000  Data: 0000
	Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
		Mapping Address Base: 00000000fee00000
	Capabilities: [80] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag+ RBE+
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #1, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd On, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet+ LinkState+
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [140 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=30us PortTPowerOnTime=70us
	Kernel driver in use: pcieport

I wonder what is the meaning of Capabilities: [50] MSI: Enable- Count=1/2 Maskable- 64bit+

while when I

cat /sys/bus/pci/devices/0000:00:02.0/msi_bus

I get the value 1

So I am confused whether the MSI from the bridge is enabled or not.

Any help will be appreciated.

2

MSI in root port/bridge is irrelevant here. Please check the same in endpoint (i.e. FPGA) device. BTW, I hope you are calling pci_enable_msi() API in your FPGA device’s device driver to enable MSI. If not, please follow documentation @ https://github.com/torvalds/linux/blob/master/Documentation/PCI/MSI-HOWTO.txt to enable MSI interrupts for your device in the respective driver

3

hi vidyas,

thanks for the reply.

For the FPGA I got:

MSI: <b>Enable+</b> Count=1/1 Maskable- 64bit+

so I guess it is OK.

Though I could not follow the MSI-HOWTO document because I could not find the following:
pci_alloc_irq_vectors(), PCI_IRQ_MSI definition (in the whole kernel).

So I had to use the deprecated function pci_enable_msi() and the request_irq() function.

Is it the correct approach?

Does the interrupt configuration affected by the fact that the system is in configuration #3, that is 3 PCIe nodes? (I can see in the file tegra186-soc-base.dtsi some MSI related configurations)

Regards.

4

Can anyone in NVidia support me on this one?
thanks.

5

>> Is it the correct approach?
Yes. Since this is an old kernel, using those APIs should be fine

Now, what is the issue you are observing exactly? If you are not receiving the MSI interrupt on host, then, this needs to be looked into from FPGA side initially. Because host and its associated SW have been verified to work with various PCIe endpoint devices that generate and use MSI interrupts to communicate with the host.

6

Vidyas, thanks.

and what about this: Does the interrupt configuration affected by the fact that the system is in configuration #3, that is 3 PCIe nodes? (I can see in the file tegra186-soc-base.dtsi some MSI related configurations) ?

7

Nope. It shouldn’t.

8

I have printed the /proc/interrupts table:

...
452:          0          0          0          0  Tegra PCIe MSI   0 Edge      eth1
453:         40          0          0          0  Tegra PCIe MSI   1 Edge      eth1-TxRx-0
454:         40          0          0          0  Tegra PCIe MSI   2 Edge      eth1-TxRx-1
455:         40          0          0          0  Tegra PCIe MSI   3 Edge      eth1-TxRx-2
456:         40          0          0          0  Tegra PCIe MSI   4 Edge      eth1-TxRx-3
<b><u>457</u>:          0          0          0          0  Tegra PCIe MSI   <u>5</u> Edge      mydriver</b>
...

The 457 is the irq value I get during registration. And I guess that the value 5 is one of the 32 MSI vectors ( 0->31). I have a register in the FPGA which supposed to generate an interrupt to the MSI vector I write to this register. I have tried all 32 values non of which has provided an interrupt.

Any ideas?

9

This seems to be an issue specific to your FPGA device. Please connect a generic PCIe device (like a NIC or USB3.0 add-on card) and see if MSI interrupts work with that respective card. If they do, then you need to look into either your driver code or FPGA manuals for further debugging. In case if MSI interrupts don’t work with any off-the-shelf cards, do let us know, and we would debug it

10

I have an Ethernet controller connected to another PCIe port and it seems that during browsing I see the number of interrupts increase (by printing out cat /proc/interrupts).

I’ll close this issue once I realize what is the root cause of the problem.

Thanks.

1 Like
11

All,

I had to invoke the function pci_set_master() in order to receive any MSI interrupts from the FPGA. I was under the impression that the FPGA is the one responsible for setting this bit in the configuration.
This is how it was implemented:

if (pci_is_enabled(dev))
{
    if (!dev->is_busmaster)
        pci_set_master(dev);
}

Regards.