I have connected to the Jetson TX2 development kit a PCIe card with 4 serial RS-485 ports, based on chip EXAR XR17V354.
The board is properly detected by the kernel as per lspci command:
When it comes to receive interrupts, the driver gets several IRQs even if the register on the device responsible for the irq emission is cleared.
Few instants later, when it should be the time to receive new interrupts, the ISR isn’t triggered anymore.
This behavior is not present on X86 CPU.
I tried to play with interrupt flags and I have found that if I try to force edge interrupts the kernel reply with this message:
genirq: Flags mismatch irq 381. 00000081 (xrserial) vs. 00000084 (PCIE)
On X86 the message is different:
genirq: Flags mismatch irq 17. 00000081 (xrserial) vs. 00000080 (snd_intel8x0)
So on NVidia CPU interrupts related to PCIe are level HIGH, on X86 are NONE.
Could be this the reason why I receive spurious interrupts ?
How to configure the PCIe properly ?
Which interrupt is enabled by the device driver for this device? Is it Legacy or MSI-X? (Because I already see MSI is not enabled)
Also, what is the procedure being followed here to force edge interrupts?
don’t you see any MSI specific code in the driver like pci_enable_msi()?
Also, I don’t see “BusMaster+” instead see “BusMaster-”.
Without enabling Bus mastering capability of the device, how is the device able to do DMA to/from the system?
Is this device driver ‘xrserial’ an upstream driver? (I don’t find it in the upstream code)
I checked and having level HIGH interrupt is not an issue.
For the interrupt to be de-asserted by the system, root port must receive a INTA_DEASSERT message from the endpoint. If there is some delay between clearing the register (that is responsible for stopping the interrupt) and endpoint device really sending that INTA_DEASSERT message to the root port, then, it is possible that we might see spurious interrupts.
BTW, if possible, can you change the driver to use MSI interrupts instead of Legacy interrupts?
The driver is provided by the chip manufacturer.
I don’t think DMA is needed since everything works with interrupts either for writing data into FIFOs or reading from.
I just see pci_enable_device(dev) but no mention about pci_enable_msi().
I see on few NVidia drivers pci_enable_msi().
Should I also check on card datasheet if MSI capability is allowed ?
Hi vidyas,
I did what you recommended and I discovered that the issue isn’t related to interrupts but, probably to memory transactions on the bus.
Here is the sequence of states compared between Intel x86 and NVidia TX2.
On Exar chip I have 2 addresses where I can read the interrupt register of the uart ports:
a global 32bit interrupt register
4 x 8bit 16550 compatible interrupt registers
I use the loopback mode to send/receive data
INTEL behavior
At the first IRQ received:
the global interrupt register reports that TX FIFO data is empty
the interrupt register of the involved uart port reports the same information
In the ISR invoked at point 1 the CPU fills the TX FIFO
Before to return from interrupts, the ISR checks again if there are data in the RX FIFO (reading the uart interrupt register): both interrupts global register and uart register reports no data available
A second IRQ is received:
the global interrupt register reports that RX FIFO has data
the uart interrupt register reports the same
In the ISR data are read from the RX FIFO
Before to return from interrupt, the ISR checks if there are data to send/read but both global and uart interrupt registers reports there aren’t anymore.
NVidia behavior
At the first IRQ received:
the global interrupt register reports that TX FIFO data is empty
the interrupt register of the involved uart port reports the same information
In the ISR invoked at point 1 the CPU fills the TX FIFO
Before to return from interrupts, the ISR checks again if there are data in the RX FIFO (reading the uart interrupt register): both interrupts global register and uart register reports no data available
A second IRQ is received:
the global interrupt register reports that RX FIFO has data
the uart interrupt register reports that THERE ARE NO DATA TO READ
A third IRQ is received:
the global interrupt register reports that RX FIFO has still data
the uart interrupt register reports that THERE ARE NO DATA TO READ
Again same behavior at points 4 and 5 up to 7 times (I’ve sent 6 characters)
No more interrupts are received
Exar datasheet says that the global interrupt register is cleared by reading the interrupt register of the uart port self.
May be an issue related to the fact that the uart interrupt register has an offset of 0x2 that is rounded to a memory transaction at 32bit ?
The device driver on Intel and on NVidia are exactely the same.
So it could be that, when I read the uart interrupt register at offset 0x2, the bus transaction reads registers at offset 0x0, 0x1, 0x2, 0x3. By the way, at the offset 0x0 there is the register RHR to download received data from the FIFO.
At each interrupt received a character is taken from FIFO until it becomes empty and no more interrupts.
Is there any way to read on PCIe bus a byte at a time ?
Nvidia Tegra (till TX2) doesn’t support true DW (Double Word) unaligned accesses. It always sends a request for 4 bytes (i.e. DW aligned address) and then uses ‘byte enable’ concept to return only the data that CPU is really looking for.
So, in this case, the 32-bit read is auto clearing the bits, issue is coming up.
Unfortunately there is no solution we can offer at this point for this issue.
Please check with vendor if is possible to have registers at 32-bit boundary also check if there is any other solution given this is how Tegra works.