This is relevant to the problems I’ve been dealing with.
I too have an FPGA attached to the NVIDIA TX1. This FPGA links up just fine (as apposed to my Spartan 6 Board).
I wrote and tested out my driver on a regular x64 Ubuntu desktop and everything worked fine.
I brought it over to the TX1 L4T 24.1 git hash: 07fb031a9cea70cb733203d3c9867000639c6e87 and have had some difficulties. I noticed some strange things.
Probe Function Modifies OV5693
If I load the driver and the probe is called I see this on dmesg every time:
[ 126.795519] [OV5693]: probing v4l2 sensor.
[ 126.795725] ov5693 6-0036: camera_common_regulator_get vana ERR: fffffffffffffdfb
[ 126.803617] ov5693 6-0036: camera_common_regulator_get vif ERR: fffffffffffffdfb
I played with the driver a bit and modified the function that is called when the PCIE device is found (probe function). Even if the only thing I do is “return 0;” I see the above.
I’m not using the OV5963 camera at the moment but I thought I would mention this either way.
pci_set_dma_mask with 32-bit MASK returns 33-bit address??
I’m still navigating my way through the PCIE protocol and the accompanying kernel interface but I thought that by calling this function
I would get a 32-bit address but instead I got a 33-bit address. I suppose technically it’s a 64-bit address but I didn’t want to split hairs:
(Second address on each line is the remapped DMA address)
[ 126.792856] nysa_pcie - construct_pcie_device : Status Buf Addr: ffffffc0e241c000 : DMA Buf Addr : 000000016241c000
[ 126.792867] nysa_pcie - construct_pcie_device : Write Buf  Addr: ffffffc0ef142000 : DMA Buf Addr : 000000016f142000
[ 126.792875] nysa_pcie - construct_pcie_device : Write Buf  Addr: ffffffc0f5c35000 : DMA Buf Addr : 0000000175c35000
[ 126.792891] nysa_pcie - construct_pcie_device : Read Buf  Addr: ffffffc0e248b000 : DMA Buf Addr : 000000016248b000
[ 126.792899] nysa_pcie - construct_pcie_device : Read Buf  Addr: ffffffc0e248a000 : DMA Buf Addr : 000000016248a000
Note: My driver is very simple. It is not using ‘scatter-gather’ it is simply using a double buffer scheme with 4096 byte blocks so I don’t think I need to use ‘coherent’. The status buffer is just a single 4096 byte block I use to read the status of the HDL controller inside my FPGA (I’ll call it FPGA Controller).
I was confused but, at first, I proceeded to use only the lower 32-bit address to send data from the FPGA to the TX1 status buffer but it wouldn’t work.
I modified my FPGA controller to use 64-bit addressing and changed the DMA mask to use 64-bit instead of 32-bit and I could send data from the FPGA to the TX1 on the status buffer so I can confirm that DMA transfers from the PCI Device to the TX1 Root Complex does work. Although I don’t know if this is true for scatter-gather based drivers.
The driver I wrote allows me to read and write data to the FPGA using a character device file. Basically I open up the file and write data into it and this data will get down to the FPGA. There is a little bit more to it than that but for the sake of brevity this is sufficient.
I wrote a simply python script that attempts to send 20 bytes down to the FPGA but it didn’t work. I have a logic analyzer built into the FPGA and have observed the FPGA controller recognize that the TX1 wants to write down 20 bytes. I can see that the controller does create the memory read request:
This is the PCIE TLP Memory Request that the FPGA controller generates:
Here is the significance of each line:
- Memory Read Transaction Request
- TLP Packet is for a 64-bit memory read transaction
- Length of read request: 512 bytes (0x80 * 4)
- Requester ID: Bus ID: 0x01, Device ID: 0x00, Function ID: 0x00
- Memory Read Tag: 0x00
- Address to read from: 0x000000016F142000
I am expecting that the root complex on the TX1 would send 512 bytes down to the FPGA but I don’t see this, instead my driver blocks the user application. This worked on the desktop computer but the TX1 is just frozen. dmesg doesn’t say anything is wrong. It seems as though the root complex never received the DMA request.
When I designed the driver I was afraid that this situation might happen so I added a back door into it using ‘sysfs’ that allows me to unblock my driver and exit gracefully. When I unblocked my driver I saw these messages using dmesg:
[ 1373.816730] pcieport 0000:00:01.0: AER: Uncorrected (Fatal) error received: id=0010
[ 1373.816750] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, id=0008(Receiver ID)
[ 1373.828269] pcieport 0000:00:01.0: device [10de:0fae] error status/mask=00040000/00000000
[ 1373.837191] pcieport 0000:00:01.0:  Malformed TLP (First)
[ 1373.843975] pcieport 0000:00:01.0: TLP Header: 20000080 01000000 00000001 6f142000
[ 1373.852172] pcieport 0000:00:01.0: broadcast error_detected message
[ 1373.852182] nysa_pcie 0000:01:00.0: device has no AER-aware driver
[ 1374.088335] pcieport 0000:00:01.0: Root Port link has been reset
[ 1374.088514] pcieport 0000:00:01.0: AER: Device recovery failed
It’s strange that the root complex thinks that this is a ‘malformed’ TLP. Using the PCIE specification I verified that this is a valid packet.
This all may be a moot point. I want to try out the above modification but I have to figure out how to preserve my root file system so I don’t have re-install all my tools each time I re-flash.
I plan to try this out tonight.