Not being able to write on a PCIe device on Jetson TX2

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Hey everyone,
My problem is that I’m not able to write on PCIe devices with the TX2.

Here’s exactly what I’m trying:
I’m trying to write, then check what I’ve written on a PCIe device. For this, I’m using tools like devmem2.c and such. I’ve natively compiled these tools on TX2. However, somehow this operation of basic write is somehow failing. Either I’m not able to write, or I’m reading gibberish.

Here’s what I’ve done so far:

  • I checked if I can see device’s vendor ID, device ID, and BAR address assignment. These seems OK; thus things in BSP layer is OK. I should be able to do what I’m trying to do at user space.
  • I checked if I’m enabling PCIe device’s memory with “setpci -s COMMAND=0x02”. I’m definitely doing that.
  • I used another card as root complex to write/read on the same PCIe device, and succeeded. So, my issue should have been connected to TX2. The device which is an end point is up and runnning, but somehow I cannot achieve to control it.
  • Finally, I plugged an NVMe SSD on the PCIe slot to check. I can definitely control it, but only through the NVMe stack so far.

I’m looking forward for suggestions, fixes etc. since this is keeping me back from what I want/need to do with the TX2.

PS: The device I’m plugging at PCIe is seen as an “unassigned class” device. I’m not sure it is something to be considered, but there is that.

3

can you paste lspci -vvv output of your pcie device ?
BusMaster should be enabled to be able to read/write BAR address space.

Please set first 3 bits of offset 0x4 and try again
setpci -s 4.l=0x7

you can use reg_dump tool (/home/ubuntu/reg_dump) in place of devmem.

4

Hey, sorry for the late response. I was away from the setup since the whole lockdown situation. Here are some of the outputs, including lspci -vvv:
lspci-vv.log (5.7 KB)
lspci-v.log (1.1 KB)
lspci-vvv.log (5.8 KB)
Since I can only upload 3 files in a comment, remaining are copy-pasted here:
lspci -n:
00:01.0 0604: 10de:10e5 (rev a1)
01:00.0 ff00: 1172:e001 (rev 01)
my configuration:
sudo setpci -s 0000:01:00.0 COMMAND=0x02
sudo setpci -s 0000:01:00.0 4.l=0x7
lspci:
00:01.0 PCI bridge: NVIDIA Corporation Device 10e5 (rev a1)
01:00.0 Unassigned class [ff00]: Altera Corporation Device e001 (rev 01)
PS: adding sudo setpci -s 0000:01:00.0 4.l=0x7 to my config did not help. Also, I couldn’t find the tool you mentioned, but that seems irrelevant with the issue.

7

from lspci -vvv output BusMaster is enabled. so write to BAR memory should be working

Control: I/O- Mem+ BusMaster+

Can you share which tool and command you are using to write to pcie device

You mentioned you used same pcie device on other host ? and it is working ?
Can you share “sudo lspci -vvv” output from other host where it is working?

8

I’m using GitHub - andre-richter/easy-pci-mmap: C++ class for easy read and write of PCI(e) device BARs via mmap to read/write on the device. I’ve changed the example/example.cpp’s main as:

#include “pci_mmap.hpp”

#define VENDOR 0x1172
#define DEVICE 0xe001
#define FUNCTION 0
#define BAR 0

#define BAR_OFFSET 0x0

int main(int argc, char *argv)
{
pci_mmap dev(VENDOR, DEVICE, FUNCTION, BAR);

    std::cout << "Domain:Bus:Device:Function: "
              << dev.dbdf() << std::endl

              << "sysfs file:                 "
              << dev.sysfs_bar_file() << std::endl

              << "BAR size:                   0x"
              << std::hex << dev.bar_size() << std::endl;

    std::cout << "read from BAR at offset:    0x"
              << std::hex << BAR_OFFSET << std::endl

              << "value:                      0x"
              << std::hex << dev.read32(BAR_OFFSET) << std::endl;

    dev.write32(BAR_OFFSET,0xff);

    std::cout << "read from BAR at offset:    0x"
              << std::hex << BAR_OFFSET << std::endl

              << "value:                      0x"
              << std::hex << dev.read32(BAR_OFFSET) << std::endl;

}

For the lspci -vvv output of the other device, I’m sorry but there is none. It was not a Linux device, running some BareMetal code, so it’s not possible for me to get that output.

9

Better will be, attach your PCIe card to any standard x86 Linux host and try to access the device. if it is working provide lspci -vvv output. This will give us some reference.

10

Hey again, I have great news with me! TX2 had nothing wrong, it was just me trying to write a memory section that is marked as read-only by the pcie device. The memory after that read-only section is perfectly fine to write on and read from. Thanks for your help, by the way. Have a good one!

11

Can you tell me how long the connection is between the TX2 PCIe header and you device? The spec sheet on the TX2 says you can only have a 440 ps delay which is something around 5 cm in trace length. I am hoping this is wrong and would love to know how long your connection is.