Using the Development kit M.2 E Key slot for an FPGA

aseipp · December 14, 2017, 9:59pm

Hello,

I have a Jetson TX2 development kit, and recently I bought this M.2 E Key slot Artix-7 FPGA, as I wanted to experiment with PCIe designs:

This is an M.2 2230 A/E compatible device, which I figured would work fine on the Jetson. It slots in nicely. I’m running the JetPack 3.1 release.

The board has 3 LEDs, and is pre-programmed with an FPGA bitstream that both[1]:

Sets LEDs B and C to constant high, so there is no blink, just a lit LED.
Sets LED A to constant blink, indicating the PCIe clock line is active.

After this, the device should be visible in sudo lspci, I would suspect, and I could then load the Xilinx xdma driver and play around. Upon installation and bootup, however, this is not the case.

First, sudo lspci -vvv returns nothing (literally zero lines of output). Second, LED A is not blinking, indicating the clock line isn’t active to the FPGA’s PCIe PHY.

Being curious, I reset the board (fully shutting down and unplugging the power, and replugging – full cold restart) and observed the LEDs on the FPGA. During bootup, after HDMI is active and I can see systemd dumping boot information, LED A does start blinking, but then stops after a second or two, as if the PCIe bus is active, then disabled.

Looking at dmesg output, this would seem to be accurate. The kernel scans the bus but ignores the device; and then powers down the PCIe power rails. Here is the relevant dmesg snippet:

nvidia@tegra-ubuntu:~$ dmesg | grep pcie
[   13.238869] tegra-pcie 10003000.pcie-controller: 4x1, 1x1 configuration
[   13.239747] tegra-pcie 10003000.pcie-controller: PCIE: Enable power rails
[   13.240130] tegra-pcie 10003000.pcie-controller: probing port 0, using 4 lanes
[   13.243106] tegra-pcie 10003000.pcie-controller: probing port 2, using 1 lanes
[   13.672856] tegra-pcie 10003000.pcie-controller: link 0 down, retrying
[   14.084691] tegra-pcie 10003000.pcie-controller: link 0 down, retrying
[   14.487015] tegra-pcie 10003000.pcie-controller: link 0 down, retrying
[   14.489035] tegra-pcie 10003000.pcie-controller: link 0 down, ignoring
[   14.891382] tegra-pcie 10003000.pcie-controller: link 2 down, retrying
[   15.294816] tegra-pcie 10003000.pcie-controller: link 2 down, retrying
[   15.708281] tegra-pcie 10003000.pcie-controller: link 2 down, retrying
[   15.710299] tegra-pcie 10003000.pcie-controller: link 2 down, ignoring
[   15.710304] tegra-pcie 10003000.pcie-controller: PCIE: no end points detected
[   15.710478] tegra-pcie 10003000.pcie-controller: PCIE: Disable power rails

At this point I figured that perhaps the M.2 key slot being intended for WiFi might be the case (as the board is marked to indicate this), but wasn’t sure how to confirm this: even if I don’t have the FPGA plugged in, the boot messages seem to be the same regardless.

Does anyone have any advice here? I’m probably missing something exceedingly obvious to check/ask. Thanks!

Here is the full information from the dmesg log at the time of scanning the bus:

nvidia@tegra-ubuntu:~$ dmesg
...
[   13.238869] tegra-pcie 10003000.pcie-controller: 4x1, 1x1 configuration
[   13.239747] tegra-pcie 10003000.pcie-controller: PCIE: Enable power rails
[   13.240130] tegra-pcie 10003000.pcie-controller: probing port 0, using 4 lanes
[   13.243106] tegra-pcie 10003000.pcie-controller: probing port 2, using 1 lanes
[   13.273909] xhci-tegra 3530000.xhci: cannot find firmware....retry after 1 second
[   13.400005] dhd_module_init in
[   13.407740] tegra_net_perf_init: cannot get wifi sclk
[   13.427846] found wifi platform device bcmdhd_wlan
[   13.436389] gpio tegra-gpio-aon wake69 for gpio=59(FF:3)
[   13.444881] wifi_platform_get_country_code_map: could not get country_code_map
[   13.456580] wifi_plat_dev_drv_probe:platform country code map is not available
[   13.467057] Power-up adapter 'DHD generic adapter'
[   13.474117] wifi_platform_set_power = 1
[   13.672856] tegra-pcie 10003000.pcie-controller: link 0 down, retrying
[   13.682835] wifi_platform_bus_enumerate device present 1
[   13.714358] wifi_platform_bus_enumerate device present 0
[   13.729019] F1 signature read @0x18000000=0x17214354
[   13.738695] F1 signature OK, socitype:0x1 chip:0x4354 rev:0x1 pkg:0x2
[   13.739350] DHD: dongle ram size is set to 786432(orig 786432) at 0x180000
[   13.739414] wifi_platform_prealloc: failed to alloc static mem section 7
[   13.739420] wifi_platform_get_mac_addr
[   13.740545] CFG80211-ERROR) wl_setup_wiphy : Registering Vendor80211
[   13.749143] wl_create_event_handler(): thread:wl_event_handler:258 started
[   13.750961] CFG80211-ERROR) wl_event_handler : tsk Enter, tsk = 0xffffffc1e1a61a70
[   13.752715] dhd_attach(): thread:dhd_watchdog_thread:25a started
[   13.752905] dhd_attach(): thread:dhd_dpc:25b started
[   13.755155] dhd_attach(): thread:dhd_rxf:25c started
[   13.755164] dhd_deferred_work_init: work queue initialized 
[   13.758377] Dongle Host Driver, version 1.201.82 (r)
               Compiled in drivers/net/wireless/bcmdhd on Jul 20 2017 at 00:47:12
[   13.758657] tegra_sysfs_register
[   13.758687] Register interface [wlan0]  MAC: 00:04:4b:8c:c0:a7

[   13.758690] dhd_prot_ioctl : bus is down. we have nothing to do
[   13.760617] sdhci-tegra 3440000.sdhci: Tuning already done, restoring the best tap value : 65
[   13.761658] wifi_platform_set_power = 0
[   14.084691] tegra-pcie 10003000.pcie-controller: link 0 down, retrying
[   14.272763] xhci-tegra 3530000.xhci: Firmware timestamp: 2017-03-31 06:21:56 UTC, Version: 55.07 release
[   14.274828] xhci-tegra 3530000.xhci: xHCI Host Controller
[   14.274841] xhci-tegra 3530000.xhci: new USB bus registered, assigned bus number 1
[   14.275652] xhci-tegra 3530000.xhci: hcc params 0x0184fd25 hci version 0x100 quirks 0x00010810
[   14.275675] xhci-tegra 3530000.xhci: irq 59, io mem 0x03530000
[   14.275796] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[   14.275798] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   14.275800] usb usb1: Product: xHCI Host Controller
[   14.275802] usb usb1: Manufacturer: Linux 4.4.38-tegra xhci-hcd
[   14.275803] usb usb1: SerialNumber: 3530000.xhci
[   14.276077] hub 1-0:1.0: USB hub found
[   14.276100] hub 1-0:1.0: 4 ports detected
[   14.294943] xhci-tegra 3530000.xhci: xHCI Host Controller
[   14.294951] xhci-tegra 3530000.xhci: new USB bus registered, assigned bus number 2
[   14.295114] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003
[   14.295117] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   14.295119] usb usb2: Product: xHCI Host Controller
[   14.295121] usb usb2: Manufacturer: Linux 4.4.38-tegra xhci-hcd
[   14.295122] usb usb2: SerialNumber: 3530000.xhci
[   14.295422] hub 2-0:1.0: USB hub found
[   14.295446] hub 2-0:1.0: 3 ports detected
[   14.295825] tegra-xotg xotg: otg: host 3530000.xhci registered
[   14.487015] tegra-pcie 10003000.pcie-controller: link 0 down, retrying
[   14.489035] tegra-pcie 10003000.pcie-controller: link 0 down, ignoring
[   14.586814] usb 1-2: new full-speed USB device number 2 using xhci-tegra
[   14.721222] usb 1-2: New USB device found, idVendor=248a, idProduct=8367
[   14.721225] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[   14.721227] usb 1-2: Product: Wireless Receiver
[   14.721228] usb 1-2: Manufacturer: Telink
[   14.721446] usb 1-2: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes
[   14.721802] xhci-tegra 3530000.xhci: tegra_xhci_mbox_work mailbox command 6
[   14.726628] input: Telink Wireless Receiver as /devices/3530000.xhci/usb1/1-2/1-2:1.0/0003:248A:8367.0001/input/input4
[   14.778992] hid-generic 0003:248A:8367.0001: input,hidraw0: USB HID v1.11 Mouse [Telink Wireless Receiver] on usb-3530000.xhci-2/input0
[   14.781498] input: Telink Wireless Receiver as /devices/3530000.xhci/usb1/1-2/1-2:1.1/0003:248A:8367.0002/input/input5
[   14.835085] hid-generic 0003:248A:8367.0002: input,hidraw1: USB HID v1.11 Keyboard [Telink Wireless Receiver] on usb-3530000.xhci-2/input1
[   14.891382] tegra-pcie 10003000.pcie-controller: link 2 down, retrying
[   14.946804] usb 1-3: new full-speed USB device number 3 using xhci-tegra
[   15.099090] usb 1-3: New USB device found, idVendor=0403, idProduct=6015
[   15.099093] usb 1-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[   15.099095] usb 1-3: Product: FT230X Basic UART
[   15.099096] usb 1-3: Manufacturer: FTDI
[   15.099098] usb 1-3: SerialNumber: DO00HAMO
[   15.099597] xhci-tegra 3530000.xhci: tegra_xhci_mbox_work mailbox command 6
[   15.102624] ftdi_sio 1-3:1.0: FTDI USB Serial Device converter detected
[   15.102673] usb 1-3: Detected FT-X
[   15.102906] usb 1-3: FTDI USB Serial Device converter now attached to ttyUSB0
[   15.294816] tegra-pcie 10003000.pcie-controller: link 2 down, retrying
[   15.326954] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: data=ordered
[   15.708281] tegra-pcie 10003000.pcie-controller: link 2 down, retrying
[   15.710299] tegra-pcie 10003000.pcie-controller: link 2 down, ignoring
[   15.710304] tegra-pcie 10003000.pcie-controller: PCIE: no end points detected
[   15.710478] tegra-pcie 10003000.pcie-controller: PCIE: Disable power rails
[   15.942796] xhci-tegra 3530000.xhci: tegra_xhci_mbox_work mailbox command 5
[   15.942799] xhci-tegra 3530000.xhci: tegra_xhci_mbox_work ignore firmware MBOX_CMD_DEC_SSPI_CLOCK request
[   18.135467] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[   18.135537] 
               Dongle Host Driver, version 1.201.82 (r)
               Compiled in drivers/net/wireless/bcmdhd on Jul 20 2017 at 00:47:12
[   18.135539] wl_android_wifi_on in
[   18.135541] wifi_platform_set_power = 1
...

(I included the full information, because at the time of booting it seems as if the system has an error loading WiFi firmware, just as a hunch/tip off of what might be wrong.)

[1] Installation · RHSResearchLLC/PicoEVB Wiki · GitHub
[2] https://github.com/RHSResearchLLC/PicoEVB/tree/master/DMADriver

Bibek · December 20, 2017, 3:53am

Hi,

The pcie lines to M2Key E needs below two SW support to get it working.

Please refer to pages 12-13 of the TX2 Platform Adaptation Guide (internal link: http://sw-mobile-docs/r28/L4T/r28.2/Tegra%20Linux%20Driver%20Package%20Development%20Guide/baggage/Tegra_Linux_Driver_Package_TX2_Adaptation_Guide.pdf). Here you will find what lane configuration you need to do via odmdata to get the pcie line up
change in the file P2771-0000.common.conf
ODMDATA=0x0090000;
You need to refer to the schematic of CVM TX2 to enable the muxes on the path to enable correct routing.

Example working dt change here : below snippet is from the devicetree for TX2. You need to make sure that the nodes under overlay are updated as mention below.

            fragment-500-e3325-pcie {
                    enable-override-on-all-matches;
                    ids = ">=3310-1000-500";
                    odm-data = "enable-pcie-on-uphy-lane0";
                    override@0 {
                            target = <&{/xhci@3530000}>;
                            _overlay_ {
                                    phys = <&tegra_xusb_padctl TEGRA_PADCTL_PHY_UTMI_P(0)>,
                                            <&tegra_xusb_padctl TEGRA_PADCTL_PHY_UTMI_P(1)>;
                                    phy-names = "utmi-0", "utmi-1";
                            };
                    };
                    override@1 {
                            target = <&tegra_xusb_padctl_pinmux_default>;
                            _overlay_ {
                                    usb3-std-A-port2 {
                                            status = "disabled";
                                    };
                            };
                    };

                    override@2 {
                            target = <&tegra_main_gpio>;
                            _overlay_ {
                                    pcie0_lane2_mux {
                                            status = "okay";
                                    };
                            };
                    };
            };

sherwoac · January 12, 2019, 3:09am

Did you manage to get the fpga going?

Thanks

Adam

moozoo · June 14, 2019, 2:04am

In case someone from Nvidia reads this.
Would it be good to have a Jetson on a M.2.card? So it could be plugged into a back plane (or PC) with other M.2. cards like the mentioned fpga.