The PCIe C4 EP in CC mode cannot be enumerated by the RC

Robotics & Edge Computing Jetson Systems Jetson Thor
1

Hi NV,

The PCIe C4 EP in CC mode cannot be enumerated by the RC in our project.

Issue: Thor PCIe C4 EP in CC mode fails to be enumerated by either a Thor RC or an x86 RC. The Thor EP kernel log reports a ‘bpmp rx.ret=-23’ error.

Note: In SRIS mode, the Thor RC successfully enumerates the Thor EP (PCIe 5.0 x8) and operates normally.

Modifications made to the EP SRIS:

  1. clk-scheme = <0x01>;
  2. add pcie-c4-endpoint-use-int-refclk in bpmp uphy;
  3. tegra264-mb1-bct-boot-clocks-defaults.dtsi ss_ctrl3 = <0x24010025>; → ss_ctrl3 = <0x37010018>;

Tests steps:

  1. Affter Thor EP boot, the driver is loaded and bound to the PCIe EP controller successfully. The PCIe EP controller starts up normally.
modprobe tegra-pcie-dma-lib
modprobe pcie-tegra264-ep
modprobe ep_side_module
cd /sys/kernel/config/pci_ep/
mkdir functions/ep_side_module/func1
echo 0x10de > functions/ep_side_module/func1/vendorid
echo 0x22d7 > functions/ep_side_module/func1/deviceid
echo 16 > functions/ep_side_module/func1/msi_interrupts
ln -s functions/ep_side_module/func1/ controllers/a808460000.pcie-ep/
echo 1 > controllers/a808460000.pcie-ep/start

Hardware check: Measure the RST and REF_CLK on the Thor EP. RST is pulled high (3.3V), while REF_CLK is not yet available.

  1. When the RC (either Thor-RC or x86-RC) is powered on, a 100MHz REF_CLK can be observed on the oscilloscope, and the RST signal is initially pulled low before being released (pulled high). During the period when RST is asserted (pulled low), the Thor-EP kernel log shows that after calling tegra_bpmp_transfer, the BPMP responds with an rx.ret=-23 error. Subsequently, the RC fails to enumerate this Thor-EP.
    image
    image2502×917 527 KB

For comparison, here is the Thor-EP kernel log when the Thor-RC successfully detects the Thor-EP in SRIS mode:

image
image1839×613 74.5 KB

add print in tegra264_pcie_bpmp_set_ep_state:

image
image1532×1156 74.8 KB

Our modifications on the Thor EP:
follow: Jetson Thor Adaptation and Bring-Up — NVIDIA Jetson Linux Developer Guide

pinmux:

pex_l4_clkreq_n_pd0 { /* zlj+ for c4 ep */
        nvidia,pins = "pex_l4_clkreq_n_pd0";
        //nvidia,function = "rsvd1";
        //nvidia,pull = <TEGRA_PIN_PULL_DOWN>;
        //nvidia,tristate = <TEGRA_PIN_ENABLE>;
        nvidia,function = "pe4_clkreq_l";
        nvidia,pull = <TEGRA_PIN_PULL_NONE>;
        nvidia,tristate = <TEGRA_PIN_DISABLE>;
        nvidia,enable-input = <TEGRA_PIN_ENABLE>;
        nvidia,drv-type = <TEGRA_PIN_1X_DRIVER>;
        nvidia,e-io-od = <TEGRA_PIN_ENABLE>;
        nvidia,e-lpbk = <TEGRA_PIN_DISABLE>;
};

pex_l4_rst_n_pd1 { /* zlj+ for c4 ep */
        nvidia,pins = "pex_l4_rst_n_pd1";
        nvidia,function = "rsvd1";
        //nvidia,pull = <TEGRA_PIN_PULL_DOWN>;
        //nvidia,tristate = <TEGRA_PIN_ENABLE>;
        nvidia,pull = <TEGRA_PIN_PULL_NONE>;
        nvidia,tristate = <TEGRA_PIN_DISABLE>;
        //nvidia,enable-input = <TEGRA_PIN_DISABLE>;
        nvidia,enable-input = <TEGRA_PIN_ENABLE>;
        nvidia,drv-type = <TEGRA_PIN_1X_DRIVER>;
        nvidia,e-io-od = <TEGRA_PIN_ENABLE>;
        nvidia,e-lpbk = <TEGRA_PIN_DISABLE>;
};

tegra264-bpmp-3834-0008-4071-xxxx.dtb:

uphy {
        status = "okay";
        uphy0-config = <0x07>;
        uphy1-config = <0x00>; /* for c4 ep */
        pcie-c4-endpoint-enable; /* for c4 ep */
        mgbe0-speed = <0x02>;
        mgbe1-speed = <0x02>;
        mgbe2-speed = <0x02>;
        mgbe3-speed = <0x02>;
};

pcie@4 {
        status = "okay"; /* enable c4 */
        pcie-id = <0x04>;
        pcie-mode = <0x02>; /* for c4 ep 0x01 -> 0x02*/
        clk-scheme = <0x00>; /* cc mode */
        max-link-speed = <0x05>;
};

pcie@5 {
        status = "disabled";  /* disable c5 */
        pcie-id = <0x05>;
        pcie-mode = <0x01>;
        max-link-speed = <0x05>;
};

source/hardware/nvidia/t264/nv-public/nv-platform/tegra264-p4071-0000.dtsi:

/ {
        bus@0 {
		....
		        /* PCIe C4 is SNN on Jedha */
                pcie@a808460000 {
                        status = "disabled"; /* diable c4 rc */
                };

                //zlj+ enable c4 ep
                pcie-ep@a808460000 {
                        status = "okay";
                };

                /* PCIe C5 to enable NVME */
                pcie@a808480000 {
                        //status = "okay";
                        status = "disabled"; //zlj+ disable c5
                };
		};
};
4

currently there are driver missing so Thor PCIe EP is not fully validated.