Jetson Orin AGX EQOS MAC issues with customer carrier board and TI PHY

I have a custom carrier board that has the Jetson Orin AGX Industrial module installed on it. I am trying to boot JetPack 6.2.1 Linux on the Orin with the RGMII interface between the EQOS MAC and a TI DP83869HM PHY (on the carrier card). The error from the serial port is:

[ 14.612967] nvethernet 2310000.ethernet: failed to connect PHY
[ 14.612984] net eth0: ether_open: Cannot attach to PHY (error: -19)

A google search on this error suggested I check several items.

Pinmux:
These are the pinmux settings in the Excel file that I am using:

Reset and Interrupt GPIOS:
image

RGMII and MDIO:

The resulting three output files were copied to the Linux build machine and put into the paths referenced by my custom board conf file.

Device Tree section I am using (PHY ID is confirmed as ID 3 on hardware):
bus@0 {
/* Removed for brevity */
ethernet@2310000 {
/* Setting status = "okay" enables the device */
status = "okay";
/* nvidia,tegra234-eqos refers to the NVIDIA Orin */
/* AGX processor. The -eqos refers to the 1Gb */
/* Ethernet MAC */
/* eqos stands for Ethernet Quality of Service */
/* snps refers to synopsis, with dwc-qos-ethernet */
/* being the base of the 1Gb Ethernet IP */
/* Having both compatible statements is required */
compatible = "nvidia,tegra234-eqos","snps,dwc-qos-ethernet";

/* Phy Handle defined below */
phy-handle = <&ti_dp83869_phy>;
/* Using the RGMII interface so specify the */
/* rgmii-id mode */
/* rgmii-id specifies that the PHY controls the */
/* timing delays instead of the MAC */
/* NVIDIA support confirmed that rgmii MAC does */
/* not support Rx/TX delays internally and they */
/* must be supplied by a PHY. Therfore, rgmii, */
/* rgmii-rxid and rgmii-txid will not work as */
/* phy-mode entries */
phy-mode = "rgmii-id";
/* This maps to the GPIO PG05 pin where the */
/* ENET_RST_N signal is connected */
nvidia,phy-reset-gpio = <&gpio TEGRA234_MAIN_GPIO(G, 5) 0>;
/* mac-addr-idx is used to index into an array in */
/* the I2C eeprom on board the GPU module */
/* Kernel throws error on invalid index right now */
/* so the nvidia,ether-max* entries are used below*/
nvidia,mac-addr-idx = <0>;
nvidia,max-platform-mtu = <8000>;
nvidia,pause_frames = <0>;
/* Some sources online say this is required - set */
/* it to the same as nvidia,ether-mac */
local-mac-adress = [48 80 2D 68 C9 1C];
nvidia,ether-mac = "48:80:2D:68:C9:1C";
nvidia,ether-mac0 = "48:80:2D:68:C9:1C";
nvidia,ether-mac1 = "48:80:2D:68:C9:1D";
nvidia,ether-mac2 = "48:80:2D:68:C9:1E";
nvidia,ether-mac3 = "48:80:2D:68:C9:1F";
nvidia,ether-mac4 = "48:80:2D:68:C9:20";
nvidia,ether-mac5 = "48:80:2D:68:C9:21";
nvidia,ether-mac6 = "48:80:2D:68:C9:22";
nvidia,ether-mac7 = "48:80:2D:68:C9:23";
nvidia,ether-mac8 = "48:80:2D:68:C9:24";
nvidia,ether-mac9 = "48:80:2D:68:C9:25";

/* Not sure if these two lines are helping or */
/* hurting currently */
clock-names = "slave_bus","master_bus","rx","tx","ptp_ref";
reset-names = "eqos";

/* MDIO section describes everything from singals */
/* to devices on this bus */
mdio {
/* Set compatibility to nvidia,eqos-mdio and */
/* synopsis dwc-qos-ethernet-mdio */
compatible = "nvidia,eqos-mdio", "snps,dwc-qos-ethernet-mdio";
/* These next two lines are required */
#address-cells = <1>;
#size-cells = <0>;

/* Define the TI DP83869HM PHY settings - */
/* phy@3 specifices Device ID 03 */
ti_dp83869_phy: ethernet-phy@3 {
/* Set the device to active */
status="okay";
/* Set compatible so that it finds the TI */
/* DP83869 device driver - must be enabled*/
/* in kernel */
compatible = "ti,dp83869";
device-type = "ethernet-phy";
/* reg = must match the PHY device ID */
/* above or 3 */
reg = <0x3>;
/* Set both TX/RX FIFO depths to 4 Bytes */
tx-fifo-depth = <DP83869_PHYCR_FIFO_DEPTH_4_B_NIB>;
rx-fifo-depth = <DP83869_PHYCR_FIFO_DEPTH_4_B_NIB>;
/* Make sure we are in RGMII Copper mode */
ti,op-mode = <DP83869_RGMII_COPPER_ETHERNET>;
ti,max-output-impedance;
ti,clk-output-sel = <DP83869_CLK_O_SEL_CHN_A_RCLK>;
rx-internal-delay-ps = <2000>;
tx-internal-delay-ps = <2000>;
nvidia,phy-rst-pdelay-msec = <224>; /* msec */
nvidia,phy-rst-duration-usec = <10000>; /* usec */
/* Set the Interrupt pin up as a gpio pin */
/* and use PG04 - maps to ENET_INT */
interrupt-parent = <&gpio>;
interrupts = <TEGRA234_MAIN_GPIO(G, 4) IRQ_TYPE_LEVEL_LOW>;
};
};
};

/* Removed for brevity */
};

Currently trying to install the mdio-tool into the Linux image to read the PHY registers.

I do not have wires on the MDIO lines currently but could install them if required.

I would greatly appreciate help in figuring out why I can’t talk to the PHY from the EQOS MAC.

*** Please note that this reply is generated by LLM automatically ***

Based on the provided information, I will attempt to help you troubleshoot the issue with your custom carrier board and the Jetson Orin AGX Industrial module.

Error Analysis

The error message “nvethernet 2310000.ethernet failed to connect PHY” and “net eth0: etheropen: Cannot attach to PHY error 19” suggests that there is an issue with the communication between the EQOS MAC and the TI DP83869HM PHY.

Pinmux Settings

You have provided the pinmux settings in the Excel file, which appear to be correct. The RGMII and MDIO pins are properly configured, and the reset and interrupt GPIOs are also set up correctly.

Device Tree

You have confirmed that the PHY ID is set to 3 in the device tree, which matches the hardware configuration.

MDIO Tool

You are trying to install the mdiotool to read the PHY registers, which is a good step in debugging the issue. However, you may not need to install wires on the MDIO lines if you can use the mdiotool to read the registers.

Possible Causes

Based on the error messages and the configuration, here are some possible causes:

  1. PHY ID mismatch: Although you have confirmed that the PHY ID is set to 3 in the device tree, it’s possible that there is a mismatch between the device tree and the hardware configuration. Double-check that the PHY ID is correct and matches the hardware configuration.
  2. Pinmux configuration: Although the pinmux settings appear to be correct, it’s possible that there is an issue with the pinmux configuration. Verify that the pinmux settings are correct and that there are no conflicts with other devices on the board.
  3. MDIO clock: The MDIO clock is not explicitly mentioned in the configuration. Ensure that the MDIO clock is properly configured and that it is enabled.
  4. PHY reset: The PHY reset signal is not explicitly mentioned in the configuration. Ensure that the PHY reset signal is properly configured and that it is asserted correctly.
  5. Hardware issue: There may be a hardware issue with the board or the PHY. Verify that the board is properly assembled and that there are no issues with the PHY or the EQOS MAC.

Comparison with Other Configurations

Since you are using a custom carrier board, it’s possible that the configuration is different from other boards. Compare your configuration with other boards that use the same PHY and EQOS MAC to ensure that there are no differences in the configuration.

Recommendations

  1. Verify the PHY ID: Double-check that the PHY ID is correct and matches the hardware configuration.
  2. Verify the pinmux configuration: Verify that the pinmux settings are correct and that there are no conflicts with other devices on the board.
  3. Check the MDIO clock: Ensure that the MDIO clock is properly configured and that it is enabled.
  4. Check the PHY reset: Ensure that the PHY reset signal is properly configured and that it is asserted correctly.
  5. Use the mdiotool: Use the mdiotool to read the PHY registers and verify that the PHY is properly configured.
  6. Check the hardware: Verify that the board is properly assembled and that there are no issues with the PHY or the EQOS MAC.

By following these recommendations, you should be able to troubleshoot the issue and resolve the problem with your custom carrier board and the Jetson Orin AGX Industrial module.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

If you get this working please let me know your solution. I’m getting a similar error, but I’m using a KSZ9031 instead of the TI part.

I am having other issues with my build that is preventing me from testing the recommendations. When I inspected our change for the PHY ID, I noticed that a wire came unsoldered which put it back at PHY ID 0. I have had the wire repaired to change the PHY ID back to 03 but have not been able to test it yet.

Got my build issues fixed. Still have several issues.

Issue 1

GPIO line for Reset_n (active low signal to PHY) is not toggling on the scope. I looked in the /sys/kernel/debug/gpio file after booting and the PG.05 line shows nothing connected. Also the Interrupt line (PG.04) shows nothing mapped as well. Latest dts/dtsi files attached.

tegra234-tpcca+p3701-0008_dts.txt (8.3 KB)

tpcca-orin-industrial_conf.txt (3.5 KB)

tegra234-tpcca-p3701-0008_dtsi.txt (2.0 KB)

orin_sys_kernel_debug_gpio.txt (7.3 KB)

tpcca-p3701-0000_conf_common.txt (3.6 KB)

tpcca-p3701_conf_common.txt (8.5 KB)

Issue 2

MDIO line and MDC line are not toggling (no command/data on MDIO and no clock on MDC) at any time during the boot process when the kernel reports it is talking to the PHY.
I have added the TI DP83869 device driver via the make menuconfig option in the Linux_for_Tegra/source/kernel/kernel-jammy-src directory with the config saved as .config

Adding the dmseg output for just the ethernet term and the full dmesg dump. Adding the list of mdio devices in the /sys/bus/mdio_bus/devices directory (it’s empty).

orin_sys_bus_mdio_bus_devices.txt (189 Bytes)

orin_dmseg_ethernet_dump.txt (1.4 KB)

orin_dmseg_dump.txt (42.8 KB)

Device Tree Questions for RGMII Ethernet

As you can see in the ethernet@2310000 entry in the current dts file, I have added much more than any of the examples to get around some of the issues we see reported in the dmesg output (DMA channels, MTL, etc.).

  • Question 1: What is the correct compatible value to use? We have seen multiple values used and currently have:
    • nvidia,tegra234-eqos
    • nvidia,nveqos
    • snps,dwc-qos-ethernet
  • Question 2: What is the correct way to map the PG.05 gpio line to the PHY Reset. Most of the examples we see have (Using JetPack 6.2.1 with R36.4.4 kernel) :
    • nvidia,phy-reset-gpio = <&tegra-main-gpio TEGRA234_MAIN_GPIO(G, 5) GPIO_ACTIVE_LOW>;
    • That statement will not compile saying that tegra-main-gpio is undefined.
    • Instead we have to use gpio in place of tegra-main-gpio
  • Question 3: What is the correct value for the mdio compatible statement. Using two currently:
    • nvidia,eqos-mdio
    • snps,dwc-eqos-ethernet-mdio

Pinmux and gpio files from the pinumx Excel sheet for the post above.

tpcca-orin-industrial-pinmux_dtsi.txt (65.7 KB)

tpcca-orin-industrial-gpio-default_dtsi.txt (5.0 KB)

It seems that current problem here is you messed up the setting in the device tree.

Please refer to the document.

Only these items are needed. I think you added or modify something else so leads to ethernet driver probe totally failed.

Jim,

In Jetpack 6.2.1 you need to do the following format for reset:

This is the example in my case. Just make sure the gpio pin mathces and polarity at the end. I’m not sure why Nvidia doesn’t update the guide to reflect this. Also don’t use $ and use & instead. For interrupt-parent use interrupt-parent = <&gpio>; for example.

nvidia,phy-reset-gpio = <&gpio TEGRA234_MAIN_GPIO(G, 5) 0>;

Thanks Wood - response was very helpful.

Hi Wayne,

I made some good progress figuring this out. I had completely missed that the tegra234_base_overlay.dtsi file in the nv-soc directory had the start of the ethernet@2310000 device tree. I added all of that to our dts file and have attached the new file for reference. It would be very helpful if this file and reference to that device tree driver was mention in the Linux Adaptation and Bringup guide in the section for RGMII PHYs.

The current issue is the polarity of the PHY Reset GPIO is not changing and fixed at active high polarity no matter how I map it in the nvidia,phy-reset-gpio entry in the device tree.

Originally, I had
nvidia,phy-reset-gpio = <&gpio TEGRA234_MAIN_GPIO(G,5), GPIO_ACTIVE_LOW>

The output of the command: sudo gpioinfo | grep PG.0* is attached in the orion_gpioinfo.txt file
line 40: "PG.05" "phy_reset" output active-high [used]
No matter what I try, I can’t get the gpio line to report active-low

Just for completeness, I changed the nvidia,phy-reset-gpio to GPIO_ACTIVE_HIGH and get the same results.

What ends up happening, is when the nvethernet module is loaded, the Phy Reset line is driven low. The TI DP83869 has an active low reset, so it is held in reset and fails the MDIO poll of the Phy ID (3 in my case). The reset pulse is active high on the wire (see image - note the delay for the PHY reset is set for 224ms so we are in the ballpark).

I can forcefully remove the nvethernet module (sudo rmmod nvethernet) and the reset line goes high.

I also noticed that we do get MDIO reads but because the PHY is held in reset, it fails.

Current device tree setup:
tegra234-tpcca+p3701-0008_dts.txt (9.4 KB)
gpioinfo output for PG pins:
orin_gpioinfo.txt (676 Bytes)

A question I have now, is where should the phy-reset-gpio statement go within the device tree? In the Ethernet section, in the mdio section or in the phy section? I currently have it in the Ethernet section before the mdio is defined.

Thanks,
Jim

Jim,

Is there a reason why you have nvidia,mdio_addr = <2>; but father down you have mdio set at phy@3 and reg=<3>;

Hi @woods.little,

It was an oversight. Changed it to 3 and still had the same issues.

I also found that the DP83869 module was not being compiled in the output directory of the kernel build and corrected that today. Still had same issues with the reset line.

A coworker tried commenting out the line that maps the phy-reset-gpio. The reset line did not toggle at any point but was driven high so the PHY was responding to all MDIO commands. It ended up loading the eth0 device. Assigned a static IP address manually and was able to ping in both directions with a laptop connected to the Orin module. I verified this worked on a second setup we have.

dmesg does give a non-fatal error about not having a gpio assigned to reset, but it does work.

I also noticed that MDIO appears to be sending two messages over and over. Did not decode them yet - will do that in the morning.

@WayneWWW - any clues as to why I am having reset line issues with the phy-reset-gpio mapped?

Thanks,
Jim

A coworker tried commenting out the line that maps the phy-reset-gpio. The reset line did not toggle at any point but was driven high so the PHY was responding to all MDIO commands. It ended up loading the eth0 device. Assigned a static IP address manually and was able to ping in both directions with a laptop connected to the Orin module. I verified this worked on a second setup we have.

So what was the difference of reset line pin between working and not working case there?

Hi Woods,

When the gpio line is not mapped, I am assuming the MAC does not control the PHY Reset line, so the only reset it gets is when our carrier card is powered on. It then goes high - which for the PHY we are using, is a non reset state and allows us to operate in a normal matter after an IP address is assigned.

When the gpio line is mapped, no matter how I have the active-high/active-low setting set, the GPIO PHY Reset acts in the active high mode and then when the PHY does not respond to MDIO commands, it goes low permanently (assuming that the MAC driver shuts it off since it did not respond).

Not mapping the GPIO in the device tree is a workaround, but I don’t think that is a good solution. I would really like to hear from NVIDIA on suggested methods to debug this issue.

Thanks,
Jim

So have you adjusted the time that GPIO keeps in high state to a very longer period by using device tree properties ?

Jim,

Do you make the gpio line an input in the pinmux and tristate the line?

Hi All,

Here is the pinmux entry for my PHY Interrupt and PHY reset lines:

soc_gpio17_pg4 {
    nvidia,pins = "soc_gpio17_pg4";
    nvidia,function = "rsvd0";
    nvidia.pull = <TEGRA_PIN_PULL_UP>;   // Our schematic has this pulled up with a 10K resistor
    nvidia,tristate = <TEGRA_PIN_ENABLE>;
    nvidia,enable-input = <TEGRA_PIN_ENABLE>;
    nvidia,lpdr = <TEGRA_PIN_DISABLED>;
}

soc_gpio18_pg5 {
    nvidia,pins = "soc_gpio18_pg5";
    nvidia,function = "rsvd0";
    nvidia.pull = <TEGRA_PIN_PULL_NONE>; // No external pullup connected - inline 100Ohm resistor on reset line
    nvidia,tristate = <TEGRA_PIN_DISABLE>;
    nvidia,enable-input = <TEGRA_PIN_DISABLE>;
    nvidia,lpdr = <TEGRA_PIN_DISABLED>;
}

Out of curiosity, what does the function “rsvd0” mean? Some of the other gpio line sections have “rsvd1” and “rsvd2”.

Verified that the PG.04 line is in the gpio_input section of the gpio dtsi file.

@WayneWWW I changed the delay to 1 second like:

nvidia,phy-rst-pdelay-msec = <224>; /* msec */
nvidia,phy-rst-duration-usec = <1000000>; // <10000> - previous value I had tried was 10 msec
                                          // TI data sheet says 1 usec minimum pulse

I have included some scope images. First image is the reset going low for the first time when booting the Orin AGX module. (Note: the 1s Horizontal scale)

Second is showing the reset held low with MDIO traffic

Third is showing the line go high when I uninstall the nvethernet module manually from the command line (sudo rmmod nvethernet)

Fourth is showing the result of reloading the nvethernet module (sudo modprobe nvethernet) with the line reset going low for approximately 1 second, go high briefly the go low and hold it.

@WayneWWW Can this issue be tied to UEFI Network boot?

I noticed on the oscilliscope that when UEFI message are outputting in the serial window, that the PHY Reset (when mapped in GPIO) is pulled low when all of the MDIO traffic starts. UEFI does not complete until this is done then hands off to the kernel.

Thanks

You could disable UEFI net boot by split the device tree in use.

Let UEFI device tree set eqos to disabled and kernel device tree still sets it to enabled.

Thus, UEFI won’t involve with eqos and only kernel will.

You could check your board config and find DTB_FILE first. This is your kernel dtb file.
There would be another field which is directly equal to DTB_FILE, and that one is UEFI dtb.

I am not having any luck compiling in either the docker container or outside the docker container to customize it. It crashes during boot and then restarts.

I then split the device tree. I am able to turn off the network interface activity during the UEFI boot. I do not see any activity on the PHY reset line or MDIO/MDC lines when the kernel boots. If I unload the nvethernet module and then reload it, the reset line for the PHY has the correct active low time and polarity, but I never see any MDIO/MDC activity, and therefore the EQOS interface is then disabled with an error about not communicating with the PHY.

I feel like I am close but do not understand how to proceed.

I got this working easily. Only difference is we used Marvel external PHY.

Refer to the abv link for more information. It might be some help.