Problem with RGMII Ethernet (88E1512) with JP 5.1.0 on AGX Orin (was working with JP 5.0.1DP and on Xavier)

Thank you.
The custom pinmux was attached on my 1st post.
As you can see in the post above, I was using tegra234-p3701-0004-p3737-0000.dtb for the 32GB module. I have only used one custom overlay. I have attached the overlay here:
tegra234-galaxy-overlay.dts (45.3 KB)

I have not modified anything for the bootloader dtb (and it uses the -0004- version for the 32GB module).

My previous post did not include all the relevant output from flash.sh. See the remaining part here.

copying recdtbfile(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/kernel/dtb/tegra234-p3701-0004-p3737-0000.dtb.rec)... done.
Existing sosfile(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/mb1_t234_prod.bin) reused.
Existing tegraboot(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/mb2_t234.bin) reused.
Existing cpu_bootloader(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/mb2_t234.bin) reused.
Existing mb2blfile(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/mb2_t234.bin) reused.
Existing xusbfile(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/xusb_t234_prod.bin) reused.
Existing dcefile(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/display-t234-dce.bin) reused.
Existing nvdecfile(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/nvdec_t234_prod.fw) reused.
Existing psc_rf(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/psc_rf_t234_prod.bin) reused.
Existing mb2_rf(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/mb2rf_t234.bin) reused.
Existing mb1file(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/mb1_t234_prod.bin) reused.
Existing bpffile(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/bpmp_t234-TE990M-A1_prod.bin) reused.
copying bpfdtbfile(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/t186ref/tegra234-bpmp-3701-0004-3737-0000.dtb)... done.
Existing scefile(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/camera-rtcpu-sce.img) reused.
Existing camerafw(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/camera-rtcpu-t234-rce.img) reused.
Existing apefile(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/adsp-fw.bin) reused.
Existing spefile(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/spe_t234.bin) reused.
Existing wb0boot(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/sc7_t234_prod.bin) reused.
Existing tosfile(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/tos-optee_t234.img) reused.
Existing eksfile(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/eks_t234.img) reused.
copying dtbfile(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/kernel/dtb/tegra234-p3701-0004-p3737-0000.dtb)... done.
Copying nv_boot_control.conf to rootfs
	populating kernel to rootfs... done.
	populating initrd to rootfs... done.
	populating kernel_tegra234-p3701-0004-p3737-0000.dtb to rootfs... done.
Making system.img... 
	Setting "FDT /boot/dtb/kernel_tegra234-p3701-0004-p3737-0000.dtb" successfully in the extlinux.conf...done.
	populating rootfs from /home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/rootfs ... 	populating /boot/extlinux/extlinux.conf ... done.
	Sync'ing system.img ... done.
	Converting RAW image to Sparse image... done.
system.img built successfully. 
Not signing of kernel-dtb
Existing tbcfile(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/uefi_jetson.bin) reused.
	Sync'ing esp.img ... done.
copying tbcdtbfile(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/kernel/dtb/tegra234-p3701-0004-p3737-0000.dtb)... done.
copying cfgfile(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/t186ref/cfg/flash_t234_qspi_sdmmc.xml) to flash.xml... done.
Existing flashapp(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/tegraflash.py) reused.
copying overlay_dtb(/home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/kernel/dtb/tegra234-galaxy-overlay.dtbo)... done.
./tegraflash.py  --bl uefi_jetson_with_dtb.bin  --odmdata gbe-uphy-config-0,hsstp-lane-map-3,nvhs-uphy-config-0,hsio-uphy-config-0  --overlay_dtb tegra234-galaxy-overlay.dtbo,  --bldtb tegra234-p3701-0004-p3737-0000.dtb --applet mb1_t234_prod.bin --cmd "flash; reboot"  --cfg flash.xml --chip 0x23 --concat_cpubl_bldtb --cpubl uefi_jetson.bin --minratchet_config tegra234-mb1-bct-ratchet-p3701-0000.dts --device_config tegra234-mb1-bct-device-p3701-0000.dts --misc_config tegra234-mb1-bct-misc-p3701-0000.dts --pinmux_config Orin-galaxy-pinmux.dtsi --gpioint_config tegra234-mb1-bct-gpioint-p3701-0000.dts --pmic_config tegra234-mb1-bct-pmic-p3701-0000.dts --pmc_config Orin-galaxy-padvoltage-default.dtsi --deviceprod_config tegra234-mb1-bct-cprod-p3701-0000.dts --prod_config tegra234-mb1-bct-prod-p3701-0000.dts --scr_config tegra234-mb2-bct-scr-p3701-0000.dts --wb0sdram_config tegra234-p3701-0000-wb0sdram-l4t.dts --br_cmd_config tegra234-mb1-bct-reset-p3701-0000.dts --dev_params tegra234-br-bct-p3701-0000.dts,tegra234-br-bct_b-p3701-0000.dts --mb2bct_cfg tegra234-mb2-bct-misc-galaxy-orin.dts  --bins "psc_fw pscfw_t234_prod.bin; mts_mce mce_flash_o10_cr_prod.bin; mb2_applet applet_t234.bin; mb2_bootloader mb2_t234.bin; xusb_fw xusb_t234_prod.bin; dce_fw display-t234-dce.bin; nvdec nvdec_t234_prod.fw; bpmp_fw bpmp_t234-TE990M-A1_prod.bin; bpmp_fw_dtb tegra234-bpmp-3701-0004-3737-0000.dtb; sce_fw camera-rtcpu-sce.img; rce_fw camera-rtcpu-t234-rce.img; ape_fw adsp-fw.bin; spe_fw spe_t234.bin; tos tos-optee_t234.img; eks eks_t234.img"  --sdram_config tegra234-p3701-0000-sdram-l4t.dts  --cust_info custinfo_out.bin  --secondary_gpt_backup  --bct_backup  --boot_chain A 
saving flash command in /home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/flashcmd.txt
saving Windows flash command to /home/mandre/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/bootloader/flash_win.bat

Are you suggesting that I use the wrong version of some of the files? Thank you!

1 Like

@WayneWWW do you have any additional inputs?

Are we sure that overlay is in use? How about we put them in the common dtb file first instead of dtbo?

Yes, I am sure. See also the device-tree-dump attached to the initial post. This dump was created on the running system using

dtc /sys/firmware/devicetree/base/

Sorry that I can only suggest you to looks into other user’s post and check what is missing on your side.

We’ve checked this 88E1512 on many other end users/partners already.

Hi Wayne
I have been doing all those changes and have it well document in my 1st post in this thread.
I went back and also confirmed the pinmux by reading some of the registers at runtime. As you can see the registers match the expectation (eqos mode, rsvd0 for interrupts, etc)

#eqos td3
sudo busybox devmem 0x02445000
0x00002400
#rst
sudo busybox devmem 0x02434070
0x00000058
#int
sudo busybox devmem 0x02434078
0x00000000

In my previous post you replied that there has been no change to RGMII between 5.0.1 and 5.0.2

I can only see that the device tree (read back from the device) is different. I especially see that >=5.0.2 I have “interconnects” nodes which I can’t see in 5.0.1. So at least something has changed.
I wonder if anything regarding the DMA architecture has changed…
Here is the device tree dump from 5.0.1:
dts.txt (328.9 KB)

Let me summarize my status:

  • I have confirmed that the pinmux and dtb get applied.
  • I can see the device being recognized (which to my understanding means that at least mdio is fine). Also I can see the connection status and it correctly negotiates the data rate.
  • I see the interrupts arriving
  • I see some packets being received and sent, but the actual communication doesn’t work.
  • The loopback test (ethertool -t) fails with -110 (timeout)
  • The very same hardware works with Xavier on JP 5.1 and on Orin with JP 5.0.1

My questions:

  • What can cause the packet counts (seen using ifconfig) rise while (not all?) packets are actually received
  • Again: What has changed from 5.0.1 to 5.0.2
  • How is the mechanism of moving the data from the phy to the system. Who triggers the DMA transfers. Is there any way to debug this?
  • Any suggestions for further debugging?

Thank you Marc

Hi Marc,

There is no direct answer from me about “what was changed”. I don’t think those changes here really matter to your issue. Most cases we helped to bring up RGMII is based on jp5.0.2. Please make sure your dtb change really follows the jp5.1 document (because jp5.0.2 doc has some problem in pinmux configuration).

or use this one instead. https://developer.nvidia.com/downloads/adaptation-and-bringup-jetson-agx-orinpdf

And Please make sure you did the full flash instead of only flash kernel or dtb.

I would like to confirm what is the exact issue we got here. For example, if just port to port connection, is this port able to ping another side? There is no error log from your dmesg.

Hi Wayne
I did all of that. This is also why I provide dtb dumps and read back the pinmux registers to confirm on the installed system that the changes were applied.

No, the port is not able to ping (and DHCP doesn’t work. I do see some random packets with tcpdump, but I couldn’t figure exactly what I see and what I don’t see.
The same setup (not changing a cable or anything…) works with 5.0.1 and Xavier.

See my note that even the ethertool -t fails. Thus I don’t think I have to search outside the Jetson/baseboard.

#ethtool -t eth0

The test result is FAIL
The test extra info:
 1. MAC Loopback			 0
 2. PHY Loopback			 -110
 3. MMC Counters			 0

Could you share the /proc/interrupts where your jp5.0.1 can work fine?

Here is the interrupts with jp5.0.1:
interrupts.txt (22.1 KB)
ethtool_t.txt (112 Bytes)

Interrupts with JP5.1:
interrupts.txt (21.3 KB)
ethtool_t.txt (133 Bytes)

Could you also provide the dts of jp5.1?

It is in the first post:
device-tree-dump.txt (523.1 KB)

What is the pin of this nvidia,phy-reset-gpio = <0x04 0x35 0x00>?

nvidia,phy-reset-gpio = <&tegra_main_gpio TEGRA234_MAIN_GPIO(G, 5) 0>;

interrupts = <TEGRA234_MAIN_GPIO(G, 4) IRQ_TYPE_LEVEL_LOW>;

Are you able to see the status of pin G5 in /sys/kernel/debug/gpio?

Yes

 gpio-388 (PG.05               |phy_reset           ) out hi 

@WayneWWW any update? Thank you! Marc

I don’t have device to check for now. But I think you can read the pinmux register and gpio register for those pins and confirm if they are same as jp5.0.1DP.

Checking pinmux register is not sufficient to prove whether this pin is in GPIO or SFIO because GPIO controller has higher priority.

For example, if you read pinmux reg and see it as SFIO, however, GPIO controller sets it as GPIO in the meantime, then it will still be GPIO.

I did find one relevant difference. PADCTL_EQOS_EQOS_TXC_0 is set to 0x00002050 for JP 5.1, while it is 0x00002400 for JP 5.0.1. Reading the TRM, 0x2400 makes more sense.

As soon as I manually set it to 0x00002400, the Ethernet starts working!

busybox devmem 0x02445058 w 0x00002400

I found that the invalid configuration of a completely different module (USB) caused the GPIO controller to grab pin E.00. I was not aware of the priorities of the gpio controller vs. pinmux. Good to know.
Fixing that pin also solved the Ethernet issue. :-)

Thank you

1 Like

Sorry. One question here.

Is that added by you (Pin E.00 as GPIO) or it is added by our BSP by default?