We are running multiple Thors with the MGBE NICs in the factory-default 10Gx4 configuration. For the most part everything is smooth sailing. However TX Pause Frame functionality is NOT normal out of the box. Is there a clear path forward for Thor to allow the 4 MGBE ports to each respect Ethernet Pause Frames from their link partner, when the link partner issues partner issues pause frames to the Thor’s TX ports. We are operating in 10Gx4 (not 25Gx4) if that matters.
Thank you in advance!
Hi,
Could you share what is your method to identify TX pause frame is not working?
Out of the box Thor with the r38.4.0 installed and the MGBEs in default 10Gx4 mode we see
$ sudo ethtool -A mgbe0_0 tx on
netlink error: Operation not supported
Obviously, we don’t expect Thor’s MGBE to respect the link partner’s Pause Frames if they are not supported. We require that support.
We came across what looks like a similar issue with the MGBE on Orin, but are not clear on how to adapt that to Thor. Thinking some kind of device tree editing is needed?
It would be super-helpful to know the exact steps needed to enable TX pause frames on Thor’s MGBE ports.
This might be relevant.
nvidia-oot/drivers/net/ethernet/nvidia/nvethernet/ether_linux.c
#ifndef OSI_STRIPPED_LIB
if ((pdata->osi_core->pause_frames == OSI_PAUSE_FRAMES_ENABLE)
&& (phydev->pause || phydev->asym_pause)) {
ioctl_data.cmd = OSI_CMD_FLOW_CTRL;
ioctl_data.arg1_u32 = pdata->osi_core->flow_ctrl;
ret = osi_handle_ioctl(pdata->osi_core, &ioctl_data);
if (ret < 0) {
netdev_err(dev, "Failed to set pause frame\n");
return;
}
}
...
#ifndef OSI_STRIPPED_LIB
/* Read Pause frame feature support */
ret = of_property_read_u32(np, "nvidia,pause_frames",
&pdata->osi_core->pause_frames);
if (ret < 0) {
dev_info(dev, "Failed to read nvida,pause_frames, so"
" setting to default support as disable\n");
pdata->osi_core->pause_frames = OSI_PAUSE_FRAMES_DISABLE;
}
source/nvidia-oot/Documentation/devicetree/bindings/platform/tegra/tegra-nvethernet.yaml
nvidia,rx_frames:
$ref: "/schemas/types.yaml#/definitions/uint32"
minimum: 0x40
maximum: 0x40
nvidia,tx_frames:
$ref: "/schemas/types.yaml#/definitions/uint32"
minimum: 0x10
maximum: 0x10
Based on the two dts excerpts at the bottom; you could try:
dtc -I dtb -O dts -o kernel_tegra264-p4071-0000+p3834-0008-nv.dts /boot/dtb/kernel_tegra264-p4071-0000+p3834-0008-nv.dtb
nano kernel_tegra264-p4071-0000+p3834-0008-nv.dts
go to this node and add pause_frames near tx_frames
ethernet@a808a10000 {
nvidia,pause_frames = <0x1>;
Save it and dtc compile it
sudo dtc -I dts -O dtb -o /boot/pause_kernel_tegra264-p4071-0000+p3834-0008-nv.dtb kernel_tegra264-p4071-0000+p3834-0008-nv.dts
Then edit /boot/extlinux/extlinux.conf and add a new LABEL that uses:
FDT /boot/pause_kernel_tegra264-p4071-0000+p3834-0008-nv.dtb
reboot and see if it sticks.
sudo dtc -I fs -O dts /sys/firmware/devicetree/base | grep -A54 “ethernet@a808a10000 {”
Linux_for_Tegra/source/hardware/nvidia/t23x/nv-public/nv-platform/tegra234-p3737-0000.dts
mgbe0: ethernet@6800000 {
nvidia,mac-addr-idx = <0>;
nvidia,max-platform-mtu = <16383>;
/* 1=enable, 0=disable */
nvidia,pause_frames = <1>;
phy-handle = <&mgbe0_aqr113c_phy>;
/* 0:XFI 10G, 1:XFI 5G, 2:USXGMII 10G, 3:USXGMII 5G */
nvidia,phy-iface-mode = <0>;
nvidia,phy-reset-gpio = <&gpio TEGRA234_MAIN_GPIO(Y, 1) 0>;
nvidia,mdio_addr = <0>;
Linux_for_Tegra/source/hardware/nvidia/t264/nv-public/nv-platform/tegra264-p4071-0000.dtsi
/* MGBE 0 */
mgbe0: ethernet@a808a10000 {
nvidia,mac-addr-idx = <1>;
nvidia,uphy-gbe-mode = <1>;
nvidia,phy-iface-mode = <0x0>;
nvidia,max-platform-mtu = <9000>;
nvidia,pps_op_ctrl = <8>;
nvidia,if-name = "mgbe0_0";
nvidia,force-restart-lane-bringup = <1>;
Thank you @whitesscott , that DT edit sounds promising, though it is concerning why Nvidia made the decision to leave pause frames off in the Thor production release. Curious if @WayneWWW thinks we are going off the reservation with this change?
We tried to do just what you described, i have done that many times on the AGX Orin. However the named dtb does not exist int /boot/. (See below). Also the /boot/extlinux/extlinux.conf does not reference a device tree. We are confused as to which .dtb is actually being used at boot.
cat /boot/extlinux/extlinux.conf
TIMEOUT 30
DEFAULT primary
MENU TITLE L4T boot options
LABEL primary
MENU LABEL primary kernel
LINUX /boot/Image
INITRD /boot/initrd
APPEND ${cbootargs} root=PARTUUID=6a98fbac-e864-479a-a7c4-f482c4ad65e6 rw rootwait rootfstype=ext4 mminit_loglevel=4 earlycon=tegra_utc,mmio32,0xc5a0000 console=tty0 console=ttyUTC0,115200 clk_ignore_unused firmware_class.path=/etc/firmware fbcon=map:0 nospectre_bhb efi=runtime bl_prof_dataptr=6225920@0x2008010000 bl_prof_ro_ptr=65536@0x20
ls /boot/kernel_tegra264-p4071-*
ls: cannot access ‘/boot/kernel_tegra264-p4071-*’: No such file or directory
ls /boot/*.dtb
/boot/tegra234-p3737-0000+p3701-0000.dtb /boot/tegra234-p3737-0000+p3701-0008-nv.dtb /boot/tegra234-p3768-0000+p3767-0000-nv-super.dtb /boot/tegra234-p3768-0000+p3767-0004-nv.dtb
/boot/tegra234-p3737-0000+p3701-0000-nv.dtb /boot/tegra234-p3740-0002+p3701-0008.dtb /boot/tegra234-p3768-0000+p3767-0001-nv.dtb /boot/tegra234-p3768-0000+p3767-0004-nv-super.dtb
/boot/tegra234-p3737-0000+p3701-0004-nv.dtb /boot/tegra234-p3740-0002+p3701-0008-nv.dtb /boot/tegra234-p3768-0000+p3767-0001-nv-super.dtb /boot/tegra234-p3768-0000+p3767-0005.dtb
/boot/tegra234-p3737-0000+p3701-0005-nv.dtb /boot/tegra234-p3768-0000+p3767-0000.dtb /boot/tegra234-p3768-0000+p3767-0003-nv.dtb /boot/tegra234-p3768-0000+p3767-0005-nv.dtb
/boot/tegra234-p3737-0000+p3701-0008.dtb /boot/tegra234-p3768-0000+p3767-0000-nv.dtb /boot/tegra234-p3768-0000+p3767-0003-nv-super.dtb /boot/tegra234-p3768-0000+p3767-0005-nv-super.dtb
I believe it is whatever is in /boot/dtb/*.dtb and there’s an identical copy in /boot/
For my Thor dev kit it is
/boot/dtb/kernel_tegra264-p4071-0000+p3834-0008-nv.dtb
Yes, using “nvidia,pause_frames = <0x1>;” in the device tree is expected.
Thank you @WayneWWW and @whitesscott . We’re going to pull the m.2, put in a new drive, do a clean jetson-r38.4.0 install, update the DT as suggested, and retest for pause frame function.
It would be nice in a future release if Pause Frames were on by default.
With the suggestions from @whitesscott and @WayneWWW , Thor’s MGBE Pause Frames are good.
john@ar-thor2:~$ ethtool --show-pause mgbe0_0
Pause parameters for mgbe0_0:
Autonegotiate: on
RX: on
TX: on
Thanks so much for your help and we can close this thread!