Thor SFP bring-up

Robotics & Edge Computing Jetson Systems Jetson Thor
1

Hi all,

We have designed a Thor carrier with just one MGBE UPHY lane broken out into an SFP, instead of an SFP-28. What device tree modifications need to be made to adapt the nvidia QSFP implementation to work with a single SFP? I cannot find where the pin assignments for the QSFP are made in the dts.

Thank you

6

Hi,
Any updates on this? I am having a similar issue.

7

As mentioned in the Jetson Thor Series Modules Interface Design Guide, the MGBE on Jetson Thor supports 2.5, 5, 10, or 25 Gbps. SFP is likely limited to a maximum of 1.25 Gbps and may not work with Jetson Thor’s MGBE. SFP+ supports up to 10 Gbps and should be used instead of SFP.

8

Hi,
So far I have been trying to use a 10G SFP28 DAC directly into a 10G capable switch, but I am not seeing any traffic over any of the MGBE interfaces. Are any changes required to the default QSFP setup provided with the Thor developer kit for this? We are not using the same GPIO as were used on the QSFP in the dev kit.

9

Are you following the Jetson Linux Developer Guide for enabling QSFP and checking for traffic?

Since you did not use the same GPIOs, have you programmed whatever was used accordingly? Is there a particular reason the GPIOs were remapped instead of following the Dev Kit?

11

Hi,
The QSFP28 and SFP28 GPIO assignments are not the same - the difference in GPIO functions are shown here. Additionally, we are using 2 SFP28 modules instead of 1 QSFP, so using all of the same GPIOs as the developer kit was not feasible. I was not able to find where nvethernet maps these GPIOs, so I have not be able to reasign them. Setting a static IP (suggested in a different post by I think NV’s Wayne) and forcing the interface up shows a couple of TX packets in ifconfig, but never an actual link. We’re a bit stuck here, do you have any leads on what we should try next?

12

10G SFP28 with DAC (Direct Attachment Cable) adds additional losses to the system, scaling with the length of the cable. The high losses could be one of the reasons why you are not able to link-up..

Can you try a Fiber transceiver (instead of DAC) for connecting the SFP28 to the 10G capable switch?

13

Just looking at my Thor and here are somethings that may be relevant.

Jetson Thor Series Modules Pinmux Template

In excel did a search for sfp and got these 4 rows.

Row259 J52	I2C0_CLK	GP190_I2C0_CLK	GEN0_I2C_SCL	unused_GEN0_I2C_SCL	GPIO3_PW.04	I2C0_CLK				B	B				I2C0_CLK	RSVD1	RSVD2	RSVD3	RSVD1	gen0_i2c_scl_pw4	vddio_gpiob				I2C0	GPIOB	ST/DD	BDGPIOLTL_DFCT6M16L_VD121812NC	20K	z	GPIO_PW4		NORMAL	NORMAL	DISABLE	0	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	VALID	GPIO3_PW.04	Output	Drive 0			Disable	Disable	Disable	Enable	Mid	Disable	1k							XFI0_RST_N (SOC_QSFP_RESET)	BM20		
Row260 J53	I2C0_DAT	GP191_I2C0_DAT	GEN0_I2C_SDA	unused_GEN0_I2C_SDA	GPIO3_PW.05	I2C0_DAT				B	B				I2C0_DAT	RSVD1	RSVD2	RSVD3	RSVD1	gen0_i2c_sda_pw5	vddio_gpiob				I2C0	GPIOB	ST/DD	BDGPIOLTL_DFCT6M16L_VD121812NC	20K	z	GPIO_PW5		NORMAL	TRISTATE	ENABLE		TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	VALID	GPIO3_PW.05	Input				Disable	Disable	Disable	Enable	Mid	Disable	1k							XFI0_INT_N (SOC_QSFP_INT)	BR20		
Row276 H17	GPIO62	GP273	SOC_GPIO273	unused_SOC_GPIO273	GPIO3_PY.03		I2S3_SDATA_OUT			B		O			RSVD0	I2S3_SDATA_OUT	RSVD2	RSVD3	RSVD0	soc_gpio273_py3	vddio_gpiod				DAP3	GPIOD	ST/DD	BDGPIOLTL_DFCT6M16L_VD121812NC	20K	pd	GPIO_PY3		NORMAL	TRISTATE	ENABLE		TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	VALID	GPIO3_PY.03	Input				Disable	Disable	Disable	Enable	Mid	Disable			10k					SOC_QSFP_RPSNT_N	BY12		
Row278 GPIO57	GP275	SOC_GPIO275	unused_SOC_GPIO275	GPIO3_PY.05		I2S3_LRCK			B		B			RSVD0	I2S3_LRCK	RSVD2	RSVD3	RSVD0	soc_gpio275_py5	vddio_gpiod				DAP3	GPIOD	ST/DD	BDGPIOLTL_DFCT6M16L_VD121812NC	20K	pd	GPIO_PY5		NORMAL	NORMAL	DISABLE	1	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE	VALID	GPIO3_PY.05	Output	Drive 1			Disable	Disable	Disable	Enable	Mid	Disable			10k					XFI3_RST_N (SOC_QSFP_LP_N)	CC12

/etc/systemd/nvpower.sh

function udev_mgbe_handler()
{
        # INTERFACE environment variable is initialized by udev based on
        # the interface detected by the kernel
        inf=$INTERFACE

        if [[ -e "/sys/class/net/$inf/carrier" ]]; then
                connected=$(cat "/sys/class/net/$inf/carrier")
                if [[ "$connected" -eq 0 ]]; then
                        ip link set dev "$inf" down
                fi
        fi
}

/etc/udev/rules.d/99-tegra-devices.rules

SUBSYSTEM=="net", KERNEL=="mgbe0_0", RUN+="/bin/bash /etc/systemd/nvpower.sh --mgbe"
SUBSYSTEM=="net", KERNEL=="mgbe1_0", RUN+="/bin/bash /etc/systemd/nvpower.sh --mgbe"
SUBSYSTEM=="net", KERNEL=="mgbe2_0", RUN+="/bin/bash /etc/systemd/nvpower.sh --mgbe"
SUBSYSTEM=="net", KERNEL=="mgbe3_0", RUN+="/bin/bash /etc/systemd/nvpower.sh --mgbe"

sudo cat /etc/netplan/90-NM-c19b17e2-f49c-3c83-99dc-b7cd9b26ca6e.yaml

network:
  version: 2
  ethernets:
    NM-c19b17e2-f49c-3c83-99dc-b7cd9b26ca6e:
      renderer: NetworkManager
      match:
        name: "mgbe0_0"
      addresses:
      - "192.168.1.60/24"
      dhcp6: true
      wakeonlan: true
      networkmanager:
        uuid: "c19b17e2-f49c-3c83-99dc-b7cd9b26ca6e"
        name: "mgbe0_0"
        passthrough:
          connection.autoconnect-priority: "-999"
          connection.timestamp: "1760080945"
          ethernet._: ""
          ipv4.address1: "192.168.1.60/24,192.168.1.1"
          ipv4.method: "manual"
          ipv6.addr-gen-mode: "default"
          ipv6.ip6-privacy: "-1"
          proxy._: ""
14

Thank you both for the information, it was helpful in determining that the setup we have should work. We have now managed to obtain link and transfer data over the DAC cable in both 10G and 25G mode. The issue was with a misbehaving clock signal on the MGBE UPHY, and fixing this allowed the link to come up.