Xavier AGX custom board with bcm53125 switch lock kernel when eth0 is up

Robotics & Edge Computing Jetson Systems Jetson AGX Xavier
,
1

Hi all,

I’ve got a custom carrier board for Xavier AGX which has Marvel PHY replaced with BCM53125 switch, using RGMII connection and MDIO for configuration.
I’m using JetPack 5.1.2 (R35.4.1).

When I try to set eth0 up, the kernel task IP is blocked, I’ve traced it down to see that the kernel is trying to lock rtnl_mutex while it is already taken.

Here are the details:

My device tree modification (tegra194-platforms-eqos.dtsi):

/ {
        eth0: ethernet@2490000 {
		interrupts =	<0 194 0x4>,    /* common */
				<0 186 0x4>,    /* tx0 */
				<0 190 0x4>;    /* rx0 */
		/* rxq_enable_ctrl = <rx0 rx1 rx2 rx3>
		 * 0x0 = Not enabled, 0x1 = Enabled for AV
		 * 0x2 = Enabled for Legacy, 0x3 = Reserved
		 */
		nvidia,rxq_enable_ctrl = <2>;
		nvidia,num-dma-chans = <1>;
		nvidia,dma-chans = <0>;
		nvidia,num-mtl-queues = <1>;
		nvidia,mtl-queues = <0>;
		nvidia,rx-queue-prio = <0x2>;
		nvidia,tx-queue-prio = <0x0>;
		/* 1=enable, 0=disable */
		nvidia,pause_frames = <1>;
		nvidia,phy-reset-gpio = <&tegra_main_gpio TEGRA194_MAIN_GPIO(G, 5) 0>;
		phy-mode = "rgmii-id";
		// phy-handle = <&qweqwe>;
		nvidia,max-platform-mtu = <9216>;
		nvidia,skip_mac_reset = <0>;

		fixed-link {
			speed = <1000>;
			duplex-full;
		};

		mdio {
			compatible = "nvidia,eqos-mdio";
			#address-cells = <1>;
			#size-cells = <0>;

			switch0: switch@1e {
				compatible = "brcm,bcm53125";
				reg = <30>;
				#address-cells = <1>;
				#size-cells = <0>;

				interrupt-parent = <&tegra_main_gpio>;
				interrupts = <TEGRA194_MAIN_GPIO(G, 4) IRQ_TYPE_LEVEL_LOW>;

				// nvidia,phy-rst-duration-usec = <1500>;
				// nvidia,phy-rst-pdelay-msec = <6>;

				ports {
					port0@0 {
						reg = <0>;
						label = "lan1";
					};
					port1@1 {
						reg = <1>;
						label = "lan2";
					};
					qweqwe: port8@8 {
						reg = <8>;
						label = "cpu";
						phy-mode = "rgmii-id";
						ethernet = <&eth0>;
						// nvidia,phy-rst-duration-usec = <1500>;
						// nvidia,phy-rst-pdelay-msec = <6>;
						fixed-link {
							speed = <1000>;
							duplex-full;
						};
					};
				};
			};

		};
	};

	thermal-zones {
		CPU-therm {
			trips {
				MAKE_EQOS_TRIP(m40, -40000, 5000);
				MAKE_EQOS_TRIP(m5, -5000, 5000);
				MAKE_EQOS_TRIP(p30, 30000, 5000);
				MAKE_EQOS_TRIP(p65, 65000, 5000);
				MAKE_EQOS_TRIP(p100, 100000, 5000);
			};

			cooling-maps {
				MAP_EQOS(m40, 1);
				MAP_EQOS(m5, 2);
				MAP_EQOS(p30, 3);
				MAP_EQOS(p65, 4);
				MAP_EQOS(p100, 5);
			};
		};
	};
};

I’ve enabled LOCKDEP and other debbuging options for tracing the issue, here is the dmesg message:

[   19.657930] bcm53xx 2490000.ethernet:1e: found switch: BCM53125, rev 4
[   19.674885] 
[   19.675130] ============================================
[   19.675265] WARNING: possible recursive locking detected
[   19.675436] 5.10.120-l4t-r35.4.ga+g76678311c10b #1 Not tainted
[   19.675595] --------------------------------------------
[   19.675752] ip/464 is trying to acquire lock:
[   19.675877] ffffd9dd115886b0 (rtnl_mutex){+.+.}-{3:3}, at: rtnl_lock+0x24/0x30
[   19.676117] 
[   19.676117] but task is already holding lock:
[   19.676299] ffffd9dd115886b0 (rtnl_mutex){+.+.}-{3:3}, at: rtnl_lock+0x24/0x30
[   19.676538] 
[   19.676538] other info that might help us debug this:
[   19.676720]  Possible unsafe locking scenario:
[   19.676720] 
[   19.676876]        CPU0
[   19.676949]        ----
[   19.677022]   lock(rtnl_mutex);
[   19.677121]   lock(rtnl_mutex);
[   19.677216] 
[   19.677216]  *** DEADLOCK ***
[   19.677216] 
[   19.677374]  May be due to missing lock nesting notation
[   19.677374] 
[   19.677561] 3 locks held by ip/464:
[   19.677667]  #0: ffffd9dd115886b0 (rtnl_mutex){+.+.}-{3:3}, at: rtnl_lock+0x24/0x30
[   19.679802]  #1: ffff374819c31990 (&dev->mutex){....}-{3:3}, at: __device_attach+0x48/0x17c
[   19.688104]  #2: ffffd9dd1159e6a0 (dsa2_mutex){+.+.}-{3:3}, at: dsa_register_switch+0x50/0xbf0
[   19.696850] 
[   19.696850] stack backtrace:
[   19.700981] CPU: 7 PID: 464 Comm: ip Not tainted 5.10.120-l4t-r35.4.ga+g76678311c10b #1
[   19.709360] Hardware name: Unknown Jetson-AGX/Jetson-AGX, BIOS v35.4.1 08/04/2023
[   19.716277] Call trace:
[   19.718899]  dump_backtrace+0x0/0x1ac
[   19.722573]  show_stack+0x30/0x40
[   19.726243]  dump_stack+0xec/0x154
[   19.729310]  __lock_acquire+0x1130/0x1a5c
[   19.733509]  lock_acquire.part.0+0xe4/0x21c
[   19.737724]  lock_acquire+0x64/0x80
[   19.741210]  __mutex_lock+0x98/0x490
[   19.744886]  mutex_lock_nested+0x64/0x74
[   19.748924]  rtnl_lock+0x24/0x30
[   19.752060]  devlink_nl_port_fill+0xb4/0x544
[   19.756780]  devlink_port_notify+0x6c/0xd0
[   19.760979]  devlink_port_register+0x16c/0x1b4
[   19.765186]  dsa_port_devlink_setup+0xb0/0x100
[   19.765201]  dsa_register_switch+0x67c/0xbf0
[   19.779707]  b53_switch_register+0x208/0x36c
[   19.783824]  b53_mdio_probe+0xe4/0x15c
[   19.787841]  mdio_probe+0x40/0x74
[   19.790997]  really_probe+0xf8/0x4b0
[   19.794585]  driver_probe_device+0x60/0xbc
[   19.798786]  __device_attach_driver+0xb4/0x104
[   19.803349]  bus_for_each_drv+0x8c/0xe0
[   19.807009]  __device_attach+0xdc/0x17c
[   19.810942]  device_initial_probe+0x28/0x3c
[   19.815143]  bus_probe_device+0xa4/0xb0
[   19.818647]  device_add+0x30c/0x790
[   19.822321]  mdio_device_register+0x44/0x6c
[   19.826353]  of_mdiobus_register+0x1c4/0x340
[   19.830981]  ether_open+0x174/0xc80
[   19.834656]  __dev_open+0x110/0x1c4
[   19.838254]  __dev_change_flags+0x1a8/0x230
[   19.842443]  dev_change_flags+0x40/0x80
[   19.845947]  devinet_ioctl+0x4ec/0x7d0
[   19.849968]  inet_ioctl+0x1f8/0x2a0
[   19.853466]  sock_do_ioctl+0x58/0x16c
[   19.857070]  sock_ioctl+0x324/0x520
[   19.860643]  __arm64_sys_ioctl+0xb0/0xf0
[   19.864236]  el0_svc_common.constprop.0+0x84/0x1d0
[   19.869481]  do_el0_svc+0x38/0xa0
[   19.872548]  el0_svc+0x1c/0x30
[   19.875695]  el0_sync_handler+0x100/0x110
[   19.879811]  el0_sync+0x16c/0x180

After adding some prints, here is the chain that is happening:

Configuring network interfaces... [   47.585830] @@@@@@@@@@@@@@@@@@@@ rtnl_lock
[   47.586506] ^^^^^^^^^^^^^^^^^^^^ rtnl_unlock
[   47.599169] ####################### devinet_ioctl 1021
[   47.599556] @@@@@@@@@@@@@@@@@@@@ rtnl_lock
[   47.599863] ####################### dev_change_flags 8559
[   47.600327] ####################### __dev_change_flags 8456
[   47.600796] ####################### __dev_open 1508
[   47.601701] ^^^^^^^^^^^^^^^^^^^^ rtnl_unlock
[   47.668505] ####################### devinet_ioctl 1021
[   47.668896] @@@@@@@@@@@@@@@@@@@@ rtnl_lock
[   47.669172] ####################### dev_change_flags 8559
[   47.669527] ####################### __dev_change_flags 8456
[   47.669955] ####################### __dev_open 1508
[   47.670307] ####################### ether_open 2497
[   47.680883] ####################### ether_mdio_register 2433
[   47.681280] ####################### of_mdiobus_register 256
[   47.681701] ####################### device_add 2854
[   47.684833] ####################### device_add 2854
[   47.685697] ####################### bus_probe_device 489
[   47.686419] ####################### bus_probe_device 489
[   47.687214] ####################### of_mdiobus_register_device 157
[   47.687765] libphy: ####################### mdio_device_register 80
[   47.688323] ####################### device_add 2854
[   47.689580] ####################### device_add 2854
[   47.690254] ####################### bus_probe_device 489
[   47.690919] ####################### bus_probe_device 489
[   47.691518] ####################### bus_probe_device 495
[   47.692008] ####################### device_initial_probe 980
[   47.692507] ####################### __device_attach 906
[   47.692990] ####################### bus_for_each_drv 428
[   47.694542] ####################### driver_probe_device 740
[   47.700195] ####################### really_probe 506
[   47.705083] libphy: ####################### mdio_probe 155
[   47.710243] ####################### b53_mdio_probe 296
[   47.715502] ####################### b53_switch_register 2729
[   47.722002] bcm53xx 2490000.ethernet:1e: found switch: BCM53125, rev 4
[   47.744192] @@@@@@@@@@@@@@@@@@@@ rtnl_lock
[   47.744732] 
[   47.745076] ============================================
[   47.745508] WARNING: possible recursive locking detected

Userspace does IOCTL which before handling locks rtnl_lock, then proceeds to bring eth0 up, it goes through the chain, probes MDIO, finds the switch, loads b53 driver, further does probing, and forwards to Linux DSA, which tries to register port, which in turn again calls rtnl_lock…

I’m running out of ideas about where even to look for faulty behaviour. Is there a way to defer the probing of the MDIO so it is not tied with EQOS “up”? Or to initialize the whole driver chain before userspace can trigger IOCTL?

3

Could you also share the original full uart log with the kernel panic and full dts?

4

Hi WayneWWW, thanks for looking into this.

I’m attaching decompiled dts and full uart log:
capture.txt (82.0 KB)
decompiled.dts.txt (483.1 KB)

In the capture.txt, after the second 25, I tried to log in but due to the kernel being stuck, the prompt didn’t show up.

I think I’ve found one trail of this bug. I think that the whole problem is triggered by the position in which function of_mdiobus_register is called.
I’ve checked other platform vendors, and they seem to call this inside ethernet probe, not open as nvethernet does. Look:

Xilinx
macb_probe → macb_mii_init → of_mdiobus_register

iMX
fec_probe → fec_enet_mii_init → of_mdiobus_register

well, even your old driver eqos does it in probe:
eqos_probe → eqos_mdio_register → of_mdiobus_register

So, do you know the reason why nvetherent does it in open?
ether_open → ether_mdio_register → of_mdiobus_register

This logic seems correct as the driver probe method is called directly from kernel space, while open is a result of IOCTL which is protected by rtnl_lock.

7

@WayneWWW
Have you found my attached document useful?
What do you think about my investigation?

9

@WayneWWW

Can I get some response about this issue I have raised?

10

@WayneWWW

Can this problem get some attention?

12

I solved the problem by manipulating the position of the of_mdiobus_register call inside nvidia ethernet driver.

For hardware designers who need to have embedded switch in their carrier boards with xavier AGX:

Use SPI, I2C, or a different bus for configuring/probing the switch. The current driver does not support MDIO, and given how much attention my problem has received from Nvidia, it will probably be broken forever.

Also a heads up, not every switch configuration for DSA with hardware tagging will work with Xavier AGX due to broken packet filtering - also not advertised anywhere, so I do not recommend using the bcm53125 switch here (not a Broadcom fault!)

Check this thread for more info: Xavier AGX ethernet truncates RX frames