Jetson TX2 Ethernet module doesn't work after update to L4T 32.6.1

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Hello.

After L4T update from 32.4.3 to 32.6.1 using “apt”, main Ethernet adapter stops working. I can see an error in dmesg, regarding the ‘eqos’ module (see log below).
If I flash the device using a L4T Framework via microUSB cable, then everything works correctly and there is no errors in dmesg.

What is the correct way to update device to 32.6.1 without flashing?

# uname -a
Linux localhost 4.9.140-tegra #1 SMP PREEMPT Thu Jun 25 21:22:12 PDT 2020 aarch64 aarch64 aarch64 GNU/Linux

# dmesg |grep -i eqos
[    1.076136] eqos 2490000.ether_qos: can't get pllrefe_vcoout clk (-2)
[    1.079419] eqos 2490000.ether_qos: Setting local MAC: 0 4 4b 8d 80 d5

# uname -a
Linux localhost 4.9.253-tegra #1 SMP PREEMPT Mon Jul 26 12:19:28 PDT 2021 aarch64 aarch64 aarch64 GNU/Linux

# dmesg |grep -i eqos
[    1.071882] eqos 2490000.ether_qos: can't get pllrefe_vcoout clk (-2)
[    1.074681] eqos 2490000.ether_qos: failed to read eqos_auto_cal_config_0_reg
[    1.074766] eqos 2490000.ether_qos: invalid settings : rx-frames must be enabled along with use_riwt in DT
[    1.089791] eqos: probe of 2490000.ether_qos failed with error -22
3

Can you describe your whole steps/commands to upgrade the system?

I’ve seen such symptom before and the cause of such issue is due to your kernel has been updated to rel-32.6.1, but your kernel dtb didn’t.

However, I don’t think this issue would happen if you follow the correct steps to do the system update.

4

Okay. I started with release 32.4.4:

/etc/lsb-release 
# cat /etc/lsb-release                                                                                            
DISTRIB_ID=Ubuntu                                                                                                                               
DISTRIB_RELEASE=18.04                                                                                                                           
DISTRIB_CODENAME=bionic                                                                                                                         
DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"
uname -a 
# uname -a                                                                                                        
Linux localhost 4.9.140-tegra #1 SMP PREEMPT Tue Oct 27 21:02:46 PDT 2020 aarch64 aarch64 aarch64 GNU/Linux
dmesg |grep -i eqos 
# dmesg |grep -i eqos
[    1.068305] eqos 2490000.ether_qos: can't get pllrefe_vcoout clk (-2)
[    1.071639] eqos 2490000.ether_qos: Setting local MAC: 0 4 4b 8d 80 d5
dpkg -l |grep -i l4t 
# dpkg -l |grep -i l4t   
ii  nvidia-l4t-apt-source          32.4.4-20201016123640               arm64        NVIDIA L4T apt source list debian package
ii  nvidia-l4t-bootloader          32.4.4-20201027211332               arm64        NVIDIA Bootloader Package
ii  nvidia-l4t-configs             32.4.4-20201016123640               arm64        NVIDIA configs debian package
ii  nvidia-l4t-core                32.4.4-20201016123640               arm64        NVIDIA Core Package
ii  nvidia-l4t-firmware            32.4.4-20201027211332               arm64        NVIDIA Firmware Package
ii  nvidia-l4t-init                32.4.4-20201027211332               arm64        NVIDIA Init debian package
ii  nvidia-l4t-initrd              32.4.4-20201016123640               arm64        NVIDIA initrd debian package
ii  nvidia-l4t-kernel              4.9.140-tegra-32.4.4-20201027211332 arm64        NVIDIA Kernel Package
ii  nvidia-l4t-kernel-dtbs         4.9.140-tegra-32.4.4-20201027211332 arm64        NVIDIA Kernel DTB Package
ii  nvidia-l4t-tools               32.4.4-20201027211332               arm64        NVIDIA Public Test Tools Package
ii  nvidia-l4t-xusb-firmware       32.4.4-20201016123640               arm64        NVIDIA USB Firmware Package
cat /etc/apt/sources.list.d/nvidia-l4t-apt-source.list 
# cat /etc/apt/sources.list.d/nvidia-l4t-apt-source.list
deb https://repo.download.nvidia.com/jetson/common r32.4 main
deb https://repo.download.nvidia.com/jetson/t186 r32.4 main

Then, I change r32.4 to r32.5 in the nvidia-l4t-apt-source.list, run apt update and apt dist-upgrade

apt list --upgradable 
# apt list --upgradable
Listing... Done
jetson-gpio-common/stable 2.0.16 arm64 [upgradable from: 2.0.11]
nvidia-l4t-apt-source/stable 32.5.1-20210219084708 arm64 [upgradable from: 32.4.4-20201016123640]
nvidia-l4t-bootloader/stable 32.5.1-20210614115015 arm64 [upgradable from: 32.4.4-20201027211332]
nvidia-l4t-configs/stable 32.5.1-20210219084708 arm64 [upgradable from: 32.4.4-20201016123640]
nvidia-l4t-core/stable 32.5.1-20210519111140 arm64 [upgradable from: 32.4.4-20201016123640]
nvidia-l4t-firmware/stable 32.5.1-20210519111140 arm64 [upgradable from: 32.4.4-20201027211332]
nvidia-l4t-init/stable 32.5.1-20210519111140 arm64 [upgradable from: 32.4.4-20201027211332]
nvidia-l4t-initrd/stable 32.5.1-20210614115015 arm64 [upgradable from: 32.4.4-20201016123640]
nvidia-l4t-kernel/stable 4.9.201-tegra-32.5.1-20210505093723 arm64 [upgradable from: 4.9.140-tegra-32.4.4-20201027211332]
nvidia-l4t-kernel-dtbs/stable 4.9.201-tegra-32.5.1-20210505093723 arm64 [upgradable from: 4.9.140-tegra-32.4.4-20201027211332]
nvidia-l4t-tools/stable 32.5.1-20210614115015 arm64 [upgradable from: 32.4.4-20201027211332]
nvidia-l4t-xusb-firmware/stable 32.5.1-20210614115015 arm64 [upgradable from: 32.4.4-20201016123640]
python-jetson-gpio/stable 2.0.16 arm64 [upgradable from: 2.0.11]
python3-jetson-gpio/stable 2.0.16 arm64 [upgradable from: 2.0.11]
u-boot-tools/bionic-updates 2020.10+dfsg-1ubuntu0~18.04.2 arm64 [upgradable from: 2019.07+dfsg-1ubuntu4~18.04.1]

After device reboot, I see error from eqos module in dmesg:

dmesg |grep -i eqos 
# dmesg |grep -i eqos
[    1.035969] eqos 2490000.ether_qos: can't get pllrefe_vcoout clk (-2)
[    1.038732] eqos 2490000.ether_qos: failed to read eqos_auto_cal_config_0_reg
[    1.038819] eqos 2490000.ether_qos: invalid settings : rx-frames must be enabled along with use_riwt in DT
[    1.054058] eqos: probe of 2490000.ether_qos failed with error -22
uname -a 
# uname -a
Linux localhost 4.9.201-tegra #1 SMP PREEMPT Wed May 5 09:32:25 PDT 2021 aarch64 aarch64 aarch64 GNU/Linux
dpkg -l |grep -i l4t 
# dpkg -l |grep -i l4t
ii  nvidia-l4t-apt-source          32.5.1-20210219084708               arm64        NVIDIA L4T apt source list debian package
ii  nvidia-l4t-bootloader          32.5.1-20210614115015               arm64        NVIDIA Bootloader Package
ii  nvidia-l4t-configs             32.5.1-20210219084708               arm64        NVIDIA configs debian package
ii  nvidia-l4t-core                32.5.1-20210519111140               arm64        NVIDIA Core Package
ii  nvidia-l4t-firmware            32.5.1-20210519111140               arm64        NVIDIA Firmware Package
ii  nvidia-l4t-init                32.5.1-20210519111140               arm64        NVIDIA Init debian package
ii  nvidia-l4t-initrd              32.5.1-20210614115015               arm64        NVIDIA initrd debian package
ii  nvidia-l4t-kernel              4.9.201-tegra-32.5.1-20210505093723 arm64        NVIDIA Kernel Package
ii  nvidia-l4t-kernel-dtbs         4.9.201-tegra-32.5.1-20210505093723 arm64        NVIDIA Kernel DTB Package
ii  nvidia-l4t-tools               32.5.1-20210614115015               arm64        NVIDIA Public Test Tools Package
ii  nvidia-l4t-xusb-firmware       32.5.1-20210614115015               arm64        NVIDIA USB Firmware Package

Dist upgrade log:

apt dist-upgrade 
# apt dist-upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  jetson-gpio-common nvidia-l4t-apt-source nvidia-l4t-bootloader
  nvidia-l4t-configs nvidia-l4t-core nvidia-l4t-firmware nvidia-l4t-init
  nvidia-l4t-initrd nvidia-l4t-kernel nvidia-l4t-kernel-dtbs nvidia-l4t-tools
  nvidia-l4t-xusb-firmware python-jetson-gpio python3-jetson-gpio u-boot-tools
15 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 106 MB of archives.
After this operation, 131 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 https://repo.download.nvidia.com/jetson/t186 r32.5/main arm64 nvidia-l4t-core arm64 32.5.1-20210519111140 [2,790 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main arm64 u-boot-tools arm64 2020.10+dfsg-1ubuntu0~18.04.2 [139 kB]
Get:3 https://repo.download.nvidia.com/jetson/t186 r32.5/main arm64 nvidia-l4t-tools arm64 32.5.1-20210614115015 [3,724 kB]
Get:4 https://repo.download.nvidia.com/jetson/t186 r32.5/main arm64 nvidia-l4t-kernel-dtbs arm64 4.9.201-tegra-32.5.1-2021050509
3723 [267 kB]
Get:5 https://repo.download.nvidia.com/jetson/t186 r32.5/main arm64 nvidia-l4t-kernel arm64 4.9.201-tegra-32.5.1-20210505093723 
[51.3 MB]
Get:6 https://repo.download.nvidia.com/jetson/t186 r32.5/main arm64 nvidia-l4t-apt-source arm64 32.5.1-20210219084708 [10.4 kB]
Get:7 https://repo.download.nvidia.com/jetson/t186 r32.5/main arm64 nvidia-l4t-configs arm64 32.5.1-20210219084708 [1,440 kB]
Get:8 https://repo.download.nvidia.com/jetson/t186 r32.5/main arm64 nvidia-l4t-init arm64 32.5.1-20210519111140 [66.8 kB]
Get:9 https://repo.download.nvidia.com/jetson/t186 r32.5/main arm64 nvidia-l4t-firmware arm64 32.5.1-20210519111140 [1,674 kB]
Get:10 https://repo.download.nvidia.com/jetson/t186 r32.5/main arm64 nvidia-l4t-xusb-firmware arm64 32.5.1-20210614115015 [151 k
B]
Get:11 https://repo.download.nvidia.com/jetson/t186 r32.5/main arm64 nvidia-l4t-initrd arm64 32.5.1-20210614115015 [7,243 kB]
Get:12 https://repo.download.nvidia.com/jetson/t186 r32.5/main arm64 nvidia-l4t-bootloader arm64 32.5.1-20210614115015 [37.4 MB]
Get:13 https://repo.download.nvidia.com/jetson/t186 r32.5/main arm64 python3-jetson-gpio arm64 2.0.16 [19.1 kB]
Get:14 https://repo.download.nvidia.com/jetson/t186 r32.5/main arm64 python-jetson-gpio arm64 2.0.16 [19.0 kB]
Get:15 https://repo.download.nvidia.com/jetson/t186 r32.5/main arm64 jetson-gpio-common arm64 2.0.16 [13.9 kB]
Fetched 106 MB in 2min 17s (774 kB/s)                                          
(Reading database ... 23178 files and directories currently installed.)
Preparing to unpack .../nvidia-l4t-core_32.5.1-20210519111140_arm64.deb ...
Unpacking nvidia-l4t-core (32.5.1-20210519111140) over (32.4.4-20201016123640) ...
Setting up nvidia-l4t-core (32.5.1-20210519111140) ...
Installing new version of config file /etc/nv_tegra_release ...
(Reading database ... 23178 files and directories currently installed.)
Preparing to unpack .../00-nvidia-l4t-tools_32.5.1-20210614115015_arm64.deb ...
Unpacking nvidia-l4t-tools (32.5.1-20210614115015) over (32.4.4-20201027211332) ...
Preparing to unpack .../01-nvidia-l4t-kernel-dtbs_4.9.201-tegra-32.5.1-20210505093723_arm64.deb ...
Unpacking nvidia-l4t-kernel-dtbs (4.9.201-tegra-32.5.1-20210505093723) over (4.9.140-tegra-32.4.4-20201027211332) ...
Preparing to unpack .../02-nvidia-l4t-kernel_4.9.201-tegra-32.5.1-20210505093723_arm64.deb ...
Unpacking nvidia-l4t-kernel (4.9.201-tegra-32.5.1-20210505093723) over (4.9.140-tegra-32.4.4-20201027211332) ...
Preparing to unpack .../03-nvidia-l4t-apt-source_32.5.1-20210219084708_arm64.deb ...
Unpacking nvidia-l4t-apt-source (32.5.1-20210219084708) over (32.4.4-20201016123640) ...
Preparing to unpack .../04-nvidia-l4t-configs_32.5.1-20210219084708_arm64.deb ...
Unpacking nvidia-l4t-configs (32.5.1-20210219084708) over (32.4.4-20201016123640) ...
Preparing to unpack .../05-nvidia-l4t-init_32.5.1-20210519111140_arm64.deb ...
Unpacking nvidia-l4t-init (32.5.1-20210519111140) over (32.4.4-20201027211332) ...
Preparing to unpack .../06-nvidia-l4t-firmware_32.5.1-20210519111140_arm64.deb ...
Unpacking nvidia-l4t-firmware (32.5.1-20210519111140) over (32.4.4-20201027211332) ...
Preparing to unpack .../07-nvidia-l4t-xusb-firmware_32.5.1-20210614115015_arm64.deb ...
Unpacking nvidia-l4t-xusb-firmware (32.5.1-20210614115015) over (32.4.4-20201016123640) ...
Preparing to unpack .../08-nvidia-l4t-initrd_32.5.1-20210614115015_arm64.deb ...
Unpacking nvidia-l4t-initrd (32.5.1-20210614115015) over (32.4.4-20201016123640) ...
Preparing to unpack .../09-nvidia-l4t-bootloader_32.5.1-20210614115015_arm64.deb ...
Unpacking nvidia-l4t-bootloader (32.5.1-20210614115015) over (32.4.4-20201027211332) ...
Preparing to unpack .../10-python3-jetson-gpio_2.0.16_arm64.deb ...
Unpacking python3-jetson-gpio (2.0.16) over (2.0.11) ...
Preparing to unpack .../11-python-jetson-gpio_2.0.16_arm64.deb ...
Unpacking python-jetson-gpio (2.0.16) over (2.0.11) ...
Preparing to unpack .../12-jetson-gpio-common_2.0.16_arm64.deb ...
Unpacking jetson-gpio-common (2.0.16) over (2.0.11) ...
Preparing to unpack .../13-u-boot-tools_2020.10+dfsg-1ubuntu0~18.04.2_arm64.deb ...
Unpacking u-boot-tools (2020.10+dfsg-1ubuntu0~18.04.2) over (2019.07+dfsg-1ubuntu4~18.04.1) ...
Setting up nvidia-l4t-firmware (32.5.1-20210519111140) ...
Setting up nvidia-l4t-init (32.5.1-20210519111140) ...
Installing new version of config file /etc/nvidia-container-runtime/host-files-for-container.d/l4t.csv ...
Installing new version of config file /etc/systemd/nv.sh ...
Installing new version of config file /etc/systemd/nvfb.sh ...
Installing new version of config file /etc/systemd/nvzramconfig.sh ...
Installing new version of config file /etc/systemd/system/nv_update_verifier.service ...
Installing new version of config file /etc/udev/rules.d/99-nv-l4t-usb-host-config.rules ...

Configuration file '/etc/udev/rules.d/99-nv-ufs-mount.rules'
 ==> Deleted (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** 99-nv-ufs-mount.rules (Y/I/N/O/D/Z) [default=N] ? 
Installing new version of config file /etc/udev/rules.d/99-tegra-devices.rules ...
Setting up nvidia-l4t-tools (32.5.1-20210614115015) ...
Installing new version of config file /etc/nvpmodel/nvpmodel_t194.conf ...
Installing new version of config file /etc/nvpmodel/nvpmodel_t194_8gb.conf ...
Installing new version of config file /etc/nvpmodel/nvpmodel_t194_p3668.conf ...
Setting up u-boot-tools (2020.10+dfsg-1ubuntu0~18.04.2) ...
Setting up nvidia-l4t-configs (32.5.1-20210219084708) ...
Installing new version of config file /etc/systemd/nvfb-early.sh ...
Setting up nvidia-l4t-apt-source (32.5.1-20210219084708) ...

Configuration file '/etc/apt/sources.list.d/nvidia-l4t-apt-source.list'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** nvidia-l4t-apt-source.list (Y/I/N/O/D/Z) [default=N] ? 
Setting up jetson-gpio-common (2.0.16) ...
Setting up nvidia-l4t-kernel (4.9.201-tegra-32.5.1-20210505093723) ...
Using the existing boot entry 'primary'
3310-B01---1--jetson-tx2-devkit-mmcblk0p1
Starting kernel post-install procedure.
rootfs AB is not enabled.
Reboot the target system for changes to take effect.
Setting up nvidia-l4t-kernel-dtbs (4.9.201-tegra-32.5.1-20210505093723) ...
Setting up nvidia-l4t-bootloader (32.5.1-20210614115015) ...
3310-B01---1--jetson-tx2-devkit-mmcblk0p1
Starting bootloader post-install procedure.
rootfs AB is not enabled.
Update bl_update_payload completed.
Reboot the target system for changes to take effect.
Updating extlinux.conf...
Root device is set in the extlinux.conf
Setting up nvidia-l4t-xusb-firmware (32.5.1-20210614115015) ...
3310-B01---1--jetson-tx2-devkit-mmcblk0p1
Setting up nvidia-l4t-initrd (32.5.1-20210614115015) ...
Setting up python3-jetson-gpio (2.0.16) ...
Setting up python-jetson-gpio (2.0.16) ...
Processing triggers for nvidia-l4t-kernel (4.9.201-tegra-32.5.1-20210505093723) ...

Device boot log attached as file: boot.log (21.4 KB)

I tried to flash 32.6.1-based rootfs using two different L4T framework versions: 32.6.1 (Jetson_Linux_R32.6.1_aarch64.tbz2) and 32.4.3 (Jetson_Linux_R32.6.1_aarch64.tbz2).
In both cases I use the same rootfs: Ubuntu 18.04 based, with 32.6.1 L4T components inside.
Ethernet is working only if I flash device using 32.6.1 L4T framework version. So I suppose that it was some changes in flashing scripts, which is not ported to the deb-packages (DTB installation?).

I also tried to copy eMMC partitions related to the DTB from the working device (32.6.1 rootfs flashed by USB via 32.6.1 framework) to broken one (using dd), but after copying device got kernel cpu lockup error and starts to reboot in a loop. So, probably, eMMC partitions is not compatible between devices (?).

One more experiment: I tried to build BUP update using l4t_generate_soc_bup.sh from 32.6.1 framework, and flash it to broken device using nv_update_engine. No luck, still no Ethernet.

Does this information enough, or I should provide more?

5

Actually, I don’t think you should mix up the version here. Better using the same version.

Also, I am wondering what is the original BSP and what is the target BSP? If you are upgrading from 32.4.4 to 32.5.1, then what does rel-32.61 do?

6

Actually, I don’t think you should mix up the version here. Better using the same version.

I just tried everything to find out the root cause.

Also, I am wondering what is the original BSP and what is the target BSP? If you are upgrading from 32.4.4 to 32.5.1, then what does rel-32.61 do?

I have a bunch of devices in production with a 32.4.4 inside. I need to update all of them to the 32.6.1 without USB flashing, just using apt and/or rootfs update. I tried to upgrade devices without skipping major versions between update, like 32.4.4 → 32.5.1 → 32.6.1.
Actually, there is no difference between upgrading to both of versions, 32.5.1 or 32.6.1. Ethernet doesn’t work in both.

7

Hi,

Do you have some extra boards here that can try below debug process?

  1. Please directly flash this board with sdkmanager 32.4.4 or 32.5.1. Don’t need to change anything. I just want the pure sdkm software.

  2. Change the source list to 32.6. And directly run dist-upgrade.

8

Does this issue seem any similar to this? Jetson TX2 crashing due to Ethernet connection - #26 by mircea.claudiu.01

I was having crashing issues with the TX2 on 3.5.1 when connected to the internet, and it is almost impossible for us to update to another version without a great deal of work beforehand. Would love to see this fixed.

9

No, this issue has nothing to do with your problem…

10

Do you have some extra boards here that can try below debug process?

Yes. All following procedures was done on another board of same type

  1. I download the L4T Driver Package (BSP) version 32.4.4 from “L4T R32.4.4 archive” page.
  2. Unpacked it
  3. Download “Sample Root Filesystem” from the same R32.4.4 archive page and unpack it into “rootfs” directory inside the Linux_for_Tegra directory from step
  4. Run chroot rootfs and then passwd root inside chroot env to set the root password
  5. Run sudo ./flash.sh -S 15032385536 jetson-tx2 mmcblk0p1

Board successfully flashed at this step:

Spoiler 
# cat /etc/nv_tegra_release                                                                                                        
# R32 (release), REVISION: 4.4, GCID: 23942405, BOARD: t186ref, EABI: aarch64, DATE: Fri Oct 16 19:37:08 UTC 2020

# cat /etc/lsb-release                                                                                                             
DISTRIB_ID=Ubuntu                                                                                                                              
DISTRIB_RELEASE=18.04                                                                                                                          
DISTRIB_CODENAME=bionic                                                                                                                        
DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"
  1. Then, I pointed sources list to 32.6:
Spoiler 
# cat /etc/apt/sources.list.d/nvidia-l4t-apt-source.list                                                                           
deb https://repo.download.nvidia.com/jetson/common r32.6 main                                                                                  
deb https://repo.download.nvidia.com/jetson/t186 r32.6 main

  1. … and run apt update && apt dist-upgrade. Apt upgrade log as file: dist-upgrade.log (137.7 KB)

  2. Unfortunately, I can’t connect to the board via UART, it just stopped working after reboot. So I connect the external display via HDMI, and see message about OEM installation mode. Message says that I need to connect to the /dev/ttyACMx. However, minicom is silent on /dev/ttyACMx devices.
    So, the only thing that I have to prove my problem is photo of external display. And, what a luck, you can see same error message from eqos module at the second line:

    photo_2021-09-03_16-06-20
    photo_2021-09-03_16-06-20960×1280 218 KB

P.S I’ll provide requested info about upgrading from 32.5 to 32.6 in a separate post

11

Hi,

I said “sdkmanager” in previous comment because I don’t want you to do anything by yourself like preparing the BSP and sample rootfs.

Please just use sdkmanager.

And UART log is a must have here. So please also dump it.

12

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

please also share the ls of below node when you are in the 32.4.4 release and also the rel-32.6.1.

/proc/device-tree/ether_qos@2490000