Jetson Nano don't working normally after OTA update

I did guide below for OTA update with debian packages but display don’t working, i still can access to device through ssh. I just only update kernel image.
Below is my step for build debian package and reinstall to device.
1.Extract the original Debian package:
cd path-to-jetpack-install/Linux_for_Tegra/kernel
mkdir kernel-debian
dpkg -x nvidia-l4t-kernel_4.9.201-tegra-32.5.1-20210219084526_arm64.deb kernel-debian/origin
dpkg -e nvidia-l4t-kernel_4.9.201-tegra-32.5.1-20210219084526_arm64.deb kernel-debian/origin/debian
cd kernel-debian
2. Create a directory in which to build your customized Debian package.
mkdir -p repack/debian/
3. Create directories.
3.1. Create a repack/debian/rules
#!/usr/bin/make -f

.PHONY: binary

clean:
dh clean

binary:
dh binary

build:

override_dh_strip:
override_dh_shlibdeps:
3.2 Create a repack/debian/control
Source: nvidia-l4t-kernel
Section: kernel
Priority: standard
Maintainer: NVIDIA Corporation
Build-Depends: debhelper (>=9)
Standards-Version: 3.9.6
Homepage: Autonomous Machines | NVIDIA Developer

Package: nvidia-l4t-kernel
Architecture: arm64
Pre-Depends: nvidia-l4t-core (>> 32.4-0), nvidia-l4t-core (<< 32.6-0)
Depends:
Recommends:
Description: NVIDIA Kernel Package
3.3 Copy these files from the original nvidia-l4t-kernel package to repack/debian directory
•postinst
•triggers
•copyright

cp origin/debian/postinst origin/debian/triggers origin/usr/share/doc/nvidia-l4t-kernel/copyright repack/debian/
3.4 Extract changelog from its archive
gunzip -c origin/usr/share/doc/nvidia-l4t-kernel/changelog.Debian.gz > repack/debian/changelog
3.5 Create repack/debian/install
debian/out/* /
3.6 Create repack/debian/compat text file and replace its contents with the single digit ‘9’; save the file.
3.7 Create a repack/debian/source/ directory.
mkdir -p repack/debian/source
3.8 Create a text file called repack/debian/source/format in a text editor, add the following line and save the file:
3.0 (native)
4. Create a directory that will contain the customized files that will be installed.
mkdir -p repack/debian/out/
5. Add in repack/debian/out/ directory your customizations by creating subdirectories and installing files as necessary
(on this step, i copy Image from official that working normally when flash by recovery mode)
boot/Image
6. Copy the kernel license file from the original Debian package:
mkdir -p repack/debian/out/usr/share/doc/nvidia-l4t-kernel/
gunzip -c origin/usr/share/doc/nvidia-l4t-kernel/LICENSE.kernel.gz > repack/debian/out/usr/share/doc/nvidia-l4t-kernel/LICENSE.kernel
7. Build the package
cd repack
export CC=aarch64-linux-gnu-gcc
dpkg-buildpackage -uc -b -d -a arm64
8. reinstall the package
sudo apt-get install --reinstall ./nvidia-l4t-kernel_4.9.201-tegra-32.5.1-20210219084526_arm64.deb
9.Reboot Jetson nano

Please help me for confirm my step and guide me for solve this issue.
Thanks so much.

Can you provide other information for your board? For example, if this is custom board or not.

Any dmesg to share?

Will this kernel work in non-OTA case?

Can you provide other information for your board? For example, if this is custom board or not.

yes, our board is custom board, it is access control device
Will this kernel work in non-OTA case?
yes, this kernel work normally in non-OTA case.
Log when re-install package:
ac@vsm:~/Downloads$ sudo apt-get install --reinstall ./nvidia-l4t-kernel_4.9.201-tegra-32.5.1-20210219084526_arm64.deb
Reading package lists… Done
Building dependency tree
Reading state information… Done
Note, selecting ‘nvidia-l4t-kernel’ instead of ‘./nvidia-l4t-kernel_4.9.201-tegra-32.5.1-20210219084526_arm64.deb’
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 127 not upgraded.
After this operation, 0 B of additional disk space will be used.
Get:1 /home/ac/Downloads/nvidia-l4t-kernel_4.9.201-tegra-32.5.1-20210219084526_arm64.deb nvidia-l4t-kernel arm64 4.9.201-tegra-32.5.1-20210219084526 [8.030 kB]
debconf: delaying package configuration, since apt-utils is not installed
(Reading database … 140011 files and directories currently installed.)
Preparing to unpack …/nvidia-l4t-kernel_4.9.201-tegra-32.5.1-20210219084526_arm64.deb …
Unpacking nvidia-l4t-kernel (4.9.201-tegra-32.5.1-20210219084526) over (4.9.201-tegra-32.5.1-20210219084526) …
Setting up nvidia-l4t-kernel (4.9.201-tegra-32.5.1-20210219084526) …
depmod: WARNING: could not open /lib/modules/4.9.201-tegra/modules.order: No such file or directory
depmod: WARNING: could not open /lib/modules/4.9.201-tegra/modules.builtin: No such file or directory
Using the existing boot entry ‘primary’
3448-300-0002–1–jetson-nano-devkit-emmc-mmcblk0p1
Processing triggers for nvidia-l4t-kernel (4.9.201-tegra-32.5.1-20210219084526) …

Please also refer to dmesg attachment.
AC_dmesg_log.txt (76.7 KB)

Thank you

It has chance that OTA does not work on custom board because OTA upgrade only install the default jetpack. It assumes a devkit is in use.

The dtb and other components would be gone.

Could you please explain detail about OTA upgrade mechanism? why dtb and other component would be gone?
I did follow guide: Customize JetPack Installation with OTA Update - RidgeRun Developer Connection for custom debian package. I think that only kernel Image is overrided( nvidia-l4t-kernel) or new dtb is used( nvidia-l4t-kernel-dtbs)
Thank you so much!

I would like to provide more information. We don’t custom anything about HW board, only change some source code for Peripheral. We only custom Peripheral.

Sorry that I think “be gone” is not a precise way.

The mechanism of OTA upgrade is just installing some debian package from the source server.

When you do that work, by default it is downloading the debian package from the source list and then install them one by one.

If you are using making your own debian package, and install that package. Then it has nothing to do with “nvidia OTA upgrade”. It is just a debian package created by you, used by you and that is all.

What I said in previous comment is some customers just directly run “apt upgrade” on their custom board, which triggers the OTA updtae.

However, since custom board always has custom dtb or other customization, it is not proper to just use the OTA upgrade from NV. Because it will install the default jetpack again. Which means the dtb is not a customized one but the one from default jetpack.

I think your command is not correct here because the log says it still wants to use the one from source list but not the one you created.

ac@vsm:~/Downloads$ sudo apt-get install --reinstall ./nvidia-l4t-kernel_4.9.201-tegra-32.5.1-20210219084526_arm64.deb
Reading package lists… Done
Building dependency tree
Reading state information… Done
Note, selecting ‘nvidia-l4t-kernel’ instead of ‘./nvidia-l4t-kernel_4.9.201-tegra-32.5.1-20210219084526_arm64.deb’

I copy nvidia-l4t-kernel_4.9.201-tegra-32.5.1-20210219084526_arm64.deb that i created to device (Dowloads folder) and run command sudo apt-get install --reinstall ./nvidia-l4t-kernel_4.9.201-tegra-32.5.1-20210219084526_arm64.deb

Is there any mistake?

Can i change repo list for update what i want?

Can i change repo list for update what i want?

Yes, of course. That is what we suggest to vendor because they need to provide their own apt source to the customers.

“If you are using making your own debian package, and install that package. Then it has nothing to do with “nvidia OTA upgrade”. It is just a debian package created by you, used by you and that is all.”

actually, i am doing it. I create debian package, and install that package but it is error.
Reading package lists… Done
Building dependency tree
Reading state information… Done
Note, selecting ‘nvidia-l4t-kernel’ instead of ‘./nvidia-l4t-kernel_4.9.201-tegra-32.5.1-20210219084526_arm64.deb’
Could you please guide me for fix this error?

Maybe you can search this over the Internet first. This is related to the how the apt tool works on ubuntu.

Thus, some external resources may help.

Question based on the above. What is visible from “ls /lib/modules/*”? You could use your host PC’s “Linux_for_Tegra/rootfs/lib/modules/” to answer this if you can’t log in via something like serial console.

The reason I ask is that kernel’s search for their modules here:
/lib/modules/$(uname -r)/

The “uname -r” is itself a combination of the kernel base version and the CONFIG_LOCALVERSION at the time of compile. Your system wants this directory:
/lib/modules/4.9.201-tegra/

This seems “normal”, but there were some other releases which might differ if mixing releases. My real question is whether it couldn’t find files “modules.*”, or if the directory itself is missing.

Also keep in mind that such an error could show up in an initrd ramdisk prior to loading a kernel from “/boot”.

Also note that if your new kernel config does not use the same device tree, then you might need to update that too for various hardware to be found.

I only want update kernel image, so i don’t copy lib folder to repack/debian/out for build package.
I need to copy lib folder from “Linux_for_Tegra/rootfs” for build package, right?

I need to clarify that if you build a kernel, then that kernel will access “/lib/modules/$(uname -r)/” whether you want it to or not. If the kernel fails to find the right directory, then you get an issue similar to this, and any previous (or even current) modules (which are part of the kernel) will not be reachable and will essentially cease to exist:

I asked about what you see not because I’m expecting you to copy it or modify it, but instead to see if what the kernel mandates be there is actually there. The act of recompiling a kernel can cause “uname -r” to change.

If you take a given kernel, and you keep the exact same configuration, but fail to configure “CONFIG_LOCALVERSION”, then this otherwise exactly matching kernel will fail to find modules. Part of the kernel will thus be missing.

To figure out what is going on, what do you see from:
ls /lib/modules
…and from:
ls /lib/modules/4.9.201-tegra

So far as the kernel image itself goes, if you can change something which is a module, then the job is trivial. A module is the kernel image, but it is dynamically loadable. The base integrated file, the “/boot/Image”, is also the kernel, but changing this has more risks and more complications. The kernel Image file, during flash, is added by flash software to “Linux_for_Tegra/rootfs/boot/Image”, but the “rootfs/” is in no way required to build kernels. Other than getting a copy of the Image file before generating a rootfs image you could consider “rootfs/” to be irrelevant for kernel build (kernel install is a different story…this is what flash software does). There is also a possible signed copy of the kernel Image in a partition.

Thanks so much for your information.
I copy lib folder from origin that extract from nvidia-l4t-kernel official to repack/debian/out for build package and result for install package as below:
ac@vsm:~/Downloads$ sudo dpkg -i --force-overwrite ./nvidia-l4t-kernel_4.9.201-tegra-32.5.1-20210219084526_arm64.deb
(Reading database … 140967 files and directories currently installed.)
Preparing to unpack …/nvidia-l4t-kernel_4.9.201-tegra-32.5.1-20210219084526_arm64.deb …
Unpacking nvidia-l4t-kernel (4.9.201-tegra-32.5.1-20210219084526) over (4.9.201-tegra-32.5.1-20210219084526) …
Setting up nvidia-l4t-kernel (4.9.201-tegra-32.5.1-20210219084526) …
Using the existing boot entry ‘primary’
3448-300-0002–1–jetson-nano-devkit-emmc-mmcblk0p1
Processing triggers for nvidia-l4t-kernel (4.9.201-tegra-32.5.1-20210219084526) …

I think that this result is ok

I have an another question for install nvidia-l4t-kernel-dtbs:

  1. I build nvidia-l4t-kernel-dtbs package and install it to device, result as below:
    ac@vsm:~/Downloads$ sudo dpkg -i --force-overwrite ./nvidia-l4t-kernel-dtbs_4.9.201-tegra-32.5.1-20210219084526_arm64.deb
    (Reading database … 140967 files and directories currently installed.)
    Preparing to unpack …/nvidia-l4t-kernel-dtbs_4.9.201-tegra-32.5.1-20210219084526_arm64.deb …
    Unpacking nvidia-l4t-kernel-dtbs (4.9.201-tegra-32.5.1-20210219084526) over (4.9.201-tegra-32.5.1-20210219084526) …
    Setting up nvidia-l4t-kernel-dtbs (4.9.201-tegra-32.5.1-20210219084526) …

  2. I modify extlinux.conf(add FDT) by access directly to /boot/extlinux/extlinux.conf
    TIMEOUT 30
    DEFAULT primary

MENU TITLE L4T boot options

LABEL primary
MENU LABEL primary kernel
LINUX /boot/Image
INITRD /boot/initrd
FDT /boot/dtb/tegra210-p3448-0002-p3449-0000-b00.dtb
APPEND ${cbootargs} quiet
3. Reboot

Device can not boot up
Could you please help me for fix this issue?
Thank you so much

You can share the log so that we can know why it cannot boot.

sorry, Do you mean uart log or which log?
Because i can not ssh to device for get log

i would like to provide more infor:
if i don’t modify extlinux.conf(add FDT) by access directly to /boot/extlinux/extlinux.conf → device boot up normally.