NVIDIA Jetson Linux 35.4.1 Customization failure

[arkady65]

I just have installed jetpack 5.12 and for beginning tried to recompile and flash kernel to my Jetson AGX Orin 64GB-DRAM develop kit.
After flashing board never come up.

Here is my proceedure:

1. Download toolchain from https://developer.nvidia.com/embedded/jetson-linux/bootlin-toolchain-gcc-93
   extract and set environment (according to Welcome — Jetson Linux Developer Guide documentation )
2. Run the following commands to export the environment variables:
   export KERNEL_DIR=kernel-5.10
   export CROSS_COMPILE=$HOME/l4t-gcc/bin/aarch64-buildroot-linux-gnu-
   export JETPACK=$HOME/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra
   export KERNEL_OUT=$JETPACK/images
   export KERNEL_MODULES_OUT=$JETPACK/images/modules
3. Download BSP sources from the Jetson Download Center for NVIDIA Jetson Linux 35.4.1 and extract the sources into Jetpack directory:
   mkdir -p sources && cd sources
   tar -xvf public_sources.tbz2 Linux_for_Tegra/source/public/kernel_src.tbz2 --strip-components=3
   tar -xvf kernel_src.tbz2
4. steps of building the BSP sources to generate the kernel Image, the external modules, and the device tree blob.
   4.1 Create the directories for the build outputs:
   mkdir -p $KERNEL_MODULES_OUT
   4.2 Clean the environment: ( install before: sudo apt-get install python3-sphinx )
   cd $JETPACK/sources/kernel/$KERNEL_DIR
   make mrproper
   4.3 Setup the default configuration:
   make ARCH=arm64 O=$KERNEL_OUT tegra_defconfig
   4.4 Build the BSP: ( install before: sudo apt-get install libssl-dev )
   make ARCH=arm64 O=$KERNEL_OUT CROSS_COMPILE=$CROSS_COMPILE -j4
   4.5. Install the modules to the output directory created in step 1:
   make modules_install ARCH=arm64 O=$KERNEL_OUT CROSS_COMPILE=$CROSS_COMPILE INSTALL_MOD_PATH=$KERNEL_MODULES_OUT
   4.6. Backup the binaries that come installed by default in Jetpack:
   BKUP_DATE=date "+%Y_%m_%d_%H_%M_%S"
   mv $JETPACK/kernel/Image{,.$BKUP_DATE}
   mv $JETPACK/kernel/kernel_supplements.tbz2{,.$BKUP_DATE}
   mv $JETPACK/kernel/dtb{,.$BKUP_DATE}
   4.7. Copy the binaries built to the default locations expected by the flashing tool:
   cd $KERNEL_OUT
   cp ./arch/arm64/boot/Image $JETPACK/kernel/
   cp -r ./arch/arm64/boot/dts/nvidia $JETPACK/kernel/dtb
   4.8. Update the kernel modules in the kernel supplements tarball:
   cd $KERNEL_MODULES_OUT
   tar --owner root --group root -cjf $JETPACK/kernel/kernel_supplements.tbz2 lib/modules
   4.9. Install tegra binaries:
   cd $JETPACK
   sudo ./apply_binaries.sh
5. Set target to recovery mode and Execute the flash script:
   ./flash.sh jetson-agx-orin-devkit mmcblk0p1

After success in all steps, after reset I tried to run
sudo screen /dev/ttyACM0 115200
failed - /dev/ttyACM0 not found
Target is not running with display, keyboard and mouse connected.

[DaveYYY]

I don’t think the UART issue is related here as it is not handled by the kernel.
Does the device work properly when flashed with the default BSP?
Did you do anything other than re-compiling to the kernel?

[arkady65]

If I flash with with defaults device is working.
I use USB port (ttyACM0) to communicate with the devise in headless mode.

[DaveYYY]

So you got nothing from UART after powering on the device?
It just does not sound reasonable.
Try going through this post to follow the method:

Sample_fs usage when building your own kernel

The documentation seems to be unclear on how to modify your file system. We downloaded everything from the docs to build a custom kernel. It appears that by default, it comes with a rootfs that is sparse so it only has SSH and serial support, no GUI (per the documentation) When we looked at how to update the rootfs so we would have GUI support (and OEM configuration), we ran the command that appears to build a sample_fs tar file with all the packages. What is not clear is what we should be d…

[arkady65]

Here is a problem - seems like device does not booting up.

[DaveYYY]

Are you sure the flashing is successful?
Any log to provide here?

[arkady65]

I’ll try all over again with logs.

[arkady65]

orin_build.zip (111.4 KB)
Here I got logs

[DaveYYY]

Did you do anything other than simply re-compiling?
You still got nothing from UART?
Before Linux kernel, there are still a lot of bootloader stuff coming up, which should be printing log and should not be broken even if the kernel is a bunch of mess.

[arkady65]

OK, I found the problem.

[kayccc]

Could you share the root cause? Thanks

[arkady65]

This is working flow

1. Install toolchain.

2. Install SDKmanager ( in my case it is version 5.1.2 ).

3. Run SDKmanager to build jetson agx orin kernel and modules and flash the board.

4. Install kernel 35.4.1 sources:
   #install tools
   sudo apt install wget lbzip2 build-essential bc zip libgmp-dev libmpfr-dev libmpc-dev vim-common
   sudo apt-get update
   sudo apt-get install flex bison
   sudo apt-get install libssl-dev

   export JETPACK=$HOME/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra
   cd $JETPACK/source
   wget https://developer.nvidia.com/downloads/embedded/l4t/r35_release_v4.1/sources/public_sources.tbz2
   tar -xjf public_sources.tbz2 Linux_for_Tegra/source/public/kernel_src.tbz2 --strip-components=3
   tar -xjf kernel_src.tbz2

5. Build kernel:
   export KERNEL_DIR=kernel-5.10
   export CROSS_COMPILE=$HOME/l4t-gcc/bin/aarch64-buildroot-linux-gnu-
   export JETPACK=$HOME/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra
   export KERNEL_OUT=$JETPACK/images
   export KERNEL_MODULES_OUT=$JETPACK/images/modules
   export CROSS_COMPILE_AARCH64_PATH=$HOME/l4t-gcc
   export CROSS_COMPILE_AARCH64=$HOME/l4t-gcc/bin/aarch64-buildroot-linux-gnu-
   export CROSS_COMPILE=$HOME/l4t-gcc/bin/aarch64-buildroot-linux-gnu-

   #Build the kernel
   mkdir -p $KERNEL_OUT
   $JETPACK/source/nvbuild.sh -o $KERNEL_OUT

   #Backup the binaries that come installed by default in Jetpack
   BKUP_DATE=date "+%Y_%m_%d_%H_%M_%S"
   mv $JETPACK/kernel/Image{,.$BKUP_DATE}
   sudo mv $JETPACK/rootfs/usr/lib/modules/5.10.120-tegra/kernel/drivers/gpu/nvgpu/nvgpu.ko{,.$BKUP_DATE}
   #Don’t move, but copy dtb!!!
   cp -r $JETPACK/kernel/dtb{,.$BKUP_DATE}

   #Update with new binaries
   sudo cp $KERNEL_OUT/drivers/gpu/nvgpu/nvgpu.ko $JETPACK/rootfs/usr/lib/modules/5.10.120-tegra/kernel/drivers/gpu/nvgpu/nvgpu.ko
   sudo cp -r $KERNEL_OUT/arch/arm64/boot/dts/nvidia $JETPACK/kernel/dtb
   cp $KERNEL_OUT/arch/arm64/boot/Image $JETPACK/kernel/Image

6. Build modules
   cd $JETPACK/source/kernel/$KERNEL_DIR
   #Install the modules to the output directory:
   make modules_install ARCH=arm64 O=$KERNEL_OUT CROSS_COMPILE=$CROSS_COMPILE INSTALL_MOD_PATH=$KERNEL_MODULES_OUT
   BKUP_DATE=date "+%Y_%m_%d_%H_%M_%S"
   mv $JETPACK/kernel/kernel_supplements.tbz2{,.$BKUP_DATE}

   #Update the kernel modules in the kernel supplements tarball
   cd $KERNEL_MODULES_OUT
   tar --owner root --group root -cjf $JETPACK/kernel/kernel_supplements.tbz2 lib/modules
   #Install tegra binaries:
   cd $JETPACK
   sudo ./apply_binaries.sh

7. Flash board
   #Execute the Flash Script from $JETPACK
   cd $JETPACK
   sudo ./flash.sh jetson-agx-orin-devkit mmcblk0p1

[arkady65]

An other problem tat found is DTB customization.
I generated custom tegra234-mb1-bct-pinmux-p3701-0000-a04.dtsi and tegra234-mb1-bct-gpio-p3701-0000-a04.dtsi files
Than copy the pinmux.dtsi file to the <l4t_top>/bootloader/t186ref/BCT/ directory, and copy the gpio.dtsi file to the <l4t_top>/bootloader/ directory.
Run flash script
sudo ./flash.sh jetson-agx-orin-devkit mmcblk0p1
But when I decompiled kernel_tegra234-p3701-0005-p3737-0000.dtb on jetson board it was unchanged, same as default (SDKmanager build)

[arkady65]

Hi. Can get some explanation about custom dtsi files?
What should be done after I generated dtsi files from EXEL table and put them in proper places on host PC?

[linuxdev]

Are you familiar with kernel builds and kernel configuration? Understanding how a .dtsi file is used is part of that. Every device tree fragment is used with a particular driver, and that driver is enabled or disabled at kernel configuration prior to building a kernel. The device tree is not actually part of the kernel, but because each non-plug-n-play driver uses the device tree to find the hardware, and to know if it applies to that hardware, the particular device tree fragment applies only when that driver is configured. So building a device tree as a whole from a series of fragments can be done when building the kernel itself based on what is configured for drivers at the time of build.

Any .dts file is a full source code tree (human readable format). When a kernel has .dtsi files, then if the symbol for a given driver is enabled, and if that driver uses that fragment, then the .dtsi gets formatted and inserted into the final .dts source file. Eventually that full .dts source file is compiled into a binary .dtb version, which is what actually gets used.

If instructions for configuring the kernel are incorrect, then you might get a boot failure. If you are only building modules, then it is a simple file copy and install is rarely a problem. The trick is to start by using an exact match to the running system’s configuration. A default dev kit (not necessarily a third party carrier board model) would be almost an exact match when building that kernel using the make target tegra_defconfig. Other than also setting the symbol CONFIG_LOCALVERSION to “-tegra”, this will match most stock running systems (there are other ways to do this as well, even when not stock, but tegra_defconfig is usually the start).

If you have a matching configuration in your kernel build, and then add a feature to it (preferably in the form of a module) using a proper config editor (they understand dependencies; some use menuconfig target, I use nconfig which has symbol search), then this same configuration will be consulted during build of the device tree target. This is how building the device tree includes your .dtsi fragment: If the symbol for the driver using that fragment is built, then the .dtsi is included during build of the build of the dtbs (device tree) target (you do have to make sure the configuration is properly propagated first; building the entire Image might be slow, but this not only propagates the configuration, it also is a nice acid test for failure cases).

You can always ask more on any detail. But there is the gist: Configure the kernel to include a driver which uses that fragment, propagate the configuration correctly, and then build the device tree. The .dtsi will be included. This is part of the Kconfig system, and so if there is any kind of patching (in addition to replacing a .dtsi file), then you might need to first patch a Kconfig file. That along with the .dtsi file is the full build.

[arkady65]

thanks a lot, I try to understand.

[arkady65]

Hi. I found one topic in the forum about enabling RGMII driver. Moderator recommends following:

I have a question - in what file shall be added this code?

[linuxdev]

I can’t tell you the specifics, but which exact driver is this for? If you were to configure to build this driver in a config editor, then it would have a “symbol” for the driver. Some sort of “CONFIG_...something...”. If I know the driver, then I might be able to answer. I can’t answer for just RGMII since I dont know enough about that driver. If you don’t know the symbol, then if you know how to get to this in a config editor would help.

[arkady65]

It’s an ethernet driver that works on AGX Xavier board. As I understand it is implemented on Orin aswel, but disabled. To enable it I found that recomendation.

[arkady65]

Also I found it in developer guide:
Jetson AGX Orin Platform Adaptation and Bring-Up — Jetson Linux Developer Guide documentation (nvidia.com)