Image does not boot up when used custom LOCALVERSION

Hello,

I am trying to build my own kernel with the additional USB driver. As a first step, I wanted to build the Image and modules without any changes in make menuconfig except adding custom LOCALVERSION.

I can successfully run the build process, but I get stuck at boot process. When I was testing just the Image, without new modules, my Jetson was booting up. Once I added the custom-compiled modules, it fails to boot.

The build process I follow:


#1. Create output directories for kernel compilation
mkdir /home/bartosz/nvidia/Linux_for_Tegra/source/kernel_out
export WORK=/home/bartosz/nvidia/Linux_for_Tegra/source/kernel_out
mkdir -p $WORK/rootfs/boot
mkdir -p $WORK/rootfs/lib

#2. Create custom .config file
cd ~/nvidia/Linux_for_Tegra/source/kernel/kernel-jammy-src/
make ARCH=arm64 menuconfig
#LOAD DEFAULT CONFIG FROM ./arch/arm64/configs/defconfig
#CHANGE LOCALVERSION
#SAVE AS ./arch/arm64/configs/gadget.config
export KBUILD_DEFCONFIG=gadget.config

#3 Build the kernel
cd ~/nvidia/Linux_for_Tegra/source
./nvbuild.sh -o $WORK
INSTALL_MOD_PATH=$WORK/rootfs ./nvbuild.sh -i

Do I miss any step? The official instruction is quite confusing I must say, It mentiones some script files like apply_binaries.sh without telling where to find them.

The error I get during boot process:

[    5.166601] hid-generic 0003:1A2C:0E24.0001: input,hidraw0: USB HID v1.10 Keyboard [SEM USB Keyboard] on usb-3610000.usb-2.2/input0
[    5.171386] input: SEM USB Keyboard Consumer Control as /devices/platform/bus@0/3610000.usb/usb1/1-2/1-2.2/1-2.2:1.1/0003:1A2C:0E24.0002/input/input2
[    5.230224] input: SEM USB Keyboard System Control as /devices/platform/bus@0/3610000.usb/usb1/1-2/1-2.2/1-2.2:1.1/0003:1A2C:0E24.0002/input/input3
[    5.231414] hid-generic 0003:1A2C:0E24.0002: input,hidraw1: USB HID v1.10 Device [SEM USB Keyboard] on usb-3610000.usb-2.2/input1
[   14.826548] ERROR: mounting PARTUUID=612e65bc-ce31-4b79-a95c-3bf422789357 as /mnt fail...
[   14.828381] ERROR: PARTUUID=612e65bc-ce31-4b79-a95c-3bf422789357 mount fail...
[   14.829785] ttyTCU0: Press [ENTER] to start bash in 30 seconds...
[   17.830232] ttyTCU0: Press [ENTER] to start bash in 27 seconds...
[   20.830594] ttyTCU0: Press [ENTER] to start bash in 24 seconds...
[   23.830955] ttyTCU0: Press [ENTER] to start bash in 21 seconds...

The kernel .config I used differs from the original .config I took from SDK only in few places:

bartosz@bartosz:~/nvidia/Linux_for_Tegra/source/kernel_out/kernel/kernel-jammy-src/arch/arm64/configs$ diff config_sdk gadget.config 
5c5
< CONFIG_CC_VERSION_TEXT="gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0"
---
> CONFIG_CC_VERSION_TEXT="aarch64-buildroot-linux-gnu-gcc.br_real (Buildroot 2022.08) 11.3.0"
7c7
< CONFIG_GCC_VERSION=110400
---
> CONFIG_GCC_VERSION=110300
32c32
< CONFIG_LOCALVERSION=""
---
> CONFIG_LOCALVERSION="-gadget"
790a791,793
> CONFIG_GCC_PLUGINS=y
> # CONFIG_GCC_PLUGIN_LATENT_ENTROPY is not set
> # CONFIG_GCC_PLUGIN_RANDSTRUCT is not set
8984a8988,8991
> # CONFIG_GCC_PLUGIN_STRUCTLEAK_USER is not set
> # CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF is not set
> # CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL is not set
> # CONFIG_GCC_PLUGIN_STACKLEAK is not set

I tried to edit the tegra-defconfig as well, effect is similar. Do you have any ideas what could cause the issue?
I use Jetson Orin with eMMC drive, I don’t know if this might be the issue…

Hi,

Is your device a dev kit or custom board?

Thanks

Hi,

It’s the Waveshares devkit which I think is pin to pin compatible with original Nvidia development kit.

Hi,

Please try use sdk manager flash first.
If sdk manager flash fail, please reach out the vendor to get the customized bsp.

Thanks

I did for the virgin flashing, and the system booted correctly. Is there a way to class the custom made kernel and drivers with SDK?

@DavidDDD if I use only Image and copy the modules to the target device, do I still need to run the apply_binaries.sh before?

Hi,

In addition to build the custom kernel, you need to rebuild dependent oot modules and dtb.
Please try to follow the doc mentioned:

Thanks

Those instructions are fine when you are trying to build the kernel without changing the kernel modules. Many user face the same problem i.e. wiping .config file by Makefile.

Only the solution proposed here made it work (with couple of tweaks).
I must admit that it is quite frustrating that official NVIDIA Guide is so inconsistent and leads to the failed builds.

Anyways, the reason of the booting problem , namely

[   14.826548] ERROR: mounting PARTUUID=612e65bc-ce31-4b79-a95c-3bf422789357 as /mnt fail...
[   14.828381] ERROR: PARTUUID=612e65bc-ce31-4b79-a95c-3bf422789357 mount fail...

was the fact that I did not build the initrd (sudo nv-update-initrd).
Initially, initrd generation using nv-update-initrd led to another problem of missing nvethernet.ko

Error: /lib/modules/5.15.148-gadget/updates/drivers/net/ethernet/nvidia/nvethernet/nvethernet.ko not found

but I copied it from other builds modules. The issue actually was that the whole “updates” directory did not exist

5.15.148-gadget-tegra is the modules directory built by following the instructions from the official Guide.
Do you maybe know why that particular directory was missing? The content of the directory is following, maybe it will help you:

bartosz@tegra-ubuntu:/lib/modules/5.15.148-gadget/updates$ ls -R
.:
drivers  nvgpu.ko  nvhwpm.ko  opensrc-disp  sound

./drivers:
bluetooth  crypto   firmware  hwmon  media  misc  nv-p2p    nvpps  platform  rtc  thermal  usb
bmi088     devfreq  gpu       i2c    mfd    net   nvpmodel  pci    pwm       spi  tty      video

./drivers/bluetooth:
realtek

./drivers/bluetooth/realtek:
rtk_btusb.ko

./drivers/bmi088:
bmi088.ko

./drivers/crypto:
tegra  tegra-se-nvrng.ko

./drivers/crypto/tegra:
tegra-se.ko

./drivers/devfreq:
governor_pod_scaling.ko  tegra_wmark.ko

./drivers/firmware:
tegra

./drivers/firmware/tegra:
ivc_ext.ko

./drivers/gpu:
drm  host1x  host1x-fence  host1x-nvhost

./drivers/gpu/drm:
tegra

./drivers/gpu/drm/tegra:
tegra-drm.ko

./drivers/gpu/host1x:
host1x.ko

./drivers/gpu/host1x-fence:
host1x-fence.ko

./drivers/gpu/host1x-nvhost:
host1x-nvhost.ko

./drivers/hwmon:
f75308.ko

./drivers/i2c:
busses

./drivers/i2c/busses:
i2c-nvvrs11.ko

./drivers/media:
i2c  platform

./drivers/media/i2c:
ar1335_common.ko  max9295.ko  max929x.ko   nv_ar0234.ko    nv_imx185.ko  nv_imx274.ko  nv_imx390.ko  nv_ov5693.ko  virtual_i2c_mux.ko
lt6911uxc.ko      max9296.ko  max96712.ko  nv_hawk_owl.ko  nv_imx219.ko  nv_imx318.ko  nv_imx477.ko  pca9570.ko

./drivers/media/platform:
tegra

./drivers/media/platform/tegra:
camera  cdi

./drivers/media/platform/tegra/camera:
tegra-camera.ko

./drivers/media/platform/tegra/cdi:
cam_cdi_tsc.ko

./drivers/mfd:
nvidia-vrs-pseq.ko

./drivers/misc:
nvsciipc  tegra-pcie-dma-test.ko

./drivers/misc/nvsciipc:
nvsciipc.ko

./drivers/net:
can  ethernet  wireless

./drivers/net/can:
mttcan

./drivers/net/can/mttcan:
mttcan.ko

./drivers/net/ethernet:
nvidia  realtek

./drivers/net/ethernet/nvidia:
nvethernet  pcie

./drivers/net/ethernet/nvidia/nvethernet:
nvethernet.ko

./drivers/net/ethernet/nvidia/pcie:
tegra_vnet.ko

./drivers/net/ethernet/realtek:
r8126  r8168

./drivers/net/ethernet/realtek/r8126:
r8126.ko

./drivers/net/ethernet/realtek/r8168:
r8168.ko

./drivers/net/wireless:
realtek

./drivers/net/wireless/realtek:
rtl8822ce

./drivers/net/wireless/realtek/rtl8822ce:
rtl8822ce.ko

./drivers/nv-p2p:
nvidia-p2p.ko

./drivers/nvpmodel:
nvpmodel-clk-cap.ko

./drivers/nvpps:
nvpps.ko

./drivers/pci:
controller  endpoint

./drivers/pci/controller:
tegra-pcie-edma.ko

./drivers/pci/endpoint:
functions

./drivers/pci/endpoint/functions:
pci-epf-dma-test.ko  pci-epf-tegra-vnet.ko

./drivers/platform:
tegra

./drivers/platform/tegra:
aon  dce  mc-hwpm.ko  mc-utils  mce  rtcpu  tegra-cactmon-mc-all.ko  tegra-fsicom.ko  uncore_pmu

./drivers/platform/tegra/aon:
tegra-aon-ivc-echo.ko  tegra234-aon.ko

./drivers/platform/tegra/dce:
tegra-dce.ko

./drivers/platform/tegra/mc-utils:
mc-utils.ko

./drivers/platform/tegra/mce:
tegra-mce.ko

./drivers/platform/tegra/rtcpu:
camchar.ko  camera-diagnostics.ko  capture-ivc.ko  hsp-mailbox-client.ko  ivc-bus.ko  rtcpu-debug.ko  tegra-camera-rtcpu.ko

./drivers/platform/tegra/uncore_pmu:
tegra23x_perf_uncore.ko

./drivers/pwm:
pwm-tegra-tachometer.ko

./drivers/rtc:
nvvrs-pseq-rtc.ko

./drivers/spi:
spi-tegra124-slave.ko  spi-tegra210-quad.ko

./drivers/thermal:
tegra234-oc-event.ko  thermal-trip-event.ko

./drivers/tty:
serial

./drivers/tty/serial:
wch_35x

./drivers/tty/serial/wch_35x:
wch.ko

./drivers/usb:
typec

./drivers/usb/typec:
fusb301.ko

./drivers/video:
tegra

./drivers/video/tegra:
camera  dc  host  nvmap  tsec

./drivers/video/tegra/camera:
tegra_camera_platform.ko

./drivers/video/tegra/dc:
bridge

./drivers/video/tegra/dc/bridge:
maxim_gmsl_dp_serializer.ko  ti_fpdlink_dp_serializer.ko

./drivers/video/tegra/host:
capture  isp  nvcsi  nvdla  pva  vi

./drivers/video/tegra/host/capture:
nvhost-capture.ko

./drivers/video/tegra/host/isp:
nvhost-isp5.ko

./drivers/video/tegra/host/nvcsi:
nvhost-nvcsi-t194.ko  nvhost-nvcsi.ko

./drivers/video/tegra/host/nvdla:
nvhost-nvdla.ko

./drivers/video/tegra/host/pva:
nvhost-pva.ko

./drivers/video/tegra/host/vi:
nvhost-vi5.ko

./drivers/video/tegra/nvmap:
nvmap.ko

./drivers/video/tegra/tsec:
tsecriscv.ko

./opensrc-disp:
nvidia-drm.ko  nvidia-modeset.ko  nvidia.ko

./sound:
soc

./sound/soc:
tegra

./sound/soc/tegra:
snd-soc-tegra-machine-driver.ko  snd-soc-tegra186-dspk.ko    snd-soc-tegra210-ahub.ko  snd-soc-tegra210-iqc.ko    snd-soc-tegra210-sfc.ko
snd-soc-tegra-utils.ko           snd-soc-tegra210-admaif.ko  snd-soc-tegra210-amx.ko   snd-soc-tegra210-mixer.ko
snd-soc-tegra186-arad.ko         snd-soc-tegra210-adx.ko     snd-soc-tegra210-dmic.ko  snd-soc-tegra210-mvc.ko
snd-soc-tegra186-asrc.ko         snd-soc-tegra210-afc.ko     snd-soc-tegra210-i2s.ko   snd-soc-tegra210-ope.ko

Hi,

Thank you for your feedback.
I would want to know above kernel build commands you run is on the host or the target device?

Thanks

On the host, I do the cross compilation. I noticed, that the oot modules were not built when I changed the LOCALVERSION in my config file ( missing nvethernet.ko)
Also, the suffix -tegra was added when compiling the rest of the modules, so after installation the modules directory looks like 5.15.148-gadget-tegra.
The reason why I need the custom version is that I don’t want to brick my device while experimenting on /lib/modules/5.15.148-tegra