Repeated reboot at slot A in jetson orin with rootfs A/B=1, but boot to slot B unexpectedly

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
1

Module: Jetson AGX orin 32G, rootfs A/B = 1

software: Jetpack-6.0

I am trying to reproduce a problem by repeated rebooting at slot A. But system boot to slot B unexpectedly after 3 times and status of slot A is normal.

slot info

pilot@gen4orin1:~$ sudo nvbootctrl dump-slots-info
Current version: 36.3.0
Capsule update status: 1
Current bootloader slot: B
Active bootloader slot: B
num_slots: 2
slot: 0,             status: normal
slot: 1,             status: normal

shell of repeated reboot, which is executed in /etc/rc.local

#!/bin/bash

COUNTER_FILE="/var/lib/reboot_counter"
MAX_REBOOTS=1000

log_to_dmesg() {
    echo "$1" > /dev/kmsg
}

check_flush_error() {
    if dmesg | grep -i "flush error" > /dev/null; then
        echo "check flush error! stop reboot"
        log_to_dmesg "reboot_test: check flush error，stop reboot test"
        exit 1
    fi
}

if [ -f "$COUNTER_FILE" ]; then
    COUNTER=$(cat "$COUNTER_FILE")
else
    COUNTER=0
fi

COUNTER=$((COUNTER + 1))
echo $COUNTER > "$COUNTER_FILE"

echo "$COUNTER reboot - $(date)"
log_to_dmesg "reboot_test: $COUNTER"

check_flush_error

if [ $COUNTER -ge $MAX_REBOOTS ]; then
    echo "$MAX_REBOOTS reboots have been finished"
    log_to_dmesg "reboot_test: $MAX_REBOOTS reboots have been finished"
    exit 0
fi

sleep 150

check_flush_error

echo "prepare $((COUNTER + 1))th reboot..."
log_to_dmesg "reboot_test: prepare $((COUNTER + 1))th reboot"
sync
sleep 5

nvbootctrl set-active-boot-slot 0
reboot

Here is the reboot log when I set active boot slot to A

reboot_A_to_B_error.log (184.0 KB)

If system boot to slot B in this case, slot A is never worked, even if I set active boot slot to A, it will still boot to B.

2

*** Please note that this reply is generated by LLM automatically ***

Based on the provided information, it appears that you are experiencing an issue with the Jetson AGX Orin module where the system is booting to slot B unexpectedly after 3 reboots, despite setting the active boot slot to A.

To troubleshoot this issue, let’s break down the information provided:

  1. Jetson AGX Orin Module: This is the specific module you are using, which has a 64G rootfs.
  2. Jetpack 4.6: This is the software version you are using, which includes the Jetpack SDK.
  3. Repeated Rebooting: You are trying to reproduce a problem by repeatedly rebooting the system at slot A.
  4. Unexpected Boot to Slot B: After 3 reboots, the system boots to slot B unexpectedly, despite setting the active boot slot to A.
  5. Normal Slot A Status: The status of slot A is normal, indicating that there are no apparent issues with the slot itself.
  6. Reboot Log: You have provided a reboot log, which may contain clues about the issue.

To compare and contrast the information, let’s consider the following:

  • Boot Architecture: The Jetson AGX Orin module uses a boot architecture that involves multiple stages, including BootROM, PSCROM, MB1, MB2, and UEFI. It’s possible that there is an issue with one of these stages that is causing the system to boot to the wrong slot.
  • Partition Configuration: The partition configuration files may be incorrect or corrupted, leading to the system booting to the wrong slot.
  • Software Configuration: The software configuration, including the Jetpack SDK, may be incorrect or corrupted, leading to the system booting to the wrong slot.

To resolve this issue, I would recommend the following steps:

  1. Review the Reboot Log: Carefully review the reboot log to identify any errors or warnings that may indicate the cause of the issue.
  2. Verify Partition Configuration: Verify that the partition configuration files are correct and not corrupted.
  3. Verify Software Configuration: Verify that the software configuration, including the Jetpack SDK, is correct and not corrupted.
  4. Try Booting from a Different Slot: Try booting the system from a different slot to see if the issue is specific to slot A.
  5. Contact NVIDIA Support: If none of the above steps resolve the issue, contact NVIDIA support for further assistance.

In summary, the issue appears to be related to the boot architecture or software configuration of the Jetson AGX Orin module. By reviewing the reboot log, verifying the partition and software configuration, and trying different boot slots, you may be able to identify and resolve the issue.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

4

I want to know the reason why slot A is unworked and how to fix it, this reply generated by LLM has no use.

6

hello YuanliFu,

please see-also developer guide, Managing Rootfs Slots with nvbootctrl, you may also check the rootfs status by using -t <target> options.
for instance, $ sudo nvbootctrl -t rootfs dump-slots-info

according to the logs.. you’re now at slot-B.

I> Current Boot-Chain Slot: 1
I> BR-BCT Boot-Chain is 1, and status is 1. Set UPDATE_BRBCT bit to 0

may I know what’s the actual scenario you’ve reproduce a problem by repeated rebooting at slot-A? is there a log starting from slot-A for checking?

7

Hi Jerry

may I know what’s the actual scenario you’ve reproduce a problem by repeated rebooting at slot-A?

Yes, I was at slot A, and write a shell to reboot repeatedly, active slot was always set to A

is there a log starting from slot-A for checking?

I do not know where the checking log stored, I upload the full booting log of rebooting from slot A to slot B, does the checking log contained in the booting log I uploaded?

8

I use nvbootctrl to check status of rootfs, find slot A is set to unbootable

pilot@gen4orin2:~$ sudo nvbootctrl -t rootfs dump-slots-info
Current rootfs slot: B
Active rootfs slot: B
num_slots: 2
slot: 0,             retry_count: 0,             status: unbootable
slot: 1,             retry_count: 3,             status: normal
9

hello YuanliFu,

please see-also developer guide for Rootfs Selection section.
the Bootloader slot and rootfs slot are configured together. for instance, Bootloader A always boots with rootfs A, and Bootloader B always boots with rootfs B.

besides.. your logs already start from slot-B. we don’t have details of slot-A crash.

10

I disabled rootfs A/B and reboot orin repeatedly, it reported error when loading kernel, here is log

[    8.234624] pci 0005:01:00.0: enabling Extended Tags
[    8.236637] pci 0005:01:00.0: PME# supported from D0 D3hot D3cold
[    8.238166] pci 0005:01:00.0: 126.024 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x8 link at 0005:00:00.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
[    8.238776] pci 0005:01:00.1: [11f8:4336] type 00 class 0x088000
[    8.239146] pci 0005:01:00.1: reg 0x10: [mem 0x00000000-0x007fffff 64bit]
[    8.239847] pci 0005:01:00.1: enabling Extended Tags
[    8.241086] pci 0005:01:00.1: PME# supported from D0 D3hot D3cold
[    8.244561] pci 0005:00:00.0: BAR 15: assigned [mem 0x2800000000-0x2809ffffff 64bit pref]
[    8.244563] pci 0005:00:00.0: BAR 14: assigned [mem 0x2b28000000-0x2b287fffff]
[    8.244565] pci 0005:01:00.0: BAR 2: assigned [mem 0x2800000000-0x2803ffffff 64bit pref]
[    8.244714] pci 0005:01:00.0: BAR 4:Checking whether device /dev/sd?1 exist [m
0x2804000000-0x280Device /dev/fff1 does not exisref
]
Checking whether devi487/dev/nvme?n1p1 exist05:
01:00.0: BAR 0: asDevice /dev/nvme?n1p1 does not exist0x2
000000-0x2808ffffff 64bit pref]
[    8.245115] pci 0005:01:00.1: BAR 0: assigned [mem 0x2b28000000-0x2b287fffff 64bit]
[    8.245246] pci 0005:00:00.0: PCI bridge to [bus 01-ff]
[    8.245257] pci 0005:00:00.0:   bridge window [mem 0x2b28000000-0x2b287fffff]
[    8.245259] pci 0005:00:00.0:   bridge window [mem 0x2800000000-0x2809ffffff 64bit pref]
[    8.245323] pcieport 0005:00:00.0: Adding to iommu group mount: [  /mnt: wrong fs type, bad option, bad superblock on /dev/mmcblk0, missing codepage or helper program, or other error.ME:
gnaling with IRQ 205
[    8.246370] pcieport 0005:0bash: 0:0cannot set terminal process group (-1): Inappropriate ioctl for device205

bash: no [  job control in this shell9]
Finbash-5.1# g OTA work dir on external storage devices
[    8.344346] mount /dev/mmcblk0p1 /mnt
[    8.363918] EXT4-fs (mmcblk0p1): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[    8.365401] is_boot_only_partition /mnt
[    8.379483] OTA work directory /mnt/ota_work is not found on /dev/mmcblk0p1
[    8.385157] mount /dev/mmcblk1p1 /mnt
[    8.409500] EXT4-fs (mmcblk1p1): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[    8.410839] is_boot_only_partition /mnt
[    8.419785] OTA work directory /mnt/ota_work is not found on /dev/mmcblk1p1
[    8.442641] Finding OTA work dir on internal storage device
[    8.443686] mount /dev/mmcblk0 /mnt
[    8.490920] Failed to mount /dev/mmcblk0 on the /mnt
[    8.491928] Failed to run "mount_ota_work_partition /dev/mmcblk0 /mnt"
[    8.492954] OTA work directory is not found on internal and external storage devices
[  267.645667] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x402, iova=0x00000000, fsynr=0x240011, cbfrsynra=0x1013, cb=1
[  267.645909] tegra-mc 2c00000.memory-controller: pcie4w: secure write @0x00000003ffffff00: VPR violation ((null))

bash-5.1#

Here is the full log for this error boot

Jetson System firmware version 36.3.0-gcid-36191598 date 2024-05-06T16:58:59+00:
00
ESC   to enter Setup.
F11   to enter Boot Manager Menu.
Enter to continue boot.
......
      L4TLauncher: Attempting Recovery Boot
EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Exiting boot services...
��debugfs initialized
��I/TC: Reserved shared memory is disabled
I/TC: Dynamic shared memory is enabled
I/TC: Normal World virtualization support is disabled
I/TC: Asynchronous notifications are disabled
��[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd421]
[    0.000000] Linux version 5.15.136-voyager-1.0.1-tegra (root@10a4792781e4) (aarch64-buildroot-linux-gnu-gcc.br_real (Buildroot 2022.08) 11.3.0, GNU ld (GNU Binutils) 2.38) #1 SMP PREEMPT Fri D)
[    0.000000] Machine model: NVIDIA Jetson AGX Orin Developer Kit
[    0.000000] efi: EFI v2.70 by EDK II
[    0.000000] efi: RTPROP=0x82804f198 SMBIOS=0xffff0000 SMBIOS 3.0=0x827a20000 MEMATTR=0x821be1018 ESRT=0x821c79618 RNG=0x8147c0018 MEMRESERVE=0x8159d9f18
[    0.000000] random: crng init done
[    0.000000] secureboot: Secure boot disabled
[    0.000000] esrt: Reserving ESRT space from 0x0000000821c79618 to 0x0000000821c79650.
[    0.000000] Reserved memory: created CMA memory pool at 0x0000000804000000, size 256 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000080000000-0x0000000833ffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x828cc9800-0x828ccbfff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x00000000ffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   [mem 0x0000000100000000-0x0000000833ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x00000000fffdffff]
[    0.000000]   node   0: [mem 0x00000000fffe0000-0x00000000ffffffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x0000000818ac9fff]
[    0.000000]   node   0: [mem 0x0000000818aca000-0x0000000818c2bfff]
[    0.000000]   node   0: [mem 0x0000000818c2c000-0x000000082610ffff]
[    0.000000]   node   0: [mem 0x0000000826110000-0x000000082804ffff]
[    0.000000]   node   0: [mem 0x0000000828050000-0x000000082c5fffff]
[    0.000000]   node   0: [mem 0x000000082c600000-0x000000082c7fffff]
[    0.000000]   node   0: [mem 0x000000082c800000-0x000000082cd6ffff]
[    0.000000]   node   0: [mem 0x000000082d700000-0x000000082d7fffff]
[    0.000000]   node   0: [mem 0x0000000832000000-0x0000000833ffffff]
[    0.000000] Initmem s��WARNING: clock_disable: clk_power_ungate on gated domain 35 for gpusysclk
��etup node 0 [mem 0x0000000080000000-0x0000000833ffffff]
[    0.000000] On node 0, zone Normal: 2448 pages in unavailable ranges
[    0.000000] On node 0, zone Normal: 18432 pages in unavailable ranges
[    0.000000] On node 0, zone Normal: 16384 pages in unavailable ranges
[    0��WARNING: clock_disable: clk_power_ungate on gated domain 35 for gpc1clk
��.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS migration not required
[    0.000000] psci: SMC Calling Convention v1.2
[    0.000000] percpu: Embe��WARNING: clock_disable: clk_power_ungate on gated domain 35 for gpc0clk
��dded 29 pages/cpu s80472 r8192 d30120 u118784
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: Address authentication (architected algorithm)
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: Virtualization Host Extensions
[    0.000000] CPU features: detected: Hardware dirty bit management
[    0.000000] CPU features��WARNING: clock_disable: clk_power_ungate on gated domain 34 for dla1_core
��: detected: Spectre-v4
[    0.000000] CPU features: detected: Spectre-BHB
[    0.000000] CPU features: kernel page table isolation forced ON by KASLR
[    0.000000] CPU features: detected: Kernel page table isolation (KPTI)
[    0.000000] alternatives: patching kernel code
[    0��WARNING: clock_disable: clk_power_ungate on gated domain 34 for nafll_dla1_core
��.000000] Built 1 zonelists, mobility grouping on.  Total pages: 7930224
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: root=/dev/initrd rw rootwait mminit_loglevel=4 console=ttyTCU0,115200 firmware_class.path=/etc/firmware fbcon=map:0 net.ifnames=0 bl_pr��WARNING: clock_disablen
��of_dataptr=2031616@0x82C610000 bl_prof_ro_ptr=65536@0x82C600000
[    0.000000] Unknown kernel command line parameters "bl_prof_dataptr=2031616@0x82C610000 bl_prof_ro_ptr=65536@0x82C600000", will be passed to user space.
[    0.000000] Dentry cache hash table entries: 4194304 (ord��WARNING: clock_disable: clk_power_ungate on gated domain 34 for nafll_dla1_falcon
��er: 13, 33554432 bytes, linear)
[    0.000000] Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] software IO TLB: mapped [mem 0x00000000fbfe0000-0x00000000fffe0000] (��WARNING: clock_disable: clk_power_ungate on gated domain 34 for dla0_core
��64MB)
[    0.000000] Memory: 31138988K/32225728K available (19456K kernel code, 4378K rwdata, 17272K rodata, 8192K init, 534K bss, 824596K reserved, 262144K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[    0.000000] trace event string ve��WARNING: clock_disable: clk_power_ungate on gated domain 34 for nafll_dla0_core
��rifier disabled
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=8.
��WARNING: clock_disable: clk_power_ungate on gated domain 34 for dla0_falcon
��
  [    0.000000]        Rude variant of Tasks RCU enabled.
[    0.000000]  Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=8
[  ��WARNING: clock_disable: clk_power_ungate on gated domain 34 for nafll_dla0_falcon
��  0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: 960 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] GICv3: Distributor has no Range Selector support
[    0.000000] Root IRQ handler: gic_handle_irq
[    0.000000] GICv3: 16 PPIs implemented
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x000000000f440000
[    0.000000] arch_timer: cp15 timer(s) running at 31.25MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xe6a171046, max_idle_ns: 881590405314 ns
[    0.000001] sched_clock: 56 bits at 31MHz, resolution 32ns, wraps every 4398046511088ns
[    0.000516] Console: colour dummy device 80x25
[    0.000657] Calibrating delay loop (skipped), value calculated using timer frequency.. 62.50 BogoMIPS (lpj=125000)
[    0.000674] pid_max: default: 32768 minimum: 301
[    0.000738] LSM: Security Framework initializing
[    0.000790] Yama: becoming mindful.
[    0.000827] SELinux:  Initializing.
[    0.001084] Mount-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.001227] Mountpoint-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.003901] rcu: Hierarchical SRCU implementation.
[    0.008351] Tegra Revision: A01 SKU: 210 CPU Process: 0 SoC Process: 0
[    0.008660] Remapping and enabling EFI services.
[    0.009760] smp: Bringing up secondary CPUs ...
[    0.010604] Detected PIPT I-cache on CPU1
[    0.010711] GICv3: CPU1: found redistributor 100 region 0:0x000000000f460000
[    0.010772] CPU1: Booted secondary processor 0x0000000100 [0x410fd421]
[    0.011530] Detected PIPT I-cache on CPU2
[    0.011545] GICv3: CPU2: found redistributor 200 region 0:0x000000000f480000
[    0.011570] CPU2: Booted secondary processor 0x0000000200 [0x410fd421]
[    0.012215] Detected PIPT I-cache on CPU3
[    0.012227] GICv3: CPU3: foundinsmod /lib/modules/5.15.136-voyagergio0.1-tegra/updates/drivers/net/ethernet/realtek/rf4a8.ko 0

[  modprobe: ERROR: could not insert 'r8168': Unknown symbol in module, or unknown parameter (see dmesg)or
000000300 [0x410fd421]
[    0.014951] Detected PIPT I-cache on CPU4
[    0.014977] GICv3: CPU4: found redistributor 10000 region 0:0x000000000f4c0000
[    0.015009] CPU4: Booted secondary processor 0x0000010000 [0x410fd421]
[    0.015727] Detected PIPT I-cache on CPU5
[    0.015741] GICv3: CPU5: found redistributor 10100 region 0:0x000000000f4e0000
[    0.015763] CPU5: Booted secondary processor 0x0000010100 [0x410fd421]
[    0.016439] Detected PIPT I-cache on CPU6
[    0.016454] GICv3: CPU6: found redistributor 10200 region 0:0x000000000f500000
[    0.016473] CPU6: Booted secondary processor 0x0000010200 [0x410fd421]
[    0.017146] Detected PIPT I-cache on CPU7
[    0.017158] GICv3: CPU7: found redistributor 10300 region 0:0x000000000f520000
[    0.017180] CPU7: Booted secondary processor 0x0000010300 [0x410fd421]
[    0.017256] smp: Brought up 1 node, 8 CPUs
[    0.017264] SMP: Total of 8 processors activated.
[    0.017271] CPU features: detected: 32-bit EL0 Support
[    0.017273] CPU features: detected: Data cache clean to the PoU not required for I/D coherence
[    0.017278] CPU features: detected: Common not Private translations
[    0.017280] CPU features: detected: CRC32 instructions
[    0.017281] CPU features: detected: Data cache clean to Point of Persistence
[    0.017285] CPU features: detected: Generic authentication (architected algorithm)
[    0.017286] CPU features: detected: RCpc load-acquire (LDAPR)
[    0.017287] CPU features: detected: LSE atomic instructions
[    0.017289] CPU features: detected: Privileged Access Never
[    0.017292] CPU features: detected: RAS Extension Support
[    0.017299] CPU features: detected: Speculative Store Bypassing Safe (SSBS)
[    0.070763] CPU: All CPU(s) started at EL2
[    0.076081] devtmpfs: initialized
[    0.094488] KASLR enabled
[    0.094705] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.094733] futex hash table entries: 2048 (order: 5, 131072 bytes, linear)
[    0.099666] pinctrl core: initialized pinctrl subsystem
[    0.100684] SMBIOS 3.6.0 present.
[    0.100694] DMI: NVIDIA NVIDIA Jetson AGX Orin Developer Kit/Jetson, BIOS 36.3.0-gcid-36191598 05/06/2024
[    0.101255] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.103263] DMA: preallocated 4096 KiB GFP_KERNEL pool for atomic allocations
[    0.103639] DMA: preallocated 4096 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.103983] DMA: preallocated 4096 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.104037] audit: initializing netlink subsys (disabled)
[    0.104177] audit: type=2000 audit(0.100:1): state=initialized audit_enabled=0 res=1
[    0.105739] thermal_sys: Registered thermal governor 'step_wise'
[    0.105742] thermal_sys: Registered thermal governor 'user_space'
[    0.105743] thermal_sys: Registered thermal governor 'power_allocator'
[    0.109773] cpuidle: using governor menu
[    0.109987] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.110103] ASID allocator initialised with 32768 entries
[    0.112064] Serial: AMBA PL011 UART driver
[    0.121921] tegra-mc 2c00000.memory-controller: pcie4w: write @0x0000000000000000: EMEM address decode error (EMEM decode error)
[    0.122852] 31d0000.serial: ttyAMA0 at MMIO 0x31d0000 (irq = 118, base_baud = 0) is a SBSA
[    0.134242] platform bpmp: Fixing up cyclic dependency with 2c60000.external-memory-controller
[    0.143326] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    0.143332] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[    0.143333] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.143334] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[    0.144277] cryptd: max_cpu_qlen set to 1000
[    0.146062] ACPI: Interpreter disabled.
[    0.148040] iommu: Default domain type: Translated
[    0.148045] iommu: DMA domain TLB invalidation policy: strict mode
[    0.148369] SCSI subsystem initialized
[    0.148550] vgaarb: loaded
[    0.148717] usbcore: registered new interface driver usbfs
[    0.148738] usbcore: registered new interface driver hub
[    0.148755] usbcore: registered new device driver usb
[    0.149177] pps_core: LinuxPPS API ver. 1 registered
[    0.149183] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.149194] PTP clock support registered
[    0.149269] EDAC MC: Ver: 3.0.0
[    0.150247] Registered efivars operations
[    0.150863] FPGA manager framework
[    0.150947] Advanced Linux Sound Architecture Driver Initialized.
[    0.151819] clocksource: Switched to clocksource arch_sys_counter
[    0.188092] VFS: Disk quotas dquot_6.6.0
[    0.188157] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.188461] pnp: PnP ACPI: disabled
[    0.191863] NET: Registered PF_INET protocol family
[    0.192400] IP idents hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.194747] tcp_listen_portaddr_hash hash table entries: 16384 (order: 6, 262144 bytes, linear)
[    0.194787] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.194824] TCP established hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.195048] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, linear)
[    0.196412] TCP: Hash tables configured (established 262144 bind 65536)
[    0.196629] UDP hash table entries: 16384 (order: 7, 524288 bytes, linear)
[    0.196697] UDP-Lite hash table entries: 16384 (order: 7, 524288 bytes, linear)
[    0.196949] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.197403] RPC: Registered named UNIX socket transport module.
[    0.197409] RPC: Registered udp transport module.
[    0.197410] RPC: Registered tcp transport module.
[    0.197411] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.197426] PCI: CLS 0 bytes, default 64
[    0.197676] Unpacking initramfs...
[    0.205363] hw perfevents: enabled with armv8_cortex_a78 PMU driver, 7 counters available
[    0.205771] kvm [1]: IPA Size Limit: 48 bits
[    0.205810] kvm [1]: GICv3: no GICV resource entry
[    0.205819] kvm [1]: disabling GICv2 emulation
[    0.205856] kvm [1]: GIC system register CPU interface enabled
[    0.205941] kvm [1]: vgic interrupt IRQ9
[    0.206125] kvm [1]: VHE mode initialized successfully
[    0.209202] Initialise system trusted keyrings
[    0.209326] workingset: timestamp_bits=42 max_order=23 bucket_order=0
[    0.212652] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.213028] NFS: Registering the id_resolver key type
[    0.213051] Key type id_resolver registered
[    0.213055] Key type id_legacy registered
[    0.213090] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.213109] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[    0.213139] ntfs: driver 2.1.32 [Flags: R/O].
[    0.213324] 9p: Installing v9fs 9p2000 file system support
[    0.225119] Key type asymmetric registered
[    0.225126] Asymmetric key parser 'x509' registered
[    0.225201] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245)
[    0.225209] io scheduler mq-deadline registered
[    0.225212] io scheduler kyber registered
[    0.235369] EINJ: ACPI disabled.
[    0.247225] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.249096] SuperH (H)SCI(F) driver initialized
[    0.249439] msm_serial: driver initialized
[    0.250033] printk: console [ttyTCU0] enabled
[    0.250875] arm-smmu 8000000.iommu: probing hardware configuration...
[    0.250881] arm-smmu 8000000.iommu: SMMUv2 with:
[    0.250885] arm-smmu 8000000.iommu:  stage 1 translation
[    0.250887] arm-smmu 8000000.iommu:  stage 2 translation
[    0.250888] arm-smmu 8000000.iommu:  nested translation
[    0.250902] arm-smmu 8000000.iommu:  stream matching with 128 register groups
[    0.250906] arm-smmu 8000000.iommu:  128 context banks (0 stage-2 only)
[    0.250929] arm-smmu 8000000.iommu:  Supported page sizes: 0x61311000
[    0.250931] arm-smmu 8000000.iommu:  Stage-1: 48-bit VA -> 48-bit IPA
[    0.250933] arm-smmu 8000000.iommu:  Stage-2: 48-bit IPA -> 48-bit PA
[    0.251944] arm-smmu 10000000.iommu: probing hardware configuration...
[    0.251948] arm-smmu 10000000.iommu: SMMUv2 with:
[    0.251950] arm-smmu 10000000.iommu:         stage 1 translation
[    0.251950] arm-smmu 10000000.iommu:         stage 2 translation
[    0.251951] arm-smmu 10000000.iommu:         nested translation
[    0.251954] arm-smmu 10000000.iommu:         stream matching with 128 register groups
[    0.251958] arm-smmu 10000000.iommu:         128 context banks (0 stage-2 only)
[    0.251965] arm-smmu 10000000.iommu:         Supported page sizes: 0x61311000
[    0.251968] arm-smmu 10000000.iommu:         Stage-1: 48-bit VA -> 48-bit IPA
[    0.251969] arm-smmu 10000000.iommu:         Stage-2: 48-bit IPA -> 48-bit PA
[    0.252314] arm-smmu 12000000.iommu: probing hardware configuration...
[    0.252319] arm-smmu 12000000.iommu: SMMUv2 with:
[    0.252321] arm-smmu 12000000.iommu:         stage 1 translation
[    0.252321] arm-smmu 12000000.iommu:         stage 2 translation
[    0.252323] arm-smmu 12000000.iommu:         nested translation
[    0.252326] arm-smmu 12000000.iommu:         stream matching with 128 register groups
[    0.252327] arm-smmu 12000000.iommu:         128 context banks (0 stage-2 only)
[    0.252337] arm-smmu 12000000.iommu:         Supported page sizes: 0x61311000
[    0.252338] arm-smmu 12000000.iommu:         Stage-1: 48-bit VA -> 48-bit IPA
[    0.252339] arm-smmu 12000000.iommu:         Stage-2: 48-bit IPA -> 48-bit PA
[    0.260671] loop: module loaded
[    0.261562] megasas: 07.717.02.00-rc1
[    0.265480] tun: Universal TUN/TAP device driver, 1.6
[    0.266119] thunder_xcv, ver 1.0
[    0.266148] thunder_bgx, ver 1.0
[    0.266166] nicpf, ver 1.0
[    0.266995] hclge is initializing
[    0.267026] hns3: Hisilicon Ethernet Network Driver for Hip08 Family - version
[    0.267033] hns3: Copyright (c) 2017 Huawei Corporation.
[    0.267066] e1000: Intel(R) PRO/1000 Network Driver
[    0.267073] e1000: Copyright (c) 1999-2006 Intel Corporation.
[    0.267101] e1000e: Intel(R) PRO/1000 Network Driver
[    0.267105] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    0.267140] igb: Intel(R) Gigabit Ethernet Network Driver
[    0.267146] igb: Copyright (c) 2007-2014 Intel Corporation.
[    0.267165] igbvf: Intel(R) Gigabit Virtual Function Network Driver
[    0.267171] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[    0.267394] sky2: driver version 1.30
[    0.268059] VFIO - User Level meta-driver version: 0.3
[    0.269609] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.269634] ehci-pci: EHCI PCI platform driver
[    0.269674] ehci-platform: EHCI generic platform driver
[    0.269756] ehci-orion: EHCI orion driver
[    0.269819] ehci-exynos: EHCI Exynos driver
[    0.269897] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.269938] ohci-pci: OHCI PCI platform driver
[    0.269965] ohci-platform: OHCI generic platform driver
[    0.270032] ohci-exynos: OHCI Exynos driver
[    0.270467] usbcore: registered new interface driver usb-storage
[    0.272293] i2c_dev: i2c /dev entries driver
[    0.273591] pps pps0: new PPS source ktimer
[    0.273600] pps pps0: ktimer PPS source registered
insmod /lib/modules/5.15.136-voyager-1.0.1-tegra/kernel/drivers/phy/tegra/phy-tegra194-p2u.ko ine
gistered
[    0.275702] device-mapper: ioctl: 4.45.0-ioctl (2021-03-22) initialised: dm-devel@redhat.com
[    0.277590] sdhci: Secure Digital Host Controller Interface driver
[    0.277596] sdhci: Copyright(c) Pierre Ossman
[    0.278028] Synopsys Designware Multimedia Card Interface Driveinsmod /lib/modules/5.27.15.13862oyager-1.0.1-teci-/kernel/drivers/pci/controller/dwc/pcie-tegra194F driver
per
[    0.279751] ledtrig-cpu: registered to indicate activity on CPUs
[    0.280679] SMCCC: SOC_ID: ID = jep106:036b:0234 Revision = 0x00000401
[    0.281215] tegra-bpmp bpmp: Adding to iommu group 0
[    0.282608] tegra-bpmp bpmp: firmware: 2aafe334513692db071c-3b0e1a33cf8
[    0.497624] Freeing initrd memory: 18428K
[    3.989462] clocksource: tsc: mask: 0xffffffffffffff max_cycles: 0xe6a171046, max_idle_ns: 881590405314 ns
[    3.989486] clocksource: osc: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 49772407460 ns
[    3.989489] clocksource: usec: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275 ns
[    3.989715] hid: raw HID events driver (C) Jiri Kosina
[    3.990078] usbcore: registered new interface driver usbhid
[    3.990086] usbhid: USB HID core driver
[    3.994759] optee: probing for conduit method.
[    3.994809] optee: revision 3.22 (e7b5ff3f)
[    4.054225] optee: dynamic shared memory is enabled
[    4.054476] optee: initialized driver
[    4.057172] usbcore: registered new interface driver snd-usb-audio
[    4.058845] NET: Registered PF_PACKET protocol family
[    4.059113] 9pnet: Installing 9P2000 support
[    4.059225] Key type dns_resolver registered
[    4.061104] printk: console [ttyTCU0]: printing thread started
[    4.061256] registered taskstats version 1
[    4.061280] Loading compiled-in X.509 certificates
[    4.074653] Loaded X.509 cert 'Build time autogenerated kernel key: 8927132e0fab911abd0bc6a65961dff6071e6023'
[    4.108100] tegra-gpcdma 2600000.dma-controller: Adding to iommu group 1
[    4.110165] tegra-gpcdma 2600000.dma-controller: GPC DMA driver register 31 channels
[    4.112443] serial-tegra 3100000.serial: RX in PIO mode
[    4.112450] serial-tegra 3100000.serial: TX in PIO mode
[    4.112562] 3100000.serial: ttyTHS1 at MMIO 0x3100000 (irq = 112, base_baud = 0) is a TEGRA_UART
[    4.114475] serial-tegra 3110000.serial: RX in PIO mode
[    4.114480] serial-tegra 3110000.serial: TX in PIO mode
[    4.114574] 3110000.serial: ttyTHS2 at MMIO 0x3110000 (irq = 207, base_baud = 0) is a TEGRA_UART
[    4.119708] tegra_rtc c2a0000.rtc: registered as rtc1
[    4.119728] tegra_rtc c2a0000.rtc: Tegra internal Real Time Clock
[    4.120375] tegra-i2c 3160000.i2c: Adding to iommu group 1
[    4.127886] tegra-i2c 3180000.i2c: Adding to iommu group 1
[    4.132723] tegra-i2c 3190000.i2c: Adding to iommu group 1
[    4.137554] tegra-i2c 31b0000.i2c: Adding to iommu group 1
[    4.142417] tegra-i2c 31c0000.i2c: Adding to iommu group 1
[    4.147266] tegra-i2c 31e0000.i2c: Adding to iommu group 1
[    4.152482] tegra-i2c c240000.i2c: Adding to iommu group 1
[    4.160869] tegra-i2c c250000.i2c: Adding to iommu group 1
[    4.190720] cpufreq: cpufreq_online: CPU0: Running at unlisted initial frequency: 2010000 KHz, changing to: 2035200 KHz
[    4.194315] cpufreq: cpufreq_online: CPU4: Running at unlisted initial frequency: 2009000 KHz, changing to: 2035200 KHz
[    4.197941] sdhci-tegra 3460000.mmc: Adding to iommu group 2
[    4.199150] sdhci-tegra 3400000.mmc: Adding to iommu group 3
[    4.199990] irq: IRQ234: trimming hierarchy from :bus@0:pmc@c360000
[    4.200046] irq: IRQ235: trimming hierarchy from :bus@0:interrupt-controller@f400000-1
[    4.200084] irq: IRQ236: trimming hierarchy from :bus@0:pmc@c360000
[    4.200140] input: gpio-keys as /devices/platform/gpio-keys/input/input0
[    4.604189] ALSA device list:
[    4.604193]   No soundcards found.
[    4.607136] mmc0: CQHCI version 5.10
[    4.607453] sdhci-tegra 3400000.mmc: Got CD GPIO
[    4.643407] mmc0: SDHCI controller on 3460000.mmc [3460000.mmc] using ADMA 64-bit
[    4.656072] irq: IRQ237: trimming hierarchy from :bus@0:interrupt-controller@f400000-1
[    4.656667] mmc1: SDHCI controller on 3400000.mmc [3400000.mmc] using ADMA 64-bit
[    4.657803] Freeing unused kernel memory: 8192K
[    4.657888] Run /init as init process
[    4.675566] Root device found: initrd
[    4.695779] Mount initrd as rootfs and enter recovery mode
[    4.741313] mmc0: Command Queue Engine enabled
[    4.741324] mmc0: new HS400 Enhanced strobe MMC card at address 0001
[    4.741576] mmcblk0: mmc0:0001 DG4064 59.2 GiB
[    4.745994]  mmcblk0: p1 p2 p3 p4 p5 p6 p7 p8 p9 p10 p11 p12 p13 p14 p15
[    4.746506] mmcblk0boot0: mmc0:0001 DG4064 4.00 MiB
[    4.746893] mmcblk0boot1: mmc0:0001 DG4064 4.00 MiB
[    4.747224] mmcblk0rpmb: mmc0:0001 DG4064 4.00 MiB, chardev (511:0)
[    4.751320] mmc1: new HS200 MMC card at address 0001
[    4.751546] mmcblk1: mmc1:0001 DJUD4R 119 GiB
[    4.753953]  mmcblk1: p1 p2
[    4.754138] mmcblk1boot0: mmc1:0001 DJUD4R 31.9 MiB
[    4.754585] mmcblk1boot1: mmc1:0001 DJUD4R 31.9 MiB
[    4.754982] mmcblk1rpmb: mmc1:0001 DJUD4R 4.00 MiB, chardev (511:1)
[    5.752807] tegra194-pcie 14100000.pcie: Adding to iommu group 4
[    5.754568] tegra194-pcie 14100000.pcie: host bridge /bus@0/pcie@14100000 ranges:
[    5.754584] tegra194-pcie 14100000.pcie:      MEM 0x2080000000..0x20a7ffffff -> 0x2080000000
[    5.754590] tegra194-pcie 14100000.pcie:      MEM 0x20a8000000..0x20afffffff -> 0x0040000000
[    5.754593] tegra194-pcie 14100000.pcie:       IO 0x0030100000..0x00301fffff -> 0x0030100000
[    5.754874] tegra194-pcie 14100000.pcie: iATU unroll: enabled
[    5.754876] tegra194-pcie 14100000.pcie: Detected iATU regions: 8 outbound, 2 inbound
[    6.861773] tegra194-pcie 14100000.pcie: Phy link never came up
[    7.863572] tegra194-pcie 14100000.pcie: Phy link never came up
[    7.863625] tegra194-pcie 14100000.pcie: PCI host bridge to bus 0001:00
[    7.863629] pci_bus 0001:00: root bus resource [io  0x0000-0xfffff] (bus address [0x30100000-0x301fffff])
[    7.863633] pci_bus 0001:00: root bus resource [mem 0x20a8000000-0x20afffffff] (bus address [0x40000000-0x47ffffff])
[    7.863635] pci_bus 0001:00: root bus resource [bus 00-ff]
[    7.863637] pci_bus 0001:00: root bus resource [mem 0x2080000000-0x20a7ffffff pref]
[    7.863690] pci 0001:00:00.0: [10de:229e] type 01 class 0x060400
[    7.863877] pci 0001:00:00.0: PME# supported from D0 D3hot
[    7.868960] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[    7.869053] pcieport 0001:00:00.0: Adding to iommu group 4
[    7.869137] pcieport 0001:00:00.0: PME: Signaling with IRQ 201
[    7.869682] pcieport 0001:00:00.0: AER: enabled with IRQ 201
[    7.869930] pci_bus 0001:01: busn_res: [bus 01-ff] is released
[    7.870006] pci 0001:00:00.0: Removing from iommu group 4
[    7.870014] pci_bus 0001:00: busn_res: [bus 00-ff] is released
[    7.871130] tegra194-pcie 14160000.pcie: Adding to iommu group 5
[    7.873871] tegra194-pcie 14160000.pcie: host bridge /bus@0/pcie@14160000 ranges:
[    7.873884] tegra194-pcie 14160000.pcie:      MEM 0x2140000000..0x2427ffffff -> 0x2140000000
[    7.873889] tegra194-pcie 14160000.pcie:      MEM 0x2428000000..0x242fffffff -> 0x0040000000
[    7.873892] tegra194-pcie 14160000.pcie:       IO 0x0036100000..0x00361fffff -> 0x0036100000
[    7.874379] tegra194-pcie 14160000.pcie: iATU unroll: enabled
[    7.874383] tegra194-pcie 14160000.pcie: Detected iATU regions: 8 outbound, 2 inbound
[    7.920537] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x402, iova=0x00000000, fsynr=0x240011, cbfrsynra=0x1013, cb=1
[    7.920672] tegra-mc 2c00000.memory-controller: pcie4w: secure write @0x00000003ffffff00: VPR violation ((null))
[    7.979992] tegra194-pcie 14160000.pcie: Link up
[    7.982658] tegra194-pcie 14160000.pcie: Link up
[    7.982698] tegra194-pcie 14160000.pcie: PCI host bridge to bus 0004:00
[    7.982701] pci_bus 0004:00: root bus resource [io  0x100000-0x1fffff] (bus address [0x36100000-0x361fffff])
[    7.982704] pci_bus 0004:00: root bus resource [mem 0x2428000000-0x242fffffff] (bus address [0x40000000-0x47ffffff])
[    7.982706] pci_bus 0004:00: root bus resource [bus 00-ff]
[    7.982708] pci_bus 0004:00: root bus resource [mem 0x2140000000-0x2427ffffff pref]
[    7.982749] pci 0004:00:00.0: [10de:229c] type 01 class 0x060400
[    7.982893] pci 0004:00:00.0: PME# supported from D0 D3hot
[    7.985606] pci 0004:01:00.0: [14e4:a00b] type 00 class 0x020000
[    7.985785] pci 0004:01:00.0: reg 0x10: [mem 0x00000000-0x0000ffff 64bit]
[    7.985892] pci 0004:01:00.0: reg 0x18: [mem 0x00000000-0x0003ffff 64bit]
[    7.985999] pci 0004:01:00.0: reg 0x20: [mem 0x00000000-0x0000ffff 64bit]
[    7.986888] pci 0004:01:00.0: PME# supported from D0 D3hot D3cold
[    7.987257] pci 0004:01:00.0: reg 0x1ac: [mem 0x00000000-0x0000ffff 64bit]
[    7.987259] pci 0004:01:00.0: VF(n) BAR0 space: [mem 0x00000000-0x0002ffff 64bit] (contains BAR0 for 3 VFs)
[    7.987366] pci 0004:01:00.0: reg 0x1b4: [mem 0x00000000-0x0003ffff 64bit]
[    7.987369] pci 0004:01:00.0: VF(n) BAR2 space: [mem 0x00000000-0x000bffff 64bit] (contains BAR2 for 3 VFs)
[    7.987475] pci 0004:01:00.0: reg 0x1bc: [mem 0x00000000-0x0000ffff 64bit]
[    7.987477] pci 0004:01:00.0: VF(n) BAR4 space: [mem 0x00000000-0x0002ffff 64bit] (contains BAR4 for 3 VFs)
[    8.002072] pci 0004:00:00.0: BAR 14: assigned [mem 0x2428000000-0x24281fffff]
[    8.002076] pci 0004:01:00.0: BAR 2: assigned [mem 0x2428000000-0x242803ffff 64bit]
[    8.002161] pci 0004:01:00.0: BAR 9: assigned [mem 0x2428040000-0x24280fffff 64bit]
[    8.002198] pci 0004:01:00.0: BAR 0: assigned [mem 0x2428100000-0x242810ffff 64bit]
[    8.002282] pci 0004:01:00.0: BAR 4: assigned [mem 0x2428110000-0x242811ffff 64bit]
[    8.002366] pci 0004:01:00.0: BAR 7: assigned [mem 0x2428120000-0x242814ffff 64bit]
[    8.002403] pci 0004:01:00.0: BAR 11: assigned [mem 0x2428150000-0x242817ffff 64bit]
[    8.002438] pci 0004:00:00.0: PCI bridge to [bus 01-ff]
[    8.002442] pci 0004:00:00.0:   bridge window [mem 0x2428000000-0x24281fffff]
[    8.002518] pcieport 0004:00:00.0: Adding to iommu group 5
[    8.002588] pcieport 0004:00:00.0: PME: Signaling with IRQ 203
[    8.003123] pcieport 0004:00:00.0: AER: enabled with IRQ 203
[    8.003408] tegra194-pcie 141a0000.pcie: Adding to iommu group 6
[    8.113148] tegra194-pcie 141a0000.pcie: host bridge /bus@0/pcie@141a0000 ranges:
[    8.113169] tegra194-pcie 141a0000.pcie:      MEM 0x2800000000..0x2b27ffffff -> 0x2800000000
[    8.113174] tegra194-pcie 141a0000.pcie:      MEM 0x2b28000000..0x2b2fffffff -> 0x0040000000
[    8.113178] tegra194-pcie 141a0000.pcie:       IO 0x003a100000..0x003a1fffff -> 0x003a100000
[    8.113699] tegra194-pcie 141a0000.pcie: iATU unroll: enabled
[    8.113703] tegra194-pcie 141a0000.pcie: Detected iATU regions: 8 outbound, 2 inbound
[    8.219987] tegra194-pcie 141a0000.pcie: Link up
[    8.230079] tegra194-pcie 141a0000.pcie: Link up
[    8.230124] tegra194-pcie 141a0000.pcie: PCI host bridge to bus 0005:00
[    8.230127] pci_bus 0005:00: root bus resource [io  0x200000-0x2fffff] (bus address [0x3a100000-0x3a1finsmod /lib/modules/5.15.136-voyager-1.0.1-tegra/kernel/drivers/nvme/host/nvme-core.ko _bu
s 0005:00: root bus resource [mem 0x2b28000000-0x2b2fffffff] (bus address [0x40000000-0x47ffffff])
[    8.230132] pci_bus 0005:00: root buinsmod /lib/modules/5.15.136-voyager-1.0.1-tegra/kernel/drivers/nvme/host/nvme.ko [
  8.230133] pci_bus 0005:00: root bus resource [mem 0x2800000000-0x2b27ffffff pref]
insmod /lib/modules/5.15.136-voyager-1.0.1-tegra/kernel/drivers/thermal/tegra/tegra-bpmp-thermal.ko 0:
[10de:229a] type 01 class 0x060400
[    8.230307] pci 0005:00:00.0: PME# supported from Dinsmod /lib/modules/5.15.13  86-voyager-1.0.1-tegra/kernel/drivers/pwm/pwm-tegra.ko 01:
0: [11f8:4336] type 00 class 0x068000
[    8.233893] pci 0005:01:00.0: regins 0x /lib/modules/5.15.136-voyager-1.0.1-tegra/kernel/drivers/hwmon/pwm-fan.ko fff
fff 64bit pref]
[    8.234164] pci 0005:01:00.0: reg 0x18: [mem 0x0Checking whether device /dev/mmcblk?p1 exist 64
 pref]
[    8.23439Loo5] g for OTA work directory on the device(s): /dev/mmcblk0p120:
em /dev/mmcblk1p1
0-0x03ffffff 64bit pref]
[    8.234624] pci 0005:01:00.0: enabling Extended Tags
[    8.236637] pci 0005:01:00.0: PME# supported from D0 D3hot D3cold
[    8.238166] pci 0005:01:00.0: 126.024 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x8 link at 0005:00:00.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
[    8.238776] pci 0005:01:00.1: [11f8:4336] type 00 class 0x088000
[    8.239146] pci 0005:01:00.1: reg 0x10: [mem 0x00000000-0x007fffff 64bit]
[    8.239847] pci 0005:01:00.1: enabling Extended Tags
[    8.241086] pci 0005:01:00.1: PME# supported from D0 D3hot D3cold
[    8.244561] pci 0005:00:00.0: BAR 15: assigned [mem 0x2800000000-0x2809ffffff 64bit pref]
[    8.244563] pci 0005:00:00.0: BAR 14: assigned [mem 0x2b28000000-0x2b287fffff]
[    8.244565] pci 0005:01:00.0: BAR 2: assigned [mem 0x2800000000-0x2803ffffff 64bit pref]
[    8.244714] pci 0005:01:00.0: BAR 4:Checking whether device /dev/sd?1 exist [m
0x2804000000-0x280Device /dev/fff1 does not exisref
]
Checking whether devi487/dev/nvme?n1p1 exist05:
01:00.0: BAR 0: asDevice /dev/nvme?n1p1 does not exist0x2
000000-0x2808ffffff 64bit pref]
[    8.245115] pci 0005:01:00.1: BAR 0: assigned [mem 0x2b28000000-0x2b287fffff 64bit]
[    8.245246] pci 0005:00:00.0: PCI bridge to [bus 01-ff]
[    8.245257] pci 0005:00:00.0:   bridge window [mem 0x2b28000000-0x2b287fffff]
[    8.245259] pci 0005:00:00.0:   bridge window [mem 0x2800000000-0x2809ffffff 64bit pref]
[    8.245323] pcieport 0005:00:00.0: Adding to iommu group mount: [  /mnt: wrong fs type, bad option, bad superblock on /dev/mmcblk0, missing codepage or helper program, or other error.ME:
gnaling with IRQ 205
[    8.246370] pcieport 0005:0bash: 0:0cannot set terminal process group (-1): Inappropriate ioctl for device205

bash: no [  job control in this shell9]
Finbash-5.1# g OTA work dir on external storage devices
[    8.344346] mount /dev/mmcblk0p1 /mnt
[    8.363918] EXT4-fs (mmcblk0p1): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[    8.365401] is_boot_only_partition /mnt
[    8.379483] OTA work directory /mnt/ota_work is not found on /dev/mmcblk0p1
[    8.385157] mount /dev/mmcblk1p1 /mnt
[    8.409500] EXT4-fs (mmcblk1p1): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[    8.410839] is_boot_only_partition /mnt
[    8.419785] OTA work directory /mnt/ota_work is not found on /dev/mmcblk1p1
[    8.442641] Finding OTA work dir on internal storage device
[    8.443686] mount /dev/mmcblk0 /mnt
[    8.490920] Failed to mount /dev/mmcblk0 on the /mnt
[    8.491928] Failed to run "mount_ota_work_partition /dev/mmcblk0 /mnt"
[    8.492954] OTA work directory is not found on internal and external storage devices
[  267.645667] arm-smmu 12000000.iommu: Unhandled context fault: fsr=0x402, iova=0x00000000, fsynr=0x240011, cbfrsynra=0x1013, cb=1
[  267.645909] tegra-mc 2c00000.memory-controller: pcie4w: secure write @0x00000003ffffff00: VPR violation ((null))

bash-5.1#
bash-5.1#
bash-5.1#
bash-5.1#
bash-5.1#
bash-5.1# ls
bin  dev  etc  init  lib  mnt  proc  root  sbin  sys  tmp  usr  var
bash-5.1#
11

Hi Jerry,

I provided the error boot log when rootfs A/B is disabled, is this useful to you?

12

hello YuanliFu,

may I know what’s your test steps? were you performing OTA update on this target?

13

No, just reboot by a shell, which I put into the /etc/rc.local to make it executed every boot.

reboot.sh:

#!/bin/bash

COUNTER_FILE="/var/lib/reboot_counter"
MAX_REBOOTS=1000

log_to_dmesg() {
    echo "$1" > /dev/kmsg
}

check_flush_error() {
    if dmesg | grep -i "flush error" > /dev/null; then
        echo "check flush error! stop reboot"
        log_to_dmesg "reboot_test: check flush error，stop reboot test"
        exit 1
    fi
}

if [ -f "$COUNTER_FILE" ]; then
    COUNTER=$(cat "$COUNTER_FILE")
else
    COUNTER=0
fi

COUNTER=$((COUNTER + 1))
echo $COUNTER > "$COUNTER_FILE"

echo "$COUNTER reboot - $(date)"
log_to_dmesg "reboot_test: $COUNTER"

check_flush_error

if [ $COUNTER -ge $MAX_REBOOTS ]; then
    echo "$MAX_REBOOTS reboots have been finished"
    log_to_dmesg "reboot_test: $MAX_REBOOTS reboots have been finished"
    exit 0
fi

sleep 150

check_flush_error

echo "prepare $((COUNTER + 1))th reboot..."
log_to_dmesg "reboot_test: prepare $((COUNTER + 1))th reboot"
sync
sleep 5

reboot

rc.local:

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

# Disable the screen blackout and disable the screen lock.
su - pilot -c 'gsettings set org.gnome.desktop.session idle-delay 0'
su - pilot -c 'gsettings set org.gnome.desktop.screensaver lock-enabled false'

/home/pilot/reboot.sh

exit 0
14

hello YuanliFu,

is it a must to stay-on JP-6.0 for testing?
there’re some known issue that might cause file system failure after reboot stress test.

15

Yes, we must test at jetpack-6.0, can you list which issues can cause file system corrupted?

We want to reproduce a mmc0 error by reboot stress test, see “Mmc0 error cause system to freeze”. And in another topic “AGX Orin 64G(DG4064型号eMMC)版本使用中出现mmc0故障报警导致系统卡死”, it mentioned this mmc0 error may be fixed in jetpack-6.1, so if we switch to jetpack-6.1, may not reproduce it, so we must test at jetpack 6.0

Thanks

16

hello YuanliFu,

are you asking the patches to address mmc0 error on JP-6?
please note that, we usually do not back-port the changes.

17

No, I am not sure if higher version of jetpack can fix it, I just want to reproduce on jetpack-6.0 by reboot stress test, but the failure of rootfs stop me from doing test.

18

Hi Jerry,

What’s the root cause of the system automatically switching slots after multiple reboots?

Is there any way to prevent this behavior?

According to the method I followed, the system can recover and the system files can boot normally.

Thanks

19

hi all,

it might due to UEFI leakage.
you may see-also Topic 314058 for some known bug fixes.