UEFI assert issue on Jetson Thor

▒▒INFO: END TASK:PCIE
INFO: enter idle task.
INFO: END TASK:MB▒▒
INFO: enter idle task.
INFO: END TASK:MB▒▒
▒▒ph =▒▒ter idle ta▒▒GPC ▒▒sk.
▒▒> logic map: 0=>0 1=>2 2=>1
▒▒ASSERT [DebugStatusCodeDxe] [NvmExpressDxe] /out/nvidia/bootloader/uefi/JetsonFF-A_RELEASE/edk2/MdeModulePkg/Bus/Pci/NvmExpressDxe/NvmExpressHci.c(778): (Private->Cap.Mpsmin + 12) <= 12

Resetting the system in 5 seconds.
▒▒Reset requested
Rebooting system …

i just enable the pcie controller(C3→uphy0 configuer#6 M.2 NVME) according the developer document in the bpmp-dt and kernel-dt and then use l4t_initrd_flash.sh to flash my custom board …It occurs during the process of flashing, when rebooting before entering the ramdisk.

tegra264-bpmp-3834-0008-4071-xxxx.dtb.txt (13.1 KB)

tegra264-p4071-0000+p3834-0008-nv.dtb.txt (245.5 KB)

tegra264-mb1-bct-pinmux-p3834-xxxx-p4071-0000.dtsi.txt (89.5 KB)

Now according to this topic ASSERT TerminalConIn.c(2078): ((BOOLEAN)(0==1)) - Jetson & Embedded Systems / Jetson Xavier NX - NVIDIA Developer Forums ,When I tried to disable the PCIe controller in BPMP and keep the PCIe controller in the kernel, the following occurred,it seems worse …

▒▒INFO: END TASK:PCIE
INFO: enter idle task.
INFO: END TASK:MB▒▒
INFO: enter idle task.
INFO: END TASK:MB▒▒
▒▒ph =▒▒ter idle ta▒▒GPC ▒▒sk.
        ▒▒> logic map: 0=>0 1=>2 2=>1
▒▒Unhandled Exception in EL3.
x30            = 0x000000000ff4103c
x0             = 0x0000000000000000
x1             = 0x00000000be000011
x2             = 0x0000000000000011
x3             = 0x0000000000000000
x4             = 0x8f145c61617a399f
x5             = 0x0000000000000001
x6             = 0x0000000000016648
x7             = 0x0000000000016680
x8             = 0x0000000001000000
x9             = 0x0000001403813000
x10            = 0x0000000400800000
x11            = 0x0000001f76c03d54
x12            = 0x0000000000016644
x13            = 0x00000000ffffffff
x14            = 0x0000001f7ebaa000
x15            = 0x0000001f7ebaa000
x16            = 0x0000001f799a0358
x17            = 0x0000000000000000
x18            = 0x0000001f799ab250
x19            = 0x000000000ff753f0
x20            = 0x0000000000000000
x21            = 0x0000000000000000
x22            = 0x0000000000000000
x23            = 0x0000000000000000
x24            = 0x0000000000000000
x25            = 0x0000000000000000
x26            = 0x0000000000000000
x27            = 0x0000000000000000
x28            = 0x0000000000000000
x29            = 0x0000000000000000
scr_el3        = 0x000000401c07073d
sctlr_el3      = 0x0000000030cd183f
cptr_el3       = 0x0000000000000100
tcr_el3        = 0x0000000080853510
daif           = 0x00000000000002c0
mair_el3       = 0x00000000004404ff
spsr_el3       = 0x0000000040000309
elr_el3        = 0x0000001f78cb1cd0
ttbr0_el3      = 0x000000000ff95001
esr_el3        = 0x00000000be000011
far_el3        = 0xe7e15bc4eb610fe5
spsr_el1       = 0x0000000000000000
elr_el1        = 0x0000000000000000
spsr_abt       = 0x0000000000000000
spsr_und       = 0x0000000000000000
spsr_irq       = 0x0000000000000000
spsr_fiq       = 0x0000000000000000
sctlr_el1      = 0x0000000030d00980
actlr_el1      = 0x0000000000000000
cpacr_el1      = 0x0000000000300000
csselr_el1     = 0x0000000000000002
sp_el1         = 0x0000000000000000
esr_el1        = 0x0000000000000000
ttbr0_el1      = 0x0000000000000000
ttbr1_el1      = 0x0000000000000000
mair_el1       = 0x0000000000000000
amair_el1      = 0x0000000000000000
tcr_el1        = 0x0000000000000000
tpidr_el1      = 0x0000000000000000
tpidr_el0      = 0x0000000080000000
tpidrro_el0    = 0x0000000000000000
par_el1        = 0x0000000000000800
mpidr_el1      = 0x0000000081000000
afsr0_el1      = 0x0000000000000000
afsr1_el1      = 0x0000000000000000
contextidr_el1 = 0x0000000000000000
vbar_el1       = 0x0000000000000000
cntp_ctl_el0   = 0x0000000000000005
cntp_cval_el0  = 0x000000029634da9f
cntv_ctl_el0   = 0x0000000000000000
cntv_cval_el0  = 0x0000000000000000
cntkctl_el1    = 0x0000000000000000
sp_el0         = 0x0000001f799ab250
isr_el1        = 0x0000000000000040
cpuectlr_el1   = 0xc0007403c0543001
gicd_ispendr regs (Offsets 0x200 - 0x278)
 Offset:                        value
0000000000000200:               0x0000000000000000
0000000000000204:               0x0000000000000000
0000000000000208:               0x0000000000000000
000000000000020c:               0x0000000000000008
0000000000000210:               0x0000000010000000
0000000000000214:               0x0000000000000000
0000000000000218:               0x0000000000000000
000000000000021c:               0x0000000000000000
0000000000000220:               0x0000000000000000
0000000000000224:               0x0000000000000000
0000000000000228:               0x0000000000000000
000000000000022c:               0x0000000000000000
0000000000000230:               0x0000000000000000
0000000000000234:               0x0000000000000000
0000000000000238:               0x0000000080000000
000000000000023c:               0x0000000000000000
0000000000000240:               0x0000000000000000
0000000000000244:               0x0000000000000000
0000000000000248:               0x0000000000000000
000000000000024c:               0x0000000000000000
0000000000000250:               0x0000000000001800
0000000000000254:               0x0000000000000000
0000000000000258:               0x0000000000000000
000000000000025c:               0x0000000000800000
0000000000000260:               0x0000000000000000
0000000000000264:               0x0000000000000000
0000000000000268:               0x0000000000000000
000000000000026c:               0x0000000000000000
0000000000000270:               0x0000000000000000
0000000000000274:               0x0000000000000000
0000000000000278:               0x0000000000000000
000000000000027c:               0x0000000000000000
▒▒swdtimer_timer_cb: poll interval 61 above target 60

flash.sh seems right.

3849 if [ "${CHIPID}" = "0x23" ] || [ "${CHIPID}" = "0x23 9" ] \
3850         || [ "${CHIPID}" = "0x26" ]; then
3851         mkfilesoft tbcdtbfile   "${TBCDTB_FILE}" "${DTB_DIR}/${TBCDTB_FILE}";
3852 else
3853         mkfilesoft tbcdtbfile   "${TBCDTB_FILE}" "${DTB_DIR}/${DTB_FILE}";
3854 fi

should i add the hot-plug-capable = <1>; to the bpmp-dt or reduce the link speed to Gen-1 in bpmp-dt?

Strangely, when I tried using a nvme from a different manufacturer, the flashing worked fine, but this specific model of nvme is supposed to work on Orin.

Hi wpceswpces,

Are you using Jetpack 7.0GA or 7.1EA for your cusotm board?

Could it boot as expected before you did these modification?

Hi,KevinFFF

No, if not modified, PCIe won’t recognize the hard drive. Although there is no assert in UEFI during the flashing process, once it enters the ramdisk, it will fail because the NVMe drive cannot be found. However, the assert can only be bypassed by replacing the NVMe hard drive.

I am mainly concerned about two things:

  1. Why does this error occur, and what is the difference between these two drives?

  2. Why does disabling C3 in bpmp-dt cause an Unhandled Exception in EL3? I am wondering if disabling bpmp-dt but still keeping Kernel-dt might help avoid this issue.

The latest:

For the hard drive that can be successfully flashed, After I re-flashed the PCIe network card firmware,a new assert occurs after rebooting the system, and the issue persists even after power cycling. The system is currently stuck in a reboot loop.Re-flashing using l4t_initrd_flash.sh doesn’t work either, because an assert has already occurred in UEFI, and the system can’t enter the ramdisk for flashing.


▒▒ Sys▒▒▒tem ▒▒5Cve▒▒firm▒▒rsio▒▒ware▒▒n 38▒▒
▒.2.1▒▒025C▒▒-gci▒▒vers▒▒d-42▒▒ion ▒▒0610▒▒38.2▒▒81
▒▒.1-g▒▒▒cid-▒▒4Cda▒▒4206▒▒te 2▒▒1081▒▒025-▒▒
▒09-1▒▒024C▒▒0T19▒▒date▒▒:13:▒▒ 202▒▒20+0▒▒5-09▒▒0:00▒▒-10T▒▒
▒19:1▒▒005;▒▒3:20▒▒001H▒▒+00:▒▒ESC ▒▒00
▒▒  to▒▒▒ ent▒▒5;00▒▒er S▒▒1HES▒▒etup▒▒C   ▒▒.
▒▒r Bo▒▒▒11  ▒▒nter▒▒ to ▒▒ Set▒▒ente▒▒up.
        F11▒▒ot M▒▒   t▒▒anag▒▒o en▒▒er M▒▒ter ▒▒enu.▒▒Boot▒▒
s ▒▒ Man▒▒    ▒▒ager▒▒to e▒▒ Men▒▒nter▒▒u.
▒▒  to▒▒s   ▒▒ll.
        Ent▒▒ ent▒▒er t▒▒er S▒▒o co▒▒hell▒▒ntin▒▒.
E▒▒ue b▒▒nter▒▒oot.▒▒ to ▒▒
.▒▒continue boot.
.▒▒.▒▒.▒▒.▒▒.▒▒.▒▒.▒▒.▒▒.▒▒.▒▒.▒▒

Synchronous Exception at 0x000000▒▒

▒▒1FB8▒▒Sync▒▒A8BA▒▒hron▒▒94
▒▒CpuD▒▒ASSE▒▒Exce▒▒RT [▒▒ptio▒▒Debu▒▒n at▒▒gSta▒▒ 0x0▒▒tusC▒▒0000▒▒odeD▒▒01FB▒▒xe] ▒▒8A8B▒▒[Arm▒▒A94
        ▒▒xe] /out/nvidia/bootloader/uefi/JetsonFF-A_RELEASE/edk2/ArmPkg/Library/DefaultExceptionHandlerLib/AArch64/DefaultExceptionHandler.c(345): ((BOOLEAN)(0==1))

Resetting the system in 5 seconds.
▒▒Reset requested
Rebooting system ...
▒▒▒▒▒▒▒
[0000.092] I> MB1 (version: 0.23.0.1-t264-75019003-35ec65c7)
[0000.092] C> Boot-mode : Coldboot
[0000.092] C> MB1 last_boot_error: 0x0
[0000.092] I> Entry timestamp: 0x00012bd1
[0000.094] C> rst_source: 0x3c, rst_level: 0x1
[0000.098] I> BR-BCT: preprod_dev_sign: 0
[0000.102] I> Socket mask: 0x1
[0000.105] I> Socket id: 0
[0000.107] I> Chip supports UFS HS mode

who can stop this?

Okay, please wait for Jetpack 7.1 GA, which should be released early next month.

Do you mean the issue is specific to one NVMe?

How about you don’t update BPMP-DTB?

Could you try using flash.sh instead to check if it could recover it?

I am curious about why it can work in the first flash but failed after you re-flashed it. What’s your modification may cause this issue?

Hi KevinFFF

It seems that I reproduce this issue in the Thor-dev-kit . I also need to use the pcie C3 port. I have do some test .

if I modify the kenerl dts to enable the pcie C3 port ,make dtbs, copy the dtb to .kernel/dtsb. flash the system with sdk 7.0GA .the T5000 module will be hang in the uefi during the flash process.

here is the dts modify information

“tegra264-p4071-000.dsi”

pcie@a808440000 {

status = "okay";
};

After flash failed ,I modify back to the orginal dts file,make dtbs ,copy dtb files .then flash . that will be flash pass.

Could you please tell me if I have some methods to use the PCIE C3 ?

Hi,KevinFFF

We are still consulting with the vendor regarding the two firmware versions they provided. Another important point is that the new firmware version without PXE works fine, but the version with PXE has issues. The assert occurs right when it’s about to load the boot device. The problem is likely related to some interaction with PXE.However, from the UEFI side, are there any other potential issues or areas that are not fully optimized?

./flash.sh --qspi-only jetson-agx-thor-devkit internal

From near the bottom of following:
./flash.sh --help

Usage: sudo ./flash.sh [options] <target_board> <rootdev>
  Where,
        target board: Valid target board name or 'autodetect'
        rootdev: Proper root device.
    options:
        -b <emc_bctfile> ------ use the specified emc_bct file.
        -c <cfgfile> ---------- Flash partition table config file.
        -d <dtbfile> ---------- device tree file.
        -e <bpmp_file> -------- use the specified bpmp file.
        -f <flashapp> --------- Path to flash application (tegraflash.py)
        -g <bpmp_dtb_file> ---- use the specified bpmp dtb file
        -h -------------------- print this message.
        -i <enc rfs key file>-- key for disk encryption support.
        -k <partition id> ----- partition name or number specified in flash.cfg.
        -l <tbcdtbfile> ------- cpu bootloader dtb file.
        -m <mts preboot> ------ MTS preboot such as mts_preboot_si.
        -n <nfs args> --------- Static nfs network assignments
                                <Client IP>:<Server IP>:<Gateway IP>:<Netmask>
        -o <odmdata> ---------- ODM data.
        -r -------------------- skip building and reuse existing system.img.
        -t <tegraboot> -------- tegraboot binary such as nvtboot.bin
        -u <PKC key file>------ PKC key or PKC keylist used for odm fused board.
                                This option is also used for cpu-bootloader signing in PV key feature.
        -v <SBK key file>------ Secure Boot Key (SBK) key used for ODM fused board.
        -w <wb0boot> ---------- warm boot binary such as nvtbootwb0.bin
        -x <tegraid> ---------- Tegra CHIPID.
        -B <boardid> ---------- BoardId.
        -C <cmdline> ---------- Kernel commandline arguments.
                                WARNING:
                                Each option in this kernel commandline gets
                                higher preference over the values set by
                                flash.sh. In case of NFS booting, this script
                                adds NFS booting related arguments, if -i option
                                is omitted.
        -F <flasher> ---------- Flash server such as cboot.bin.
        -G <file name> -------- Read partition and save image to file.
        -I <initrd> ----------- initrd file. Null initrd is default.
        -K <kernel> ----------- Kernel image file such as zImage or Image.
        -L <bootloader> ------- Bootloader such as cboot.bin or u-boot-dtb.bin.
        -M <mts boot> --------- MTS boot file such as mts_si.
        -N <nfsroot> ---------- i.e. <my IP addr>:/my/exported/nfs/rootfs.
        -R <rootfs dir> ------- Sample rootfs directory.
        -S <size> ------------- Rootfs size in bytes. Valid only for internal
                                rootdev. KiB, MiB, GiB short hands are allowed,
                                for example, 1GiB means 1024 * 1024 * 1024 bytes.
        -T <ext num sectors> ---The number of the sectors of the external storage device.
                                The default value is 119537664 if this option is not set.
        -U <int num sectors> ---The number of the sectors of the internal storage device.
                                The default value is 119537664 if this option is not set.
        -Z -------------------- Print configurations and then exit.
        --no-flash ------------ perform all steps except physically flashing the board.
                                This will create a system.img.
                                If combined with --rcm-boot, the command will generate rcmboot_blob.
                                If combined with --read-info, the command will generate readinfocmd.txt.
        --external-device------ Generate flash images for external devices
        --sparseupdate--------- only flash partitions that have changed. Currently only support SPI flash memory
        --no-systemimg -------- Do not create or re-create system.img.
        --bup ----------------- Generate bootloader update payload(BUP).
        --single-image-bup <part name> Generate specified single image BUP, this must work with --bup.
        --bup-type <type> ----- Generate specific type bootloader update payload(BUP), such as bl or kernel.
        --multi-spec----------- Enable support for building multi-spec BUP.
        --clean-up------------- Clean up BUP buffer when multi-spec is enabled.
        --usb-instance <id> --- Specify the USB instance to connect to;
                                <id> = USB port path (e.g. 3-14).
        --no-root-check ------- Typical usage of this script require root permissions.
                                Pass this option to allow running the script as a
                                regular user, in which case only specific combinations
                                of command-line options will be functional.
        --uefi-keys <keys_conf> Specify UEFI keys configuration file.
        --rcm-boot ------------ Do RCM boot instead of physically flashing the board.
        --sign ---------------- Sign images and store them under "bootloader/signed"
                                directory. The board will not be physically flashed.
        --image --------------- Specify the image to be written into board.
        --boot-chain-flash <c>  Flash only a specific boot chain (ex. "A, "B", "all").
                                Defaults to "all", inputs are case insensitive.
                                Not suitable for production.
        --boot-chain-select <c> Specify booting chain (ex. "A" or "B") after the board is flashed.
                                Defaults to "A", inputs are case insensitive.
        --pv-crt -------------- The certificate for the key that is used to sign cpu_bootloader
        --with-systemimg ------ Generate system images also when using -k option
        --pv-enc <enc_key>----- The encryption key that is used to encrypt cpu_bootloader.
        --uefi-enc <uefi_enc_key> Key file (0x19: 16-byte; 0x23: 32-byte) to encrypt UEFI payloads
        --uda-dir-------------- Directory to store user data that will be encrypted in UDA partition.
        --generic-passphrase -- Use generic passphrase for disk encryption.
        --disable-random-iv --- Disable generation of random IV, SALT1, SALT2 and DERSTR.
        --read-info ----------- Read and display board related info, fuse info (based on fuse_t234.xml),
                                and EEPROM content.
        --reuse-uuid ---------  Reuse uuid which is already generated first time.
        --qspi-only  ---------  Flash QSPI device only.
        --passthrough --------  Additional options to passthrough the tegraflash command
        --read-ramcode -------  Generate read_ramcode script in the flashcmd.txt command script.
        --gen-read-eeprom ----  Generate read_eeprom script with all signed binaries. User later can run
                                read_eeprom.sh from host to dump EEPROM content on a target device after
                                putting the target device in RCM mode. Since all binaries are signed already,
                                there is no key needed even the device has PKC fuse burned.
        --hsm ----------------  Enable HSM for signing (and encrypting) images. This option must be used with
                                -u option (image signing). -v or --pv-enc is optional (image encryption).
        --debug --------------  Enable debug mode, and enable '--keep' option to tegraflash.py
                                For now, the debug mode will enable saving of unencrypted and unsigned files
                                for UEFI encryption/signing debugging.
        --gen-systemimg-only -  Generate system.img only, program then exits.

Don’t know what nvme revision ubuntu 24.04, but this document may explain the variance. There
are more mps entries in this document NVM Express® Base Specification, Revision 2.2.

Figure 36: Offset 0h: CAP – Controller Capabilities
Bits Type Reset Description
55:52 RO Impl Spec
Memory Page Size Maximum (MPSMAX): This field indicates the maximum host
memory page size that the controller supports. The maximum memory page size
is (2 ^ (12 + MPSMAX)). The host shall not configure a memory page size in
CC.MPS that is larger than this value.
For Discovery controllers this field shall be cleared to 0h.

51:48 RO Impl Spec
Memory Page Size Minimum (MPSMIN): This field indicates the minimum host
memory page size that the controller supports. The minimum memory page size is
(2 ^ (12 + MPSMIN)). The host shall not configure a memory page size in CC.MPS
that is smaller than this value.
For Discovery controllers this shall be cleared to 0h.

Figure 41: Offset 14h: CC – Controller Configuration
Bits Type Reset Description
10:07 RW 0h
Memory Page Size (MPS): This field indicates the host memory page size. The
memory page size is (2 ^ (12 + MPS)). Thus, the minimum host memory page
size is 4 KiB and the maximum host memory page size is 128 MiB. The value
set by host software shall be a supported value as indicated by the
CAP.MPSMAX and CAP.MPSMIN fields. This field describes the value used for
PRP entry size. This field shall only be modified when CC.EN is cleared to ‘0’.
For Discovery controllers this property shall be cleared to 0h


NVMe® over PCIe® Transport Specification, Revision 1.2

Figure 50: Offset PXCAP + 4h: PXDCAP – PCI Express Device Capabilities

02:00 RO Impl Spec Max_Payload_Size Supported (MPS): This field indicates the maximum payload size that the Function may support for TLPs.

Figure 51: Offset PXCAP + 8h: PXDC – PCI Express Device
Max_Payload_Size (MPS): This field sets the maximum TLP payload size for the
Function. As a receiver, the Function shall handle TLPs as large as the set value. As
a transmitter, the Function shall not generate TLPs exceeding the set value. Functions
that support only the 128-byte max payload size are permitted to hardwire this field to 0h.

I’m not sure about this as I cannot reproduce it on the devkit with the NVMe SSD we have.

Is the page size on this problematic NVMe different from other 2 working NVMe?

Flash.sh script would not need the board booting up to initrd for the flashing.

Hi,KevinFFF

We contacted the disk manufacturer and have already resolved the issue. Thank you for your help.