Custom carrier board debugging/not booting

Autonomous Machines Jetson & Embedded Systems Jetson AGX Orin
, ,
1

Hi,
We are currently trying to boot our jetson orin on our custom carrier board which has been design following the devkit. It is configurated with 1 usb-a port on USB0 port of the jetson. The power on sequence seems to be working fine and we detect nvidia-corp when doing lsusb. However, since the jetson doesnt seem to boot, we are trying to get the boot log from UART console. The problem with our carrier board is that we only have access to UART5 which is not the default boot log port i think. So we would need to change default output port so that we can get the boot log and understand what is stopping the jetson from booting. Another way would be to record the boot log, that way we could switch back to the dev kit and go read it.

Please let us know what you think about our situation and how to fix it!

Thank you and have a great day!

Loic & Joel

3

Hi Loic & Joel,

What’s the Jetpack version in use?

You could refer to the following thread about the UART usage in AGX Orin.
Unable to correspond to "func" and "uart_tx" - #3 by KevinFFF
UART3 (/dev/ttyTCU0) is the default debug UART port, which is also from the microUSB port for AGX Orin devkit.
Please review the HW design of your custom board for this UART and compare with the devkit. It should output some messages if you are using the default BSP package.

4

5.0.2 if i remember correctly! We coukd try to install the newest version if it helps!

I will try using this port, and will let you know! Thank you!

5

Hi Kevin!

We believe to have found a possible source why the jetson is not booting properly :

image
image852×364 32.6 KB

So now I would like to know how to do this modification to the dtsi file yet I’m not sure where it is and if I need to reflash the whole system or this is a config file that is read each time the jetson is powered up.

Could you point me in the right direction please?

UPDATE:

I have followed those instructions, and modified the file like it was specified (tegra234-mb2-bct-common.dtsi).
https://docs.nvidia.com/jetson/archives/r35.1/DeveloperGuide/text/IN/QuickStart.html#assumptions

After having successfully flashed the jetson, i tried plugging it back to our custom carrier board. Yet it still doesnt seem to boot; I can still see the USB device but not ping it (ping 192.168.55.1).

I feel like we are going to need the UART debug log to be able to understand why. If you have any other idea to help us make it work, I would love to hear them!

For the UART log, we are going to need to change the pin configuration to UART5_TX and UART5_RX instead of the UART3 since it is the only UART interface connected to the Jetson 699 pins connector. Can you tell us how to do that?

->EndOfUpdate

Thank you alot for your precious help!

Loic

6

If you are using the custom carrier board w/o EEPROM, you would need to do this modification.

What do you mean about “the only UART interface connected to the Jetson 699 pins connector”? I would suggest checking with your HW team about where’s the output of UART3.

Why you could only access UART5? Could you ask HW team to rework for UART3 debug output? It should be just 3 pins (TXD/RXD/GND).

7

Hi kevin!

Sorry if I wasn’t clear!

Here are the schematics for the jetson orin UART ports :

image
image715×556 47.2 KB

As you can see, we only have access to the UART5 interface. To rework and add a physical interface to UAR3 would mean to redesign and fabricate our carrier board which would be very expensive and long. So that is why I want to switch the debug output from UART3 to UART5.

Here is what I tried :
Following the misc configuration page (Miscellaneous Configuration — Jetson Linux Developer Guide documentation), I changed the uart_instance value from <2> to <1> since we deducted that it would be UARTB which in the pinmux is showed to be UART5. However, when trying to boot the devkit we have an error (showed at the end of this message). What are we doing wrong?

If it’s not possible, would there be any alternative like reading previous bootlogs on your devkit? That would let us try to boot on our custom carrier and then change to your devkit and read the previous boot log.

Sorry if what I’m saying is not correct, i’m pretty new to custom carrier boards!

Thank you for your help!
Loic

PS: I can send you by email the full HW schematics if that can help, just let me know!

Error message (output on host from terminal with this command: sudo minicom -D /dev/ttyACM0 -b 115200:

> Welcome to minicom 2.7.1
> 
>  
> 
> OPTIONS: I18n 
> Compiled on Dec 23 2019, 02:06:26.
> Port /dev/ttyACM0, 16:01:53
> 
>  
> 
> Press CTRL-A Z for help on special keys
> 
>  
> 
> ����spe: early_init
> vic initialized
> tsc initialized
> aon lic initialized
> spe: tag is 243b265b351d6bb9cc7b2e3acc5d90f8
> spe: SafeRTOS v8.4
> spe: init
> scheduler initialized
> aon hsp initialized
> tag initialized
> tcu initialized
> bpmp ipc initialized
> spe: late init
> cpu_nic clock initialized
> apb clock initialized                                                           
> pm initialized                                                                  
> bpmp hsp initialized                                                            
> top1 hsp initialized                                                            
> ccplex ipc initialized                                                          
> spe: start scheduler                                                            
> ��                                                                              
>   I> Task: Trigger mailbox for PSC-BL1 exit (0x50016afc)                        
> I> Sending opcode 0x4d420802 to psc                                             
> ��INFO: Entering psc_monitor_init!                                              
> INFO: GSC22 BOM:0x848002000 SIZE:0x1000000 CLIENT_ACCESS1:00180000              
> INFO: PSCFW BUILD VERSION: 8a33b23-73b589c-8a15f76-rel-t234                     
> INFO: mstatus:0xa00000808                                                       
> INFO: Supervisor entry_point:c108c00                                            
> ��I> Received ACK from psc                                                      
> I> Tas��INFO: MONITOR: user task addr:0x848022000, blob offset:0x00020000       
> INFO: MONITOR: populated user images:13                                         
> INFO: mret to Supervisor!                                                       
> INFO: psc supervisor init.                                                      
> INFO: psc_irq_init...                                                           
> INFO: enter idle task.                                                          
> ��                                                                              
>   bpmp: socket 0                                                                
> bpmp: base binary md5 is 1377b684fe55be78e1d7fc3e0f143b55                       
> bpmp: combined binary md5 is 463f4d5b75234b74bb7f05b524d2a015                   
> bpmp: firmware tag is 463f4d5b75234b74bb7f-1377b684fe5                          
> initialized vwdt                                                                
> initialized mail_early                                                          
> initialized fuse                                                                
> initialized vfrel                                                               
> initialized hwwdt                                                               
> initialized adc                                                                 
> fmon_populate_monitors: found 199 monitors                                      
> initialized fmon                                                                
> initialized mc                                                                  
> initialized reset                                                               
> initialized uphy_early                                                          
> initialized emc_early                                                           
> 471 clocks registered                                                           
> initialized clk_mach                                                            
> initialized clk_cal_early                                                       
> initialized clk_mach_early_config                                               
> initialized io_dpd                                                              
> initialized soctherm                                                            
> initialized tj_init                                                             
> initialized regime                                                              
> initialized i2c                                                                 
> vrmon_dt_init: vrmon node not found                                             
> vrmon_chk_boot_state: found 0 rail monitors                                     
> initialized vrmon                                                               
> initialized regulator                                                           
> ��k: Start secure NOR provision (0x5001997c)                                    
> I> Task: Load FSI keyblob (0x50016f20)                                          
> I> Skipping FSI key blob copy                                                   
> I> Task: Unhalt��initialized avfs_clk_platform                                  
> initialized powergate                                                           
> �� AUXPs (0x50026a7c)                                                           
> I> SCE unhalt skipped                                                           
> I> Unhalting RCE                                                                
> I> RCE unhalt successful                                                        
> I> DCE unhal��initialized dvs                                                   
> initialized clk_mach_config                                                     
> initialized pm                                                                  
> initialized suspend                                                             
> initialized strap                                                               
> initialized mce_dbell                                                           
> ��[     0.773418] Camera-FW on t234-rce-safe started                            
> TCU early console enabled.                                                      
> ��t successful                                                                  
> I> APE unhalt skipped                                                           
> I> FSI unhalt skipped                                                           
> I> Task: Load CPUBL (0x50017f04)                                                
> I> Task: Load TOS (0x50018608)                                                  
> I> Tas��                                                                        
>         ��k: Load and authenticate registered FWs (0x5001ce48)                  
> I> Partition name: A_cpu-bootloader                                             
> I> Size of partition: 3670016                                                   
> I> Binary@ device:3/0 block-24832 (partition size: 0x380000), name: A_cpu-bootlr
> ��initialized emc                                                               
> initialized emc_mrq                                                             
> initialized clk_cal                                                             
> initialized uphy_dt                                                             
> initialized uphy_mrq                                                            
> HSIO UPHY reset has been de-asserted 0x0                                        
> initialized uphy                                                                
> initialized pg_late                                                             
> initialized pg_mrq_init                                                         
> swdtimer_init: 0 reg polling start w period 47 ms                               
> initialized swdtimer                                                            
> initialized hwwdt_late                                                          
> initialized bwmgr                                                               
> initialized thermal_host_trip                                                   
> initialized thermal_mrq                                                         
> initialized oc_mrq                                                              
> initialized reset_mrq                                                           
> initialized mail_mrq                                                            
> initialized fmon_mrq                                                            
> initialized clk_mrq                                                             
> initialized avfs_mrq                                                            
> initialized i2c_mrq                                                             
> initialized tag_mrq                                                             
> initialized bwmgr_mrq                                                           
> initialized console_mrq                                                         
> ��I> Partition name: A_secure-os                                                
> I> Size of partition: 4194304                                                   
> I> Binary@ device:3/0��missing prod DT calibration data for 199 fmons           
> initialized clk_sync_fmon_post                                                  
> �� block-32000 (partition size: 0x400000), name: A_secure-os                    
> I> MB2-params @ 0x40060000                                                      
> I> cpubl: Authentication Finalize Done                                          
> I> Binary cpubl loaded successfully at 0x82ea00000                              
> ��initialized clk_cal_late                                                      
> initialized noc_late                                                            
> initialized cvc                                                                 
> ��I> tos: Authentication Finalize Done                                          
> I> Binary tos loaded successfully��initialized avfs_clk_mach_post               
> initialized avfs_clk_platform_post                                              
> initialized cvc_late                                                            
> initialized rm                                                                  
> initialized console_late                                                        
> handling unreferenced clks                                                      
> enable can1_core                                                                
> enable can1_host                                                                
> enable can2_core                                                                
> enable can2_host                                                                
> enable pwm3                                                                     
> enable mss_encrypt                                                              
> enable maud                                                                     
> enable pllg_ref                                                                 
> enable dsi_core                                                                 
> enable aza_2xbit                                                                
> enable pllc4_muxed                                                              
> enable xusb_ss                                                                  
> enable xusb_fs                                                                  
> enable xusb_falcon                                                              
> enable xusb_core_mux                                                            
> enable dsi_lp                                                                   
> enable sdmmc_legacy_tm                                                          
> enable uartb                                                                    
> dvs_clk_state: failed uartb opp(rate 408000000, uv 0)                           
> 
> FATAL ERROR [FILE=platform/drivers/clk/clk-fwk/clk.c, ERR_UID=4059]: assertion d
> r0_usr  0x5009496d                                                              
> r1_usr  0x00000fdb                                                              
> r2_usr  0x50003058                                                              
> r3_usr  0x00000000                                                              
> r4_usr  0x50001f20                                                              
> r5_usr  0x50003058                                                              
> r6_usr  0x00000fdb                                                              
> r7_usr  0x5009496d                                                              
> r8_usr  0x00000000      r8_fiq  0x00000000                                      
> r9_usr  0x500538e5      r9_fiq  0x00000000                                      
> r10_usr 0x50181f98      r10_fiq 0x00000000                                      
> r11_usr 0x00000001      r11_fiq 0x00000000                                      
> r12_usr 0x40000000      r12_fiq 0x00000000                                      
> sp_usr  0x50181e20      sp_fiq  0x5001a620      sp_irq  0x5001a600              
> sp_svc  0x5001a400      sp_abt  0x5001ac00      sp_und  0x5001ab70              
> lr_usr  0x500030a1      lr_fiq  0x00000000      lr_irq  0x50071bca              
> lr_svc  0x5000a6a8      lr_abt  0x00000000      lr_und  0x50001f24              
> pc 0x50001f20                                                                   
> spsr 0x60000010                                                                 
> fpscr 0x80000010                                                                
> 00: base: 50000000 size: 00800000 XN P_RW U_RW Non-shareable outer: WB, no WA iA
> 01: base: 00000000 size: 40000000 XN P_RW U_RW Shareable strongly-ordered       
> 02: base: 00000000 size: 00000040 X  P_RO U_NA Non-shareable outer: WB, no WA iA
> 03: base: 50000000 size: 00010000 X  P_RO U_RO Non-shareable outer: WB, no WA iA
> 04.1: base: 50020000 size: 00020000 X  P_RO U_RO Non-shareable outer: WB, no WAA
> 04.2: base: 50040000 size: 00020000 X  P_RO U_RO Non-shareable outer: WB, no WAA
> 04.3: base: 50060000 size: 00020000 X  P_RO U_RO Non-shareable outer: WB, no WAA
> 04.4: base: 50080000 size: 00020000 X  P_RO U_RO Non-shareable outer: WB, no WAA
> 04.5: base: 500a0000 size: 00020000 X  P_RO U_RO Non-shareable outer: WB, no WAA
> 05: base: 500c0000 size: 00020000 XN P_RW U_RW Non-shareable outer: WB, no WA iA
> 06: base: 50180000 size: 00080000 XN P_RO U_NA Non-shareable outer: WB, no WA iA
> 07.7: base: 507c0000 size: 00040000 XN P_RO U_RO Non-shareable outer: WB, no WAA
> 08: base: 50202000 size: 00002000 XN P_RW U_RW Non-shareable outer: WT, no WA iA
> 10: base: 40070000 size: 00010000 XN P_RW U_RW Non-shareable outer: WB, no WA iA
> 15: base: 50180000 size: 00002000 XN P_RW U_RW Non-shareable outer: WB, no WA iA
> Bootstrap@0x50218e2c sp 0x50181e20 stack: 50180000 - 50181ffc                   
> Per task regions for Bootstrap                                                  
> 04: base: 50180000 size: 00002000 XN P_RW U_RW Non-shareable outer: WB, no WA iA
> Enable MMIO: 1, Enable data R/W: 1                                              
> call stack:                                                                     
> sp 0x50181e20 pc 0x50001f20                                                     
> sp 0x50181e20 pc 0x500030a0                                                     
> sp 0x50181e38 pc 0x5000307c                                                     
> sp 0x50181e50 pc 0x00000002                                                     
> eht_idx_find: 0x00000002 not a valid code address                               
> no eidx for 0x00000002                                                          
> �� at 0x83fdfe000                                                               
> I> Relocating OP-TEE dtb from: 0x83feff770 to 0x82ee30000, size: 0x1976         
> I> [0] START: 0x80000000, SIZE: 0x7aee70000                                     
> I> [1] START: 0x836000000, SIZE: 0x2000000                                      
> I> Setting NS memory ranges to OP-TEE dtb finished.                             
> I> Partition name: A_eks                                                        
> I> Size of partition: 262144                                                    
> I> Binary@ device:3/0 block-44288 (partition size: 0x40000), name: A_eks        
> I> eks: Authentication Finalize Done                                            
> I> Binary eks loaded successfully at 0x830000400                                
> I> EKB detected (length: 0x410) @ VA:0x830000400                                
> I> Task: Prepare TOS params (0x50018580)                                        
> I> Setting EKB blob info to OPTEE dtb finished.                                 
> I> Setting OPTEE arg3: 0x82ee30000                                              
> I> Task: OEM SC7 context save (0x500197dc)                                      
> I> OEM sc7 context saved                                                        
> I> Task: Disable MSS perf stats (0x50026b08)                                    
> I> Task: Program display sticky bits (0x50026a84)                               
> I> Task: Storage device deinit (0x50001eec)                                     
> I> Task: SMMU external bypass disable (0x50016a60)                              
> I> Task: SMMU init (0x5001697c)                                                 
> I> Task: Program GICv3 registers (0x50026ba8)                                   
> I> Task: Audit firewall settings (0x50023bd0)                                   
> I> Task: Bootchain failure check (0x50002434)                                   
> I> Current Boot-Chain Slot: 0                                                   
> I> BR-BCT Boot-Chain is 0, and status is 1. Set UPDATE_BRBCT bit to 0           
> I> MB2 finished                                                                 
> 
> ��NOTICE:  BL31: v2.6(release):07eea4970                                        
> NOTICE:  BL31: Built : 07:55:15, Mar 19 2023                                    
> I/TC:                                                                           
> I/TC: Non-secure external DT found                                              
> I/TC: OP-TEE version: 3.19 (gcc version 9.3.0 (Buildroot 2020.08)) #2 Sun Mar 14
> I/TC: WARNING: This OP-TEE configuration might be insecure!                     
> I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecturel
> I/TC: Primary CPU initializing                                                  
> I/TC: WARNING: Test OEM keys are being used!                                    
> I/TC: This is only for TZ-SE testing and should NOT be used for a shipping prod!
> I/TC: Primary CPU switching to normal world boot                                
> ��ERROR: camera-ip/isp5/isp5.c:1977 [isp5_pm_init] "ERROR: Failed to turn isp p"
> BUG: core/init/init.c:85 [init_all] "*** FIRMWARE INIT FAILED AT LEVEL 95 ***"
8

It seems there’re still some messages output to /dev/ACM0.

About this issue, you could refer to the following thread to fix for your custom board.
ISP power on issue cause Orin boot failed - #12 by Bibek

9

Thank you for your direction toward the topic, I’m pretty sure it would hvae been a reason why the jetson doesnt boot on our carrier.

To be clear, the error message i showed on my last reply was for the DEVKIT and not our custom carrier, since it does not boot correctly.

However, we are still trying to get the debug data from the boot. We are trying to find a way to connect a usb on the uart 3 pins from the jetson connector, but it wont be easy. So I am still trying to find a way to change the debug console output from uart3 to uart5. By finding 2 topics which answers how to enable uartb → uart5, I can tryo to follow theire solution which is to add this to device tree (Enabling the correct UARTs on the Orin NX):

// UART B - UART 2
serial@3110000 {
status = “okay”;
};

However, I dont know where to add those lines and (since it’s probably going to be a .dts file, how to edit those files), Could you tell me if this right ? If so, where is the device tree file that i need to modify?.

Also, we have access to JTAG debug ports, is that something that we should investigate ? Can we output the boot log on JTAG?

I appreciate alot your help, thank you!

Loic

10

You could refer to the following steps to modify device tree.
Let me take AGX Orin devkit as example:

Step 1: Check flash log to know which dtb is in use
copying dtbfile(/home/nvidia/nvidia_sdk/JetPack_5.1_Linux_JETSON_AGX_ORIN_TARGETS/Linux_for_Tegra/kernel/dtb/tegra234-p3701-0000-p3737-0000.dtb)... done.

Step 2: Dissemble the dtb to dts
dtc -I dtb -O dts -o temp.dts tegra234-p3701-0000-p3737-0000.dtb

Step 3: Modify the content in temp.dts

Step 4: Assemble the dts back to dtb
dtc -I dts -O dtb -o tegra234-p3701-0000-p3737-0000.dtb temp.dts
11

Hi Kevin,

I followed your instructions and figured out the serial@3110000 was already enabled. However, i did find this topic for a jetson Nano (UART1 acts as default debug port on TX2, R28.1 - #13 by jburman). It does exactly what i want to do but for a TX2 and also he wants it on UART2 which is different from me. Could you make the same guide adapted for jeson Orin AGX and for UART5 please?

Thanks!

Loic

12

Could you use ssh to log-in to your device and share the dmesg for further check?

So, which UART do you want to use as debug port?

13

Yes!

I want to be able to read the kernel booting on serial console on UART5 instead of UART3.

I joined the output of dmesg in a file.
dmesg.txt (79.6 KB)

14

You need update the kernel cmdline and update the device tree for debug UART on UART5.

1. Kernel cmdline : Linux_for_Tegra/p3701.conf.common

- CMDLINE_ADD="mminit_loglevel=4 console=ttyTCU0,115200 console=ttyAMA0,115200 console=tty0 firmware_class.path=/etc/firmware fbcon=map:0 net.ifnames=0"
+ CMDLINE_ADD="mminit_loglevel=4 console=ttyS1,115200 console=ttyTCU0,115200 console=ttyAMA0,115200 console=tty0 firmware_class.path=/etc/firmware fbcon=map:0 net.ifnames=0"

2. Device tree for both kernel and bootloader : tegra234-p3701-0000-p3737-0000.dtb

    serial@3110000 {
-           compatible = "nvidia,tegra194-hsuart";
+           compatible = "nvidia,tegra20-uart";
            iommus = <0x2 0x4>;
            dma-coherent;
            reg = <0x0 0x3110000 0x0 0x10000>;
            reg-shift = <0x2>;
            interrupts = <0x0 0x71 0x4>;
            nvidia,memory-clients = <0xe>;
            dmas = <0x3 0x9 0x3 0x9>;
            dma-names = "rx", "tx";
            clocks = <0x4 0x9c 0x4 0x66>;
            clock-names = "serial", "parent";
            resets = <0x4 0x65>;
            reset-names = "serial";
            status = "okay";
            phandle = <0x320>;
    };

3. Device tree for bpmp: tegra234-bpmp-3701-0000-3737-0000.dtb

		init {
			nafll_bpmp = <0xc5 0x0 0xc350000 0x0>;
			nafll_seu1 = <0x14f 0x0 0x1c3a9000 0x0>;
			bpmp_cpu_nic = <0x133 0xc5 0x0 0x0>;
			fr_seu1 = <0x14b 0x14f 0x0 0x0>;
			i2c_slow = <0x8a 0x121 0xffffffff 0x0>;
			aon_i2c_slow = <0x75 0x121 0xffffffff 0x0>;
			tach0 = <0x98 0xe 0x30d400 0x0>;
			tach1 = <0x16d 0xe 0x30d400 0x0>;
			i2c1 = <0x30 0x0 0x0 0x2>;
			qspi0_2x_pm = <0xc0 0x0 0x0 0x2>;
			sdmmc4 = <0x7b 0x0 0x0 0x2>;
			seu1 = <0x150 0x0 0x0 0x8>;
			uartc = <0x9d 0x0 0x0 0x8>;
			uarta = <0x9b 0x66 0x1c1b40 0x8>;
+			uartb = <0x9c 0x66 0x1c1b40 0x8>;
			pllref_vcoout = <0x120 0x0 0x0 0xa>;
			eqos_tx_divider = <0x19f 0x0 0xffffffff 0x0>;
			eqos_axi = <0x20 0x0 0x7735940 0x2>;
			eqos_tx = <0x23 0x0 0x0 0x2>;
			eqos_macsec_tx = <0x19e 0x0 0x0 0x2>;
			eqos_ptp_ref = <0x21 0x0 0xffffffff 0x2>;
		};
15

Hi!

I followed your advice and modified both the device tree using the dtc command on tegra234-p3701-0000-p3737-0000.dtb file and the cmdline_add variable inside the conf file p3701.conf.common.

After flashing the jetson (still on the devkit), I soldered a cable to the UART5 TX pin which is located on the M2KeyE. I used a USB to TTL serial cable to connect to the transmitter line. And when I rebooted the jetson i did not receive any data. What did i do wrong?

Thank you and have a great week!

Loic

16

Could you share the current dmesg for further check?

Which pins are you using? (J58 and H58!?)

17

Hi!

Sorry for late reply!

I’m using the H62 pin.

Thank you.

Loic

18

H62 is UART3_TX_DEBUG, not for UART5_TX.

Do you mean that you short H62 to J58?

Please share the current dmesg.

19

Sorry, was thinking about my other post.

I wont be able to spend much time on this topic since we decided to connect the H62 and k60 (uart3 debug) pins using small cables and solder.

Thanks for your help.

20

If you have time, you could refer to the instruction from above steps again to redirect debug UART output from UART5.

22

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.