[Potential Solution] USB HUB Issues on custom carrier board jetson nano

Hello, i wanted to share some experience troubleshooting usb issues with my custom jetson carrier board.
I designed custom carrier board for jetson nano, with dual independent usb hubs:

  • USB5744 (USB3.0 hub connected to USB1 + USBSS)
  • USB2514 (USB2.0 hub connected to USB2)

The issue was that:

  1. On one of the jetson modules (B01) it sometimes worked perfectly, but sometimes during boot process those 2 hubs were not correctly enumerated (BOTH at the same time). So to be clear:
    CASE 1 - everything is working both hubs and devices connected to them as well, stable under load
    CASE 2 - none of them is enumerated during boot (error -71)

  2. On two others jetson modules (also B01) it would not boot, or everything works fine
    CASE 1 - everything is working fine, both hubs and devices detected
    CASE 2 - jetson not boots at all (error: Invalid Oscillator Freq)

My entire board is powered from pololu D24V90F5 step-down which is soldered onboard, and POWER_GOOD signal is used to drive jetson enable logic
Screenshot from 2021-03-26 17-44-43

Now look at the boot process logs (from UART2 debug port)

[0000.118] [L4T TegraBoot] (version 00.00.2018.01-l4t-c536fca9)
[0000.123] Processing in cold boot mode Bootloader 2
[0000.128] A02 Bootrom Patch rev = 1023
[0000.131] Power-up reason: pmc por
[0000.135] No Battery Present
[0000.137] pmic max77620 reset reason
[0000.141] pmic max77620 NVERC : 0x50
[0000.144] RamCode = 0
[0000.146] Platform has DDR4 type RAM
[0000.150] max77620 disabling SD1 Remote Sense
[0000.154] Setting DDR voltage to 1125mv
[0000.158] Serial Number of Pmic Max77663: 0x609f7
[0000.165] Entering ramdump check
[0000.168] Get RamDumpCarveOut = 0x0
[0000.172] RamDumpCarveOut=0x0,  RamDumperFlag=0xe59ff3f8
[0000.177] Last reboot was clean, booting normally!
[0000.181] Sdram initialization is successful 
[0000.185] SecureOs Carveout Base=0x00000000ff800000 Size=0x00800000
[0000.192] Lp0 Carveout Base=0x00000000ff780000 Size=0x00001000
[0000.197] BpmpFw Carveout Base=0x00000000ff700000 Size=0x00080000
[0000.203] GSC1 Carveout Base=0x00000000ff600000 Size=0x00100000
[0000.209] GSC2 Carveout Base=0x00000000ff500000 Size=0x00100000
[0000.215] GSC4 Carveout Base=0x00000000ff400000 Size=0x00100000
[0000.221] GSC5 Carveout Base=0x00000000ff300000 Size=0x00100000
[0000.226] GSC3 Carveout Base=0x000000017f300000 Size=0x00d00000
[0000.243] RamDump Carveout Base=0x00000000ff280000 Size=0x00080000
[0000.249] Platform-DebugCarveout: 0
[0000.252] Nck Carveout Base=0x00000000ff080000 Size=0x00200000
[0000.258] Non secure mode, and RB not enabled.
[0000.262] Read GPT from (4:0)
[0000.451] Csd NumOfBlocks=124735488
[0000.456] Set High speed to 1
[0000.460] Error mask set in wait for cmd complete with error 0x3 in HwSdmmcWaitForCommandComplete func at 278 line 
[0000.470] Command complete wait failed with error 0x3 Interrupt 0x18001
[0000.476] Number of retries left 4
[0000.481] Using GPT Primary to query partitions
[0000.486] Loading Tboot-CPU binary
[0000.495] Verifying TBC in OdmNonSecureSBK mode
[0000.505] Bootloader load address is 0xa0000000, entry address is 0xa0000258
[0000.512] Bootloader downloaded successfully.
[0000.517] Downloaded Tboot-CPU binary to 0xa0000258
[0000.522] MAX77620_GPIO5 configured
[0000.525] CPU power rail is up
[0000.528] CPU clock enabled
[0000.532] Performing RAM repair
[0000.534] Updating A64 Warmreset Address to 0xa00002e9
[0000.540] Loading NvTbootBootloaderDTB
[0000.567] Verifying NvTbootBootloaderDTB in OdmNonSecureSBK mode
[0000.639] Bootloader DTB Load Address: 0x83000000
[0000.644] Loading NvTbootKernelDTB
[0000.668] Verifying NvTbootKernelDTB in OdmNonSecureSBK mode
[0000.740] Kernel DTB Load Address: 0x83100000
[0000.744] Loading cboot binary
[0000.773] Verifying EBT in OdmNonSecureSBK mode
[0000.815] Bootloader load address is 0x92c00000, entry address is 0x92c00258
[0000.822] Bootloader downloaded successfully.
[0000.826] Read PT from (2:0)
[0000.842] Using BFS PT to query partitions 
[0000.846] PT: Partition NCT NOT found ! 
[0000.849] Warning: Find Partition via PT Failed
[0000.854] Next binary entry address: 0x92c00258 
[0000.858] BoardId: 3448
[0000.863] Overriding pmu board id with proc board id
[0000.868] Display board id is not available 
[0000.896] Verifying SC7EntryFw in OdmNonSecureSBK mode
[0000.950] /bpmp deleted
[0000.952] SC7EntryFw header found loaded at 0xff700000
[0001.151] OVR2 PMIC
[0001.153] Bpmp FW successfully loaded
[0001.157] WB0 init successfully at 0xff780000
[0001.161] Set NvDecSticky Bits
[0001.165] GSC2 address ff53fffc value c0edbbcc
[0001.171] GSC MC Settings done
[0001.174] TOS Image length 53680
[0001.177]  Monitor size 53680
[0001.180]  OS size 0
[0001.187] Secure Os AES-CMAC Verification Success!
[0001.192] TOS image cipher info: plaintext
[0001.196] Loading and Validation of Secure OS Successful
[0001.211] SC7 Entry Firmware - 0xff700000, 0x4000
[0001.216] NvTbootPackSdramParams: start. 
[0001.221] NvTbootPackSdramParams: done. 
[0001.225] Tegraboot started after 44812 us
[0001.229] Basic modules init took 848536 us
[0001.233] Sec Bootdevice Read Time = 12 ms, Read Size = 65 KB
[0001.238] Sec Bootdevice Write Time = 0 ms, Write Size = 0 KB
[0001.244] Storage Device Read Time = 108 ms, Read Size = 2226 KB
[0001.250] Storage Device Write Time = 0 ms, Write Size = 0 KB
[0001.255] Next stage binary read took 24150 us
[0001.259] Carveout took -35834 us
[0001.263] CPU initialization took 373123 us
[0001.267] Total time taken by TegraBoot 1209975 us

[0001.271] Starting CPU & Halting co-processor 

64NOTICE:  BL31: v1.3(release):41d46a9cf
NOTICE:  BL31: Built : 21:18:12, Jun 25 2020
ERROR:   Error initializing runtime service trusty_fast
[0001.394] RamCode = 0
[0001.398] LPDDR4 Training: Read DT: Number of tables = 2
[0001.403] EMC Training (SRC-freq: 204000; DST-freq: 1600000)
[0001.416] EMC Training Successful
[0001.419] 408000 not found in DVFS table
[0001.426] RamCode = 0
[0001.429] DT Write: emc-table@204000 succeeded
[0001.435] DT Write: emc-table@1600000 succeeded
[0001.439] LPDDR4 Training: Write DT: Number of tables = 2
[0001.482] 
[0001.484] Debug Init done
[0001.486] Marked DTB cacheable
[0001.489] Bootloader DTB loaded at 0x83000000
[0001.494] Marked DTB cacheable
[0001.497] Kernel DTB loaded at 0x83100000
[0001.501] DeviceTree Init done
[0001.514] Pinmux applied successfully
[0001.518] gicd_base: 0x50041000
[0001.521] gicc_base: 0x50042000
[0001.524] Interrupts Init done
[0001.528] Using base:0x60005090 & irq:208 for tick-timer
[0001.534] Using base:0x60005098 for delay-timer
[0001.538] platform_init_timer: DONE
[0001.542] Timer(tick) Init done
[0001.545] osc freq = 38400 khz
[0001.550] 
[0001.551] Welcome to L4T Cboot
[0001.554] 
[0001.555] Cboot Version: 00.00.2018.01-t210-3a7ad565
[0001.560] calling constructors
[0001.563] initializing heap
[0001.565] initializing threads
[0001.568] initializing timers
[0001.571] creating bootstrap completion thread
[0001.576] top of bootstrap2()
[0001.578] CPU: ARM Cortex A57
[0001.581] CPU: MIDR: 0x411FD071, MPIDR: 0x80000000
[0001.586] initializing platform
[0001.594] Manufacturer: MF = 0xc2, ID MSB = 0x25
[0001.598] ID LSB = 0x36, ID-CFI len = 194 bytes
[0001.603] Macronix QSPI chip present
[0001.606] SPI device register
[0001.609] init boot device
[0001.612] allocating memory for boot device(SPI)
[0001.616] registering boot device
[0001.626] sdmmc node status = okay
[0001.629] sdcard instance = 0
[0001.632] sdcard gpio handle 0x56
[0001.635] sdcard gpio pin 0xc9
[0001.638] sdcard gpio flags 0x0
[0001.641] vmmc-supply 0x99
[0001.644] Instance: 0
[0001.646] Allocating memory for context
[0001.650] enabling clock
[0001.652] sd card init
[0001.654] Check card present and stable
[0001.658] Send command 0
[0001.683] Send command 3
[0001.687] Set RCA for the card
[0001.690] Query card specific data by command 9
[0001.696] Parse CSD data
[0001.699] Send command 7
[0001.709] Calling sd device register
[0001.712] Init sdcard
[0001.715] Allocating memory for boot device
[0001.719] Registering user device
[0001.732] Enable APE clock
[0001.735] Un-powergate APE partition
[0001.738] of_register: registering tegra_udc to of_hal
[0001.743] of_register: registering inv20628-driver to of_hal
[0001.749] of_register: registering ads1015-driver to of_hal
[0001.754] of_register: registering lp8557-bl-driver to of_hal
[0001.760] of_register: registering bq2419x_charger to of_hal
[0001.766] of_register: registering bq27441_fuel_gauge to of_hal
[0001.778] gpio framework initialized
[0001.781] of_register: registering tca9539_gpio to of_hal
[0001.786] of_register: registering tca9539_gpio to of_hal
[0001.792] of_register: registering i2c_bus_driver to of_hal
[0001.797] of_register: registering i2c_bus_driver to of_hal
[0001.803] of_register: registering i2c_bus_driver to of_hal
[0001.808] pmic framework initialized
[0001.812] of_register: registering max77620_pmic to of_hal
[0001.817] regulator framework initialized
[0001.821] of_register: registering tps65132_bl_driver to of_hal
[0001.827] initializing target
[0001.833] gpio_driver_register: register 'tegra_gpio_driver' driver
[0001.841] board ID = D78, board SKU = 0
[0001.845] Skipping Z3!
[0001.850] fixed regulator driver initialized
[0001.868] initializing OF layer
[0001.871] NCK carveout not present
[0001.874] Skipping dts_overrides
[0001.878] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0001.897] I2C Bus Init done
[0001.899] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0001.910] I2C Bus Init done
[0001.912] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0001.923] I2C Bus Init done
[0001.925] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0001.936] I2C Bus Init done
[0001.939] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0001.949] I2C Bus Init done
[0001.952] of_children_init: Ops found for compatible string maxim,max77620
[0001.962] max77620_init using irq 118
[0001.967] register 'maxim,max77620' pmic
[0001.972] gpio_driver_register: register 'max77620-gpio' driver
[0001.978] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0001.988] I2C Bus Init done
[0001.992] NCK carveout not present
[0002.003] Find /i2c@7000c000's alias i2c0
[0002.007] get eeprom at 1-a0, size 256, type 0
[0002.016] Find /i2c@7000c500's alias i2c2
[0002.020] get eeprom at 3-a0, size 256, type 0
[0002.024] get eeprom at 3-ae, size 256, type 0
[0002.028] pm_ids_update: Updating 1,a0, size 256, type 0
[0002.034] I2C slave not started
[0002.037] I2C write failed
[0002.039] Writing offset failed
[0002.042] eeprom_init: EEPROM read failed
[0002.046] pm_ids_update: eeprom init failed
[0002.051] pm_ids_update: Updating 3,a0, size 256, type 0
[0002.081] pm_ids_update: The pm board id is 3448-0000-400
[0002.087] Adding plugin-manager/ids/3448-0000-400=/i2c@7000c500:module@0x50
[0002.095] pm_ids_update: pm id update successful
[0002.100] pm_ids_update: Updating 3,ae, size 256, type 0
[0002.105] I2C slave not started
[0002.108] I2C write failed
[0002.111] Writing offset failed
[0002.114] eeprom_init: EEPROM read failed
[0002.118] pm_ids_update: eeprom init failed
[0002.148] eeprom_get_mac: EEPROM invalid MAC address (all 0xff)
[0002.154] shim_eeprom_update_mac:267: Failed to update 0 MAC address in DTB
[0002.162] eeprom_get_mac: EEPROM invalid MAC address (all 0xff)
[0002.168] shim_eeprom_update_mac:267: Failed to update 1 MAC address in DTB
[0002.177] updating /chosen/nvidia,ethernet-mac node 00:04:4b:ec:63:01
[0002.183] Plugin Manager: Parse ODM data 0x00094000
[0002.194] shim_cmdline_install: /chosen/bootargs: earlycon=uart8250,mmio32,0x70006000 
[0002.209] Find /i2c@7000c000's alias i2c0
[0002.213] get eeprom at 1-a0, size 256, type 0
[0002.222] Find /i2c@7000c500's alias i2c2
[0002.226] get eeprom at 3-a0, size 256, type 0
[0002.231] get eeprom at 3-ae, size 256, type 0
[0002.235] pm_ids_update: Updating 1,a0, size 256, type 0
[0002.240] I2C slave not started
[0002.243] I2C write failed
[0002.246] Writing offset failed
[0002.249] eeprom_init: EEPROM read failed
[0002.253] pm_ids_update: eeprom init failed
[0002.257] pm_ids_update: Updating 3,a0, size 256, type 0
[0002.287] pm_ids_update: The pm board id is 3448-0000-400
[0002.294] Adding plugin-manager/ids/3448-0000-400=/i2c@7000c500:module@0x50
[0002.301] pm_ids_update: pm id update successful
[0002.305] pm_ids_update: Updating 3,ae, size 256, type 0
[0002.311] I2C slave not started
[0002.314] I2C write failed
[0002.316] Writing offset failed
[0002.319] eeprom_init: EEPROM read failed
[0002.323] pm_ids_update: eeprom init failed
[0002.354] Add serial number:1423619092284 as DT property
[0002.362] Applying platform configs
[0002.369] platform-init is not present. Skipping
[0002.373] calling apps_init()
[0002.393] Found 14 GPT partitions in "sd0"
[0002.397] Proceeding to Cold Boot
[0002.400] starting app android_boot_app
[0002.404] Device state: unlocked
[0002.407] display console init
[0002.416] could not find regulator
[0002.419] hdmi cable connected
[0002.455] DT entry for leds-pwm not found
[0004.430] I2C transfer timeout
[0004.433] I2C write failed
[0004.436] Writing offset failed
[0004.439] could not read edid
[0004.443] tmds-config node not found
[0004.447] pmc_set_io_pad_voltage: Error -2 retrieving platform-io-pad-voltagepropsetting 'avdd-io-hdmi-dp' regulator to 1050000 micro volts
[0004.462] setting 'vdd-1v8' regulator to 1800000 micro volts
[0004.470] could not find regulator
[0004.473] could not find regulator
[0004.476] could not find regulator
[0004.505] using default cmu settings
[0004.508] dc_hdmi_enable, starting HDMI initialisation
[0004.515] dc_hdmi_enable, HDMI initialisation complete
[0004.520] list and configure display window
[0004.524] display console init completed
[0004.532] subnode volume_up is not found !
[0004.536] subnode back is not found !
[0004.540] subnode volume_down is not found !
[0004.544] subnode menu is not found !
[0004.548] Gpio keyboard init success
[0004.591] found decompressor handler: lz4-legacy
[0004.606] decompressing blob (type 1)...
[0004.639] load_bmp_blob: panelresolution=480 type=0
[0004.690] decompressor handler not found
[0004.694] load_firmware_blob: Firmware blob loaded, entries=2
[0004.700] -------> se_aes_verify_sbk_clear: 747
[0004.704] se_aes_verify_sbk_clear: Error
[0004.708] bl_battery_charging: connected to external power supply
[0004.716] xusb is supported
[0004.722] error while finding nvidia,portmap
[0005.226] xusb blob version 0 size 124416
[0005.230] firmware size 124416
[0005.234] Firmware timestamp: 0x5da88fc3, Version: 50.25 release
[0005.241] xhci0: 64 bytes context size, 32-bit DMA
[0005.281] usbus0: 5.0Gbps Super Speed USB v3.0
[0005.301] uhub0: <Nvidia XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
[0005.951] uhub0: 9 ports with 9 removable, self powered
[0015.281] get polling status timeout
[0015.285] failed to init xhci or no usb device attached
[0015.295] device_query_partition_size: failed to open partition sd0:MSC !
[0015.302] MSC Partition not found
[0015.308] device_query_partition_size: failed to open partition sd0:USP !
[0015.315] USP partition read failed!
[0015.319] blob_init: blob-partition USP header read failed
[0015.324] android_boot Unable to update recovery partition
[0015.330] kfs_getpartname: name = LNX
[0015.333] Loading kernel from LNX
[0015.431] load kernel from storage
[0015.443] decompressor handler not found
[0015.500] Successfully loaded kernel and ramdisk images
[0015.507] load_bmp_blob: panelresolution=480 type=0
[0015.580] display bmp image done
[0015.583] NCK carveout not present
[0015.587] Skipping dts_overrides
[0015.590] NCK carveout not present
[0015.601] Find /i2c@7000c000's alias i2c0
[0015.604] get eeprom at 1-a0, size 256, type 0
[0015.614] Find /i2c@7000c500's alias i2c2
[0015.618] get eeprom at 3-a0, size 256, type 0
[0015.622] get eeprom at 3-ae, size 256, type 0
[0015.626] pm_ids_update: Updating 1,a0, size 256, type 0
[0015.632] I2C slave not started
[0015.635] I2C write failed
[0015.637] Writing offset failed
[0015.640] eeprom_init: EEPROM read failed
[0015.644] pm_ids_update: eeprom init failed
[0015.648] pm_ids_update: Updating 3,a0, size 256, type 0
[0015.678] pm_ids_update: The pm board id is 3448-0000-400
[0015.685] Adding plugin-manager/ids/3448-0000-400=/i2c@7000c500:module@0x50
[0015.693] pm_ids_update: pm id update successful
[0015.698] pm_ids_update: Updating 3,ae, size 256, type 0
[0015.703] I2C slave not started
[0015.706] I2C write failed
[0015.709] Writing offset failed
[0015.712] eeprom_init: EEPROM read failed
[0015.716] pm_ids_update: eeprom init failed
[0015.746] eeprom_get_mac: EEPROM invalid MAC address (all 0xff)
[0015.752] shim_eeprom_update_mac:267: Failed to update 0 MAC address in DTB
[0015.760] eeprom_get_mac: EEPROM invalid MAC address (all 0xff)
[0015.766] shim_eeprom_update_mac:267: Failed to update 1 MAC address in DTB
[0015.775] updating /chosen/nvidia,ethernet-mac node 00:04:4b:ec:63:01
[0015.781] Plugin Manager: Parse ODM data 0x00094000
[0015.792] shim_cmdline_install: /chosen/bootargs: earlycon=uart8250,mmio32,0x70006000 
[0015.800] Add serial number:1423619092284 as DT property
[0015.809] "bpmp" doesn't exist, creating 
[0015.815] Updated bpmp info to DTB
[0015.820] Updated initrd info to DTB
[0015.823] "proc-board" doesn't exist, creating 
[0015.829] Updated board info to DTB
[0015.832] "pmu-board" doesn't exist, creating 
[0015.838] Updated board info to DTB
[0015.841] "display-board" doesn't exist, creating 
[0015.847] Updated board info to DTB
[0015.850] "reset" doesn't exist, creating 
[0015.855] Updated reset info to DTB
[0015.858] Cmdline: tegraid=21.1.2.0.0 ddr_die=4096M@2048M section=512M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 console=ttyS0,115200n8 debug_uartport=lsport,2 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=0x1000@0xff780000 core_edp_mv=1125 core_edp_ma=4000 tegra_fbmem=0x140000@0x92cb0000 is_hdmi_initialised=1 
[0015.897] DTB cmdline: earlycon=uart8250,mmio32,0x70006000 
[0015.903] boot image cmdline: root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 
[0015.916] Updated bootarg info to DTB
[0015.919] Adding uuid 00000001644612841400000016fc0340 to DT
[0015.925] Adding eks info 0 to DT
[0015.931] WARNING: Failed to pass NS DRAM ranges to TOS, err: -7
[0015.937] Updated memory info to DTB
[0015.946] set vdd_core voltage to 1125 mv
[0015.950] setting 'vdd-core' regulator to 1125000 micro volts
[0023.891] failed to get HID devices
[0023.911] Found secure-pmc; disable BPMP


U-Boot 2016.07-ge6da093be3 (Jun 25 2020 - 21:18:08 -0700)

TEGRA210
Model: NVIDIA P3450-Porg
Board: NVIDIA P3450-PORG
DRAM:  4 GiB
MMC:   Tegra SD/MMC: 0, Tegra SD/MMC: 1
SF: Detected MX25U3235F with page size 256 Bytes, erase size 4 KiB, total 4 MiB
*** Warning - bad CRC, using default environment

In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
Hit any key to stop autoboot:  1  0 
switch to partitions #0, OK
mmc1 is current device
Scanning mmc 1:1...
Found /boot/extlinux/extlinux.conf
Retrieving file: /boot/extlinux/extlinux.conf
852 bytes read in 132 ms (5.9 KiB/s)
1:	primary kernel
Retrieving file: /boot/initrd.img
19993314 bytes read in 1140 ms (16.7 MiB/s)
Retrieving file: /boot/Image
34322440 bytes read in 1537 ms (21.3 MiB/s)
append: tegraid=21.1.2.0.0 ddr_die=4096M@2048M section=512M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 console=ttyS0,115200n8 debug_uartport=lsport,2 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=0x1000@0xff780000 core_edp_mv=1125 core_edp_ma=4000 tegra_fbmem=0x140000@0x92cb0000 is_hdmi_initialised=1  earlycon=uart8250,mmio32,0x70006000  root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 quiet root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 
## Flattened Device Tree blob at 83100000
   Booting using the fdt blob at 0x83100000
   reserving fdt memory region: addr=80000000 size=20000
   Using Device Tree in place at 0000000083100000, end 000000008317d857

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.9.140-tegra (jetson@jetson-desktop) (gcc version 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701] (Linaro GCC 7.3-2018.05) ) #3 SMP PREEMPT Wed Jan 27 11:41:26 CET 2021
[    0.000000] Boot CPU: AArch64 Processor [411fd071]
[    0.000000] OF: fdt:memory scan node memory@80000000, reg size 48,
[    0.000000] OF: fdt: - 80000000 ,  7ee00000
[    0.000000] OF: fdt: - 100000000 ,  7f200000
[    0.000000] Found tegra_fbmem: 00140000@92cb0000
[    0.000000] earlycon: uart8250 at MMIO32 0x0000000070006000 (options '')
[    0.000000] bootconsole [uart8250] enabled
[    1.776962] csx307 7-003b: cs_imx307_board_setup: error during i2c read probe (-121)
[    1.784793] csx307 7-003b: board setup failed
[    1.789943] csx307 8-003b: cs_imx307_board_setup: error during i2c read probe (-121)
[    1.797731] csx307 8-003b: board setup failed
[   11.634026] tegra-i2c 7000c700.i2c: rx dma timeout txlen:28 rxlen:128
[   11.640477] tegra-i2c 7000c700.i2c: --- register dump for debugging ----
[   11.647178] tegra-i2c 7000c700.i2c: I2C_CNFG - 0x22c00
[   11.652315] tegra-i2c 7000c700.i2c: I2C_PACKET_TRANSFER_STATUS - 0x10001
[   11.659012] tegra-i2c 7000c700.i2c: I2C_FIFO_CONTROL - 0x1c
[   11.664582] tegra-i2c 7000c700.i2c: I2C_FIFO_STATUS - 0x800040
[   11.670412] tegra-i2c 7000c700.i2c: I2C_INT_MASK - 0x6c
[   11.675634] tegra-i2c 7000c700.i2c: I2C_INT_STATUS - 0x2
[   11.680971] tegra-i2c 7000c700.i2c: i2c transfer timed out addr: 0x50
[   11.688758] tegradc tegradc.1: dpd enable lookup fail:-19
[   13.360813] cgroup: cgroup2: unknown option "nsdelegate"
[   13.945998] usb 1-2: device not accepting address 2, error -71
[   14.130287] usb 1-2: Device not responding to setup address.
[   14.342234] usb 1-2: Device not responding to setup address.
[   14.554001] usb 1-2: device not accepting address 3, error -71
[   14.978642] random: crng init done
[   14.982049] random: 7 urandom warning(s) missed due to ratelimiting
[   15.688223] using random self ethernet address
[   15.692701] using random host ethernet address
[   16.482111] usb 1-2: device descriptor read/64, error -71
[   16.650177] using random self ethernet address
[   16.658052] using random host ethernet address
[   16.774127] usb 1-2: device descriptor read/64, error -71
[   17.250114] usb 1-2: device descriptor read/64, error -71
[   17.443567] ------------[ cut here ]------------
[   17.448195] WARNING: CPU: 3 PID: 2675 at /home/jetson/l4t-gcc/Linux_for_Tegra/source/public/kernel/kernel-4.9/kernel/jump_label.c:151 __static_key_slow_dec+0xa8/0xb0
[   17.463088] ---[ end trace 0720df7e0406dd1c ]---
[   17.542254] usb 1-2: device descriptor read/64, error -71
[   17.654087] usb usb1-port2: unable to enumerate USB device
[   17.922271] usb 1-3: Device not responding to setup address.
[   18.134283] usb 1-3: Device not responding to setup address.
[   18.345993] usb 1-3: device not accepting address 6, error -71
[   18.530415] usb 1-3: Device not responding to setup address.
[   18.742322] usb 1-3: Device not responding to setup address.
[   18.954029] usb 1-3: device not accepting address 7, error -71


Ubuntu 18.04.4 LTS localhost ttyS0

localhost login: jetson

Password: 
Last login: Fri Mar 26 13:11:32 UTC 2021 on ttyS0
Welcome to Ubuntu 18.04.4 LTS (GNU/Linux 4.9.140-tegra aarch64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

This system has been minimized by removing packages and content that are
not required on a system that users do not log into.

To restore this content, you can run the 'unminimize' command.

383 packages can be updated.
266 updates are security updates.

[   19.480532] edid invalid
[   20.883926] usb 1-3: device descriptor read/64, error -71
[   21.174296] usb 1-3: device descriptor read/64, error -71
[   21.646214] usb 1-3: device descriptor read/64, error -71
[   21.874712] tegra-i2c 7000c700.i2c: rx dma timeout txlen:28 rxlen:128
[   21.874754] tegra-i2c 7000c700.i2c: --- register dump for debugging ----
[   21.874795] tegra-i2c 7000c700.i2c: I2C_CNFG - 0x22c00
[   21.874828] tegra-i2c 7000c700.i2c: I2C_PACKET_TRANSFER_STATUS - 0x10001
[   21.874862] tegra-i2c 7000c700.i2c: I2C_FIFO_CONTROL - 0x1c
[   21.874894] tegra-i2c 7000c700.i2c: I2C_FIFO_STATUS - 0x800040
[   21.874928] tegra-i2c 7000c700.i2c: I2C_INT_MASK - 0x6c
[   21.874961] tegra-i2c 7000c700.i2c: I2C_INT_STATUS - 0x2
[   21.875046] tegra-i2c 7000c700.i2c: i2c transfer timed out addr: 0x50
[   21.935487] usb 1-3: device descriptor read/64, error -71
[   22.042980] usb usb1-port3: unable to enumerate USB device
[   32.114584] tegra-i2c 7000c700.i2c: rx dma timeout txlen:28 rxlen:128
[   32.114620] tegra-i2c 7000c700.i2c: --- register dump for debugging ----
[   32.114656] tegra-i2c 7000c700.i2c: I2C_CNFG - 0x22c00
[   32.114685] tegra-i2c 7000c700.i2c: I2C_PACKET_TRANSFER_STATUS - 0x10001
[   32.114714] tegra-i2c 7000c700.i2c: I2C_FIFO_CONTROL - 0x1c
[   32.114744] tegra-i2c 7000c700.i2c: I2C_FIFO_STATUS - 0x800040
[   32.114776] tegra-i2c 7000c700.i2c: I2C_INT_MASK - 0x6c
[   32.114806] tegra-i2c 7000c700.i2c: I2C_INT_STATUS - 0x2
[   32.114893] tegra-i2c 7000c700.i2c: i2c transfer timed out addr: 0x50
jetson@linux:~$ lsusb
Bus 002 Device 002: ID 0424:5744 Standard Microsystems Corp. 
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
jetson@linux:~$ lsusb -v

Bus 002 Device 002: ID 0424:5744 Standard Microsystems Corp. 
Couldn't open device, some information will be missing
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               3.20
  bDeviceClass            9 Hub
  bDeviceSubClass         0 Unused
  bDeviceProtocol         3 
  bMaxPacketSize0         9
  idVendor           0x0424 Standard Microsystems Corp.
  idProduct          0x5744 
  bcdDevice            2.21
  iManufacturer           2 
  iProduct                3 
  iSerial                 0 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           31
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 Unused
      bInterfaceProtocol      0 Full speed (or root) hub
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes           19
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Feedback
        wMaxPacketSize     0x0002  1x 2 bytes
        bInterval               8
        bMaxBurst               0

Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Couldn't open device, some information will be missing
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               3.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 Unused
  bDeviceProtocol         3 
  bMaxPacketSize0         9
  idVendor           0x1d6b Linux Foundation
  idProduct          0x0003 3.0 root hub
  bcdDevice            4.09
  iManufacturer           3 
  iProduct                2 
  iSerial                 1 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           31
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 Unused
      bInterfaceProtocol      0 Full speed (or root) hub
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0004  1x 4 bytes
        bInterval              12
        bMaxBurst               0

Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Couldn't open device, some information will be missing
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 Unused
  bDeviceProtocol         1 Single TT
  bMaxPacketSize0        64
  idVendor           0x1d6b Linux Foundation
  idProduct          0x0002 2.0 root hub
  bcdDevice            4.09
  iManufacturer           3 
  iProduct                2 
  iSerial                 1 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           25
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 Unused
      bInterfaceProtocol      0 Full speed (or root) hub
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0004  1x 4 bytes
        bInterval              12
jetson@linux:~$ [   42.354103] tegra-i2c 7000c700.i2c: rx dma timeout txlen:28 rxlen:128
[   42.360606] tegra-i2c 7000c700.i2c: --- register dump for debugging ----
[   42.367355] tegra-i2c 7000c700.i2c: I2C_CNFG - 0x22c00
[   42.372537] tegra-i2c 7000c700.i2c: I2C_PACKET_TRANSFER_STATUS - 0x10001
[   42.379359] tegra-i2c 7000c700.i2c: I2C_FIFO_CONTROL - 0x1c
[   42.384964] tegra-i2c 7000c700.i2c: I2C_FIFO_STATUS - 0x800040
[   42.390859] tegra-i2c 7000c700.i2c: I2C_INT_MASK - 0x6c
[   42.396106] tegra-i2c 7000c700.i2c: I2C_INT_STATUS - 0x2
[   42.401451] tegra-i2c 7000c700.i2c: i2c transfer timed out addr: 0x50

I noticed that jetson hangs for few seconds here:
[0015.946] set vdd_core voltage to 1125 mv
[0015.950] setting ‘vdd-core’ regulator to 1125000 micro volts
[0023.891] failed to get HID devices
[0023.911] Found secure-pmc; disable BPMP

Whereas when everything is working fine it goes smoothly:
[0009.899] set vdd_core voltage to 1125 mv
[0009.903] setting ‘vdd-core’ regulator to 1125000 micro volts
[0009.925] Found secure-pmc; disable BPMP

The other jetsons that fail to boot:

[0001.322] Starting CPU & Halting co-processor 

64NOTICE:  BL31: v1.3(release):de895fd9e
NOTICE:  BL31: Built : 12:23:48, Oct 16 2020
ERROR:   Error initializing runtime service trusty_fast
[0001.444] RamCode = 0
[0001.449] LPDDR4 Training: Read DT: Number of tables = 2
[0001.454] EMC Training (SRC-freq: 204000; DST-freq: 1600000)
[0001.467] EMC Training Successful
[0001.470] 408000 not found in DVFS table
[0001.476] RamCode = 0
[0001.480] DT Write: emc-table@204000 succeeded
[0001.485] DT Write: emc-table@1600000 succeeded
[0001.489] LPDDR4 Training: Write DT: Number of tables = 2
[0001.534] 
[0001.535] Debug Init done
[0001.538] Marked DTB cacheable
[0001.541] Bootloader DTB loaded at 0x83000000
[0001.545] Marked DTB cacheable
[0001.548] Kernel DTB loaded at 0x83100000
[0001.552] DeviceTree Init done
[0001.565] Pinmux applied successfully
[0001.569] gicd_base: 0x50041000
[0001.573] gicc_base: 0x50042000
[0001.576] Interrupts Init done
[0001.580] Using base:0x60005090 & irq:208 for tick-timer
[0001.585] Using base:0x60005098 for delay-timer
[0001.590] platform_init_timer: DONE
[0001.593] Timer(tick) Init done
[0001.597] osc freq = 38400 khz
[0001.601] 
[0001.602] Welcome to L4T Cboot
[0001.605] 
[0001.606] Cboot Version: 00.00.2018.01-t210-4686ce50
[0001.611] calling constructors
[0001.614] initializing heap
[0001.617] initializing threads
[0001.620] initializing timers
[0001.622] creating bootstrap completion thread
[0001.627] top of bootstrap2()
[0001.630] CPU: ARM Cortex A57
[0001.633] CPU: MIDR: 0x411FD071, MPIDR: 0x80000000
[0001.637] initializing platform
[0001.645] Manufacturer: MF = 0xc2, ID MSB = 0x25
[0001.649] ID LSB = 0x36, ID-CFI len = 194 bytes
[0001.654] Macronix QSPI chip present
[0001.657] SPI device register
[0001.660] init boot device
[0001.663] allocating memory for boot device(SPI)
[0001.667] registering boot device
[0001.677] sdmmc node status = okay
[0001.680] sdcard instance = 0
[0001.683] sdcard gpio handle 0x5b
[0001.686] sdcard gpio pin 0xc9
[0001.689] sdcard gpio flags 0x0
[0001.692] vmmc-supply 0x9e
[0001.695] Instance: 0
[0001.697] Allocating memory for context
[0001.701] enabling clock
[0001.703] sd card init
[0001.705] Check card present and stable
[0001.709] Send command 0
[0001.733] Send command 3
[0001.738] Set RCA for the card
[0001.741] Query card specific data by command 9
[0001.747] Parse CSD data
[0001.749] Send command 7
[0001.760] Calling sd device register
[0001.763] Init sdcard
[0001.765] Allocating memory for boot device
[0001.770] Registering user device
[0001.783] Enable APE clock
[0001.785] Un-powergate APE partition
[0001.789] of_register: registering tegra_udc to of_hal
[0001.794] of_register: registering inv20628-driver to of_hal
[0001.800] of_register: registering ads1015-driver to of_hal
[0001.805] of_register: registering lp8557-bl-driver to of_hal
[0001.811] of_register: registering bq2419x_charger to of_hal
[0001.817] of_register: registering bq27441_fuel_gauge to of_hal
[0001.828] gpio framework initialized
[0001.832] of_register: registering tca9539_gpio to of_hal
[0001.837] of_register: registering tca9539_gpio to of_hal
[0001.842] of_register: registering i2c_bus_driver to of_hal
[0001.848] of_register: registering i2c_bus_driver to of_hal
[0001.853] of_register: registering i2c_bus_driver to of_hal
[0001.859] pmic framework initialized
[0001.862] of_register: registering max77620_pmic to of_hal
[0001.868] regulator framework initialized
[0001.872] of_register: registering tps65132_bl_driver to of_hal
[0001.878] initializing target
[0001.884] gpio_driver_register: register 'tegra_gpio_driver' driver
[0001.892] board ID = D78, board SKU = 0
[0001.895] Skipping Z3!
[0001.900] fixed regulator driver initialized
[0001.918] initializing OF layer
[0001.921] NCK carveout not present
[0001.924] Skipping dts_overrides
[0001.928] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0001.945] I2C Bus Init done
[0001.948] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0001.958] I2C Bus Init done
[0001.961] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0001.971] I2C Bus Init done
[0001.974] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0001.984] I2C Bus Init done
[0001.987] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0001.997] I2C Bus Init done
[0002.000] of_children_init: Ops found for compatible string maxim,max77620
[0002.010] max77620_init using irq 118
[0002.015] register 'maxim,max77620' pmic
[0002.020] gpio_driver_register: register 'max77620-gpio' driver
[0002.026] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0002.036] I2C Bus Init done
[0002.040] NCK carveout not present
[0002.050] Find /i2c@7000c000's alias i2c0
[0002.054] get eeprom at 1-a0, size 256, type 0
[0002.062] Find /i2c@7000c500's alias i2c2
[0002.066] get eeprom at 3-a0, size 256, type 0
[0002.071] get eeprom at 3-ae, size 256, type 0
[0002.075] pm_ids_update: Updating 1,a0, size 256, type 0
[0002.080] I2C slave not started
[0002.083] I2C write failed
[0002.086] Writing offset failed
[0002.089] eeprom_init: EEPROM read failed
[0002.093] pm_ids_update: eeprom init failed
[0002.097] pm_ids_update: Updating 3,a0, size 256, type 0
[0002.127] pm_ids_update: The pm board id is 3448-0000-400
[0002.134] Adding plugin-manager/ids/3448-0000-400=/i2c@7000c500:module@0x50
[0002.142] pm_ids_update: pm id update successful
[0002.147] pm_ids_update: Updating 3,ae, size 256, type 0
[0002.152] I2C slave not started
[0002.155] I2C write failed
[0002.158] Writing offset failed
[0002.161] eeprom_init: EEPROM read failed
[0002.165] pm_ids_update: eeprom init failed
[0002.195] eeprom_get_mac: EEPROM invalid MAC address (all 0xff)
[0002.201] shim_eeprom_update_mac:267: Failed to update 0 MAC address in DTB
[0002.209] eeprom_get_mac: EEPROM invalid MAC address (all 0xff)
[0002.214] shim_eeprom_update_mac:267: Failed to update 1 MAC address in DTB
[0002.223] updating /chosen/nvidia,ethernet-mac node 00:04:4b:ec:0d:a7
[0002.229] Plugin Manager: Parse ODM data 0x00094000
[0002.241] shim_cmdline_install: /chosen/bootargs: earlycon=uart8250,mmio32,0x70006000 
[0002.256] Find /i2c@7000c000's alias i2c0
[0002.260] get eeprom at 1-a0, size 256, type 0
[0002.269] Find /i2c@7000c500's alias i2c2
[0002.273] get eeprom at 3-a0, size 256, type 0
[0002.277] get eeprom at 3-ae, size 256, type 0
[0002.282] pm_ids_update: Updating 1,a0, size 256, type 0
[0002.287] I2C slave not started
[0002.290] I2C write failed
[0002.293] Writing offset failed
[0002.296] eeprom_init: EEPROM read failed
[0002.300] pm_ids_update: eeprom init failed
[0002.304] pm_ids_update: Updating 3,a0, size 256, type 0
[0002.334] pm_ids_update: The pm board id is 3448-0000-400
[0002.340] Adding plugin-manager/ids/3448-0000-400=/i2c@7000c500:module@0x50
[0002.347] pm_ids_update: pm id update successful
[0002.352] pm_ids_update: Updating 3,ae, size 256, type 0
[0002.357] I2C slave not started
[0002.360] I2C write failed
[0002.363] Writing offset failed
[0002.366] eeprom_init: EEPROM read failed
[0002.370] pm_ids_update: eeprom init failed
[0002.399] Add serial number:1423119072463 as DT property
[0000.120] [L4T TegraBoot] (version 00.00.2018.01-l4t-c536fca9)
[0000.126] Processing in cold boot mode Bootloader 2
[0000.130] A02 Bootrom Patch rev = 1023
[0000.134] Power-up reason: pmc por
[0000.137] No Battery Present
[0000.140] pmic max77620 reset reason
[0000.143] pmic max77620 NVERC : 0x50
[0000.146] RamCode = 0
[0000.149] Platform has DDR4 type RAM
[0000.152] max77620 disabling SD1 Remote Sense
[0000.156] Setting DDR voltage to 1125mv
[0000.160] Serial Number of Pmic Max77663: 0x1e1df9
[0000.168] Entering ramdump check
[0000.171] Get RamDumpCarveOut = 0x0
[0000.174] RamDumpCarveOut=0x0,  RamDumperFlag=0xe59ff3f8
[0000.179] Last reboot was clean, booting normally!
[0000.184] Sdram initialization is successful 
[0000.188] SecureOs Carveout Base=0x00000000ff800000 Size=0x00800000
[0000.194] Lp0 Carveout Base=0x00000000ff780000 Size=0x00001000
[0000.200] BpmpFw Carveout Base=0x00000000ff700000 Size=0x00080000
[0000.206] GSC1 Carveout Base=0x00000000ff600000 Size=0x00100000
[0000.211] GSC2 Carveout Base=0x00000000ff500000 Size=0x00100000
[0000.217] GSC4 Carveout Base=0x00000000ff400000 Size=0x00100000
[0000.223] GSC5 Carveout Base=0x00000000ff300000 Size=0x00100000
[0000.229] GSC3 Carveout Base=0x000000017f300000 Size=0x00d00000
[0000.245] RamDump Carveout Base=0x00000000ff280000 Size=0x00080000
[0000.251] Platform-DebugCarveout: 0
[0000.254] Nck Carveout Base=0x00000000ff080000 Size=0x00200000
[0000.260] Non secure mode, and RB not enabled.
[0000.264] Read GPT from (4:0)
[0000.453] Csd NumOfBlocks=124735488
[0000.458] Set High speed to 1
[0000.461] Error mask set in wait for cmd complete with error 0x3 in HwSdmmcWaitForCommandComplete func at 278 line 
[0000.472] Command complete wait failed with error 0x3 Interrupt 0x18001
[0000.478] Number of retries left 4
[0000.487] Using GPT Primary to query partitions
[0000.492] Loading Tboot-CPU binary
[0000.517] Verifying TBC in OdmNonSecureSBK mode
[0000.527] Bootloader load address is 0xa0000000, entry address is 0xa0000258
[0000.534] Bootloader downloaded successfully.
[0000.538] Downloaded Tboot-CPU binary to 0xa0000258
[0000.543] MAX77620_GPIO5 configured
[0000.546] CPU power rail is up
[0000.549] Invalid Oscillator Freq
[0000.552] 

Assert on NvTbootGetOscFreqKhz:104: 0
[0000.557] 

Assert on NvTbootInitPllX:199: OscFreqKhz

My guess is that my very simplified power logic enables jetson before power had stabilized inside module and that’s why it hangs on ‘vdd-core’ setting.

My temporary solution that appears to be working:
I desoldered AND gate from my board, supplied power, and then manually enabe jetson by shorting POWER_EN wire to +5V. This ensures that power everywere on board is stable and then boot proces begins.

The key point of power sequence is to let module rails ON earlier than carrier board during power-on and OFF later than carrier during power-off. You can check your custom design to make sure about that.