Unable to flash: Error: probing the target board failed. and boot: Error in NvTbootLoadBinary: 0x14

Hello, could you please let me know if my Jetson TX1 is totally bricked, it won’t boot and I am unable to flash it.

Carrier board P2597: S/N 0321216047352
Jetson TX1: S/N: 0325115012411

Normal boot (serial console) always stops at:

...
[0000.571] NvTbootKernelDTB is not valid
[0000.575] Error in NvTbootLoadBinary: 0x14 !
[0000.579] Error is 14 

Device appears to enter recovery mode and 0955:7721 NVidia Corp. appears in lsusb output.

All flashing attempts fail with:

$ sudo ./flash.sh jetson-tx1 mmcblk0p1
###############################################################################
# L4T BSP Information:
# R28 (release), REVISION: 5.0, GCID: 27628514, BOARD: t210ref, EABI: aarch64, 
# DATE: Thu Jun 17 18:05:37 UTC 2021
###############################################################################
Error: probing the target board failed.
Make sure the target board is connected through 
micro-B USB port and is in recovery mode.

additional relevant info/logs:

$ pwd
/home/nvidia/nvidia/nvidia_sdk/JetPack_3.3.4_Linux_JETSON_TX1/Linux_for_Tegra


$ sudo ./flash.sh jetson-tx1 mmcblk0p1
###############################################################################
# L4T BSP Information:
# R28 (release), REVISION: 5.0, GCID: 27628514, BOARD: t210ref, EABI: aarch64, 
# DATE: Thu Jun 17 18:05:37 UTC 2021
###############################################################################
Error: probing the target board failed.
Make sure the target board is connected through 
micro-B USB port and is in recovery mode.


$ lsusb -t
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/12p, 480M
|__ Port 4: Dev 5, If 0, Class=Vendor Specific Class, Driver=, 480M
|__ Port 8: Dev 2, If 0, Class=Wireless, Driver=btusb, 12M
|__ Port 8: Dev 2, If 1, Class=Wireless, Driver=btusb, 12M


$ lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 8087:0a2b Intel Corp. 
Bus 001 Device 005: ID 0955:7721 NVidia Corp. 
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub



$ dmesg -w
...
[ 873.644588] usb 1-4: new high-speed USB device number 6 using xhci_hcd
[ 873.771810] usb 1-4: New USB device found, idVendor=0955, idProduct=7721, bcdDevice= 1.02
[ 873.771824] usb 1-4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 873.771830] usb 1-4: Product: APX
[ 873.771835] usb 1-4: Manufacturer: NVIDIA Corp.





[0000.171] [TegraBoot] (version 00.00.2014.50-mobile-a2950310) [5/45076]
[0000.177] Processing in cold boot mode Bootloader 2
[0000.182] A02 Bootrom Patch rev = 63 
[0000.185] Power-up reason: reset button 
[0000.189] No Battery Present
[0000.191] RamCode = 0
[0000.194] Platform has Ddr4 type ram 
[0000.197] max77620 disabling SD1 Remote Sense
[0000.201] Setting Ddr voltage to 1125mv
[0000.205] Serial Number of Pmic Max77663: 0x118a1 
[0000.213] Entering ramdump check
[0000.216] Get RamDumpCarveOut = 0xff280000
[0000.220] RamDumpCarveOut=0xff280000, RamDumperFlag=0x0
[0000.225] Last reboot was clean, booting normally!
[0000.229] Sdram initialization is successful 
[0000.233] SecureOs Carveout Base=0x00000000ff800000 Size=0x00800000
[0000.240] Lp0 Carveout Base=0x00000000ff780000 Size=0x00001000
[0000.245] BpmpFw Carveout Base=0x00000000ff700000 Size=0x00080000
[0000.251] GSC1 Carveout Base=0x00000000ff600000 Size=0x00100000
[0000.257] GSC2 Carveout Base=0x00000000ff500000 Size=0x00100000
[0000.263] GSC4 Carveout Base=0x00000000ff400000 Size=0x00100000
[0000.269] GSC5 Carveout Base=0x00000000ff300000 Size=0x00100000
[0000.274] GSC3 Carveout Base=0x000000017f300000 Size=0x00d00000
[0000.291] RamDump Carveout Base=0x00000000ff280000 Size=0x00080000
[0000.297] Platform-DebugCarveout: 0
[0000.300] Nck Carveout Base=0x00000000ff080000 Size=0x00200000
[0000.306] Non secure mode, and RB not enabled.
[0000.359] Using GPT Primary to query partitions 
[0000.364] Loading Tboot-CPU binary
[0000.459] Verifying TBC in OdmNonSecureSBK mode
[0000.468] Bootloader load address is 0xa0000000, entry address is 0xa0000258
[0000.475] Bootloader downloaded successfully.
[0000.479] Downloaded Tboot-CPU binary to 0xa0000258
[0000.484] MAX77620_GPIO1 Configured.
[0000.488] MAX77620_GPIO5 Configured.
[0000.491] CPU power rail is up
[0000.494] CPU clock enabled
[0000.498] Performing RAM repair
[0000.501] Updating A64 Warmreset Address to 0xa00002e9
[0000.506] Loading NvTbootBootloaderDTB
[0000.532] NvTbootBootloaderDTB is not valid
[0000.536] Error in NvTbootLoadBinary: 0x14 !
[0000.540] Bootloader DTB Load Address: 0x83000000 
[0000.545] Loading NvTbootKernelDTB
[0000.571] NvTbootKernelDTB is not valid
[0000.575] Error in NvTbootLoadBinary: 0x14 !
[0000.579] Error is 14

You’ll need to post which release you are flashing, if the TX1 is a dev kit, and the exact command used (if it is command line flash…or mention if using SDKM GUI). Also, any use of VM would be a problem.

I don’t know what “dev kit” is specially, but the model of the carrier board is in the OP as well as serials and lsusb output. The module on the carrier board is TX1.

As for the versions, the logs I posted are flashing 3.3.4 via CLI, though same result if using GUI. I’ve also tried all the later available versions with exactly the same result.

I am using an Intel NUC, not running a VM.

I suspect the flashing utility is not getting the data is expects when it tries to query the USB device. Possibly because the boot loader is corrupted, but that’s conjecture.

Are there any more specific diagnostics specs I can run to confirm there is a chance of revival, or is this kit ready for the dumpster?

Cheers!

  1. Devkit is the term we use to say NVIDIA developer kit. If this carrier board is from other vendor and some custom board, then it is not a devkit.

  2. Just need to confirm, do you have any experience in flashing any jetson platform or not? If this is your first time using this host to flash a jetson board, then we need to clarify whether there is anything wrong in your environment.

Carrier board P2597: S/N 0321216047352

This is made by Nvidia as far as I can tell. It certainly has nvidia logo all over it.

I’ve included some logs in the original post. Do these not contain enough information?

I don’t think it’s relevant whether or not I personally have experience in flashing jetsons. I have experience in flashing other platforms and I have followed the nvidia documentation on flashing a tx1 as far as I can tell. This seems like a distraction tbh.

Both GUI and CLI methods trip with exactly the same fault, which seems to be around the flashing tool unable to determine the board specifics. Details in the logs.

I don’t flee this is a communication issue with USB ports/cables, between the board and host since I get the same consistent error. I have tired a few different USB cables and ports though.

If there is any other tools I can run and provide output from, please let me know…

I am sorry for the distraction because this is our first time doing communication. So I have to understand how you are familiar with the platofrm.

We have lots of other users who don’t know how to flash board and turns out they are using VM or their environment is not proper.

Do you have any other TX1 board that can do A/B test? Entering recovery mode is basically hardware triggered, if host cannot detect recovery mode, then it might be hardware problem.

Unfortunately this is the only board I have access to, so assuming it is a hardware problem, do I have any other option other than throwing it away?

  1. From software side suggestion, I would say you can also try rel-32 based release and see if they can flash the board. I don’t think they may work, but still able to try before abandon this board.

  2. I can only suggest you can file RMA request if this board is still in warranty.
    https://developer.nvidia.com/embedded/faq

What you describe does not sound like hardware failure. There could be hardware failure, but it is highly unlikely.

About this:

That version is quite old. Is there a reason for picking this, e.g., mandatory use of a particular software? If not, then you should try the most recent release. Versions are listed here:
https://developer.nvidia.com/embedded/jetpack-archive

It is too early to say, but I have heard of a number Intel NUCs which have USB issues. This is just speculation, but it has happened enough I am wondering if you have another computer you can test flash from?

Unfortunately I’ve only got NUCs as far as x86 arch. goes.

I gave 3.3.4 as en example, but I’ve also tried 4.5.1, 4.6 and 4.6.1 - all produce the same error.

Would running the Docker image with Docker for Desktop on macOS (x86-64) work? This is the only other bare metal x86 platform I have.

Not sure if relevant, but from this:

ubuntu@ubuntu:~$ lsusb -d 0955:7721
Bus 001 Device 003: ID 0955:7721 NVidia Corp. 

ubuntu@ubuntu:~$ cat /dev/bus/usb/001/003 | head -c 1024 | hexdump
0000000 0112 0110 0000 4000 0955 7721 0102 0201
0000010 0100 0209 0020 0101 c000 0910 0004 0200
0000020 ffff 00ff 0507 0281 0200 0700 0105 0002
0000030 0002                                   
0000032

From the other post I linked, it seems like the response from TX1 in recovery mode is incomplete.

For posterity, I’ve managed to get a little further.

  1. install ubuntu 18.04 VM on VirtualBox v.Latest on macOS
  2. install sdk
  3. auto-map nvidia USB device to VM using USB port filter
  4. pushd ./nvidia/nvidia_sdk/JetPack_4.6.1_Linux_JETSON_TX1_TARGETS/Linux_for_Tegra
  5. sudo ./flash.sh jetson-tx1 mmcblk0p1

The full log (serial console):

[0760.695] Enabled early print                                                                      
[0760.698] [L4T TegraBoot] (version J)                                                              
[0760.702] Processing in recovery mode                                                              
[0760.705] A02 Bootrom Patch rev = 63                                                               
[0760.709] Power-up reason: software reset                                                          
[0760.712] Established communication link with host                                                 
[0761.837] Odmdata from BCT: 0x000a4000                                                             
[0761.841] DebugPort= 0x3                                                                           
[0761.843] SkipQspiOnly= 0x0                                                                        
[0761.846] BoardId read from EEPROM/NCT: 2180                                                       
[0761.874] Turned on fan for Jetson board                                                           
[0761.878] Key release before reqd. time for fastboot                                               
[0761.883] No Battery Present                                                                       
[0761.885] RamCode = 0                                                                              
[0761.888] Platform has Ddr4 type ram                                                               
[0761.891] max77620 disabling SD1 Remote Sense                                                      
[0761.895] Setting Ddr voltage to 1125mv  
[0761.899] Serial Number of Pmic Max77663: 0x118a1                                                  
[0761.907] Entering ramdump check          
[0761.910] Get RamDumpCarveOut = 0x0                                                                
[0761.913] RamDumpCarveOut=0x0,  RamDumperFlag=0xe59ff3f8                                           
[0761.919] Last reboot was clean, booting normally!                                                 
[0761.923] Sdram initialization is successful                                                       
[0761.927] SecureOs Carveout Base=0x00000000ff800000 Size=0x00800000                                
[0761.934] Lp0 Carveout Base=0x00000000ff780000 Size=0x00001000                                     
[0761.939] BpmpFw Carveout Base=0x00000000ff700000 Size=0x00080000                                  
[0761.945] GSC1 Carveout Base=0x00000000ff600000 Size=0x00100000
[0761.951] GSC2 Carveout Base=0x00000000ff500000 Size=0x00100000
[0761.957] GSC4 Carveout Base=0x00000000ff400000 Size=0x00100000
[0761.963] GSC5 Carveout Base=0x00000000ff300000 Size=0x00100000
[0761.969] GSC3 Carveout Base=0x000000017f300000 Size=0x00d00000
[0761.985] RamDump Carveout Base=0x00000000ff280000 Size=0x00080000
[0761.991] Platform-DebugCarveout: 0
[0761.994] Downloaded BCT successfully
[0762.416] Downloaded Bootloader successfully
[0762.713] Downloaded rp1 successfully
[0762.964] MAX77620_GPIO1 Configured.
[0762.968] MAX77620_GPIO5 Configured.
[0762.971] CPU power rail is up
[0762.974] CPU clock enabled
[0762.978] Performing RAM repair
[0762.981] Updating A64 Warmreset Address to 0x92c002e9
[0762.987] Enable APE clock/reset
[0762.990] Error in NvTbootGetTOSBinaryLength: 0x11 !                                      [107/670]
[0762.995] Loading Secure OS image failed. 
[0762.999] Set NvDecSticky Bits
[0763.002] GSC2 address ff53fffc value c0edbbcc
[0763.009] GSC MC Settings done
[0763.013] Next binary entry address: 0x92c00258 
[0763.017] BoardId: 2180
[0763.020] Rail not supported
[0763.023] Starting CPU & Halting co-processor 

[0781.346] 
[0781.347] Debug Init done
[0781.350] Marked DTB cacheable
[0781.353] Bootloader DTB loaded at 0x83000400
[0781.358] DeviceTree Init done
[0781.372] Pinmux applied successfully
[0781.377] gicd_base: 0x50041000
[0781.381] gicc_base: 0x50042000
[0781.384] Interrupts Init done
[0781.388] Using base:0x60005090 & irq:208 for tick-timer
[0781.393] Using base:0x60005098 for delay-timer
[0781.398] platform_init_timer: DONE
[0781.401] Timer(tick) Init done
[0781.406] osc freq = 38400 khz
[0781.410] 
[0781.411] Welcome to L4T Cboot
[0781.414] 
[0781.415] Cboot Version: 00.00.2018.01-t210-a2f2e4b8
[0781.420] calling constructors
[0781.423] initializing heap
[0781.426] initializing threads
[0781.429] initializing timers
[0781.432] creating bootstrap completion thread
[0781.436] top of bootstrap2()
[0781.439] CPU: ARM Cortex A57
[0781.442] CPU: MIDR: 0x411FD071, MPIDR: 0x80000000
[0781.447] initializing platform
[0781.515] Config for emmc ddr50 mode completed
[0781.519] sdmmc bdev is already initialized
[0781.524] of_register: registering tegra_udc to of_hal
[0781.529] of_register: registering inv20628-driver to of_hal
[0781.534] of_register: registering ads1015-driver to of_hal
[0781.540] of_register: registering lp8557-bl-driver to of_hal
[0781.546] of_register: registering bq2419x_charger to of_hal
[0781.551] of_register: registering bq27441_fuel_gauge to of_hal                            [63/670]
[0781.565] gpio framework initialized
[0781.569] of_register: registering tca9539_gpio to of_hal
[0781.574] of_register: registering tca9539_gpio to of_hal
[0781.580] of_register: registering i2c_bus_driver to of_hal
[0781.585] of_register: registering i2c_bus_driver to of_hal
[0781.591] of_register: registering i2c_bus_driver to of_hal
[0781.596] pmic framework initialized
[0781.600] of_register: registering max77620_pmic to of_hal
[0781.605] regulator framework initialized
[0781.609] of_register: registering tps65132_bl_driver to of_hal
[0781.615] initializing target
[0781.622] gpio_driver_register: register 'tegra_gpio_driver' driver
[0781.630] board ID = 884, board SKU = 3E8
[0781.638] fixed regulator driver initialized
[0781.682] initializing OF layer
[0781.686] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0781.709] I2C Bus Init done
[0781.711] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0781.723] I2C Bus Init done
[0781.726] of_children_init: Ops found for compatible string ti,tca9539
[0781.736] tca9539_init: i2c bus: 2, slave addr: 0xe8
[0781.744] gpio_driver_register: register 'tca9539_gpio_driver' driver
[0781.750] of_children_init: Ops found for compatible string ti,tca9539
[0781.761] tca9539_init: i2c bus: 2, slave addr: 0xee
[0781.768] gpio_driver_register: register 'tca9539_gpio_driver' driver
[0781.775] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0781.786] I2C Bus Init done
[0781.789] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0781.801] I2C Bus Init done
[0781.803] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0781.815] I2C Bus Init done
[0781.817] of_children_init: Ops found for compatible string maxim,max77620
[0781.829] max77620_init using irq 118
[0781.834] register 'maxim,max77620' pmic
[0781.838] gpio_driver_register: register 'max77620-gpio' driver
[0781.844] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0781.856] I2C Bus Init done
[0781.860] Applying platform configs
[0781.869] platform-init is not present. Skipping
[0781.874] calling apps_init()
[0781.898] Found 22 GPT partitions in "sdmmc3_user"
[0781.903] Proceeding to flashing Server
[0781.906] usbdcd_reinit Initialize driver to use already enumerated device
[0781.913] nv3p_priv_usbf_open USB configuration success                                    [19/670]
[0782.440] Writing GPT partition
[0782.490] Erasing Boot device
[0782.493] device_write_gpt: Erasing boot device sdmmc3_boot
[0782.527] Erasing Storage Device
[0782.991] Sanitizing Storage Device
[0783.119] Writing protective mbr
[0783.128] Writing primary GPT
[0783.143] Writing secondary GPT
[0783.173] Found 22 GPT partitions in "sdmmc3_user"
[0783.260] Writing PT partition
[0783.362] partition PT write successful.
[0783.437] Writing PT-1 partition
[0783.567] partition PT-1 write successful.
[0783.637] Writing NVC partition
[0783.812] partition NVC write successful.
[0783.902] Writing TBC partition
[0784.028] partition TBC write successful.
[0784.108] Writing RP1 partition
[0784.354] partition RP1 write successful.
[0784.434] Writing EBT partition
[0784.713] partition EBT write successful.
[0784.809] Writing WB0 partition
[0784.966] partition WB0 write successful.
[0785.064] Writing BPF partition
[0785.216] partition BPF write successful.
[0785.297] Writing NVC-1 partition
[0785.479] partition NVC-1 write successful.
[0785.566] Writing TBC-1 partition
[0785.707] partition TBC-1 write successful.
[0785.795] Writing RP1-1 partition
[0786.078] partition RP1-1 write successful.
[0786.201] Writing EBT-1 partition
[0786.474] partition EBT-1 write successful.
[0786.573] Writing WB0-1 partition
[0786.722] partition WB0-1 write successful.
[0786.810] Writing BPF-1 partition
[0786.952] partition BPF-1 write successful.
[0787.051] Writing VER_b partition
[0787.177] partition VER_b write successful.
[0787.279] Writing VER partition
[0787.386] partition VER write successful.
[0787.468] Writing APP partition
[0871.906] portspeed = 2
[0888.572] nv3p_usbf_receive transfer timedout
[0888.577] ERROR: USB RECEIVE FAILED
[0903.496] udc_priv_add_req_epOnly one req at a time for ep
[0953.493] nv3p_usbf_send transfer timedout
[0953.497] ERROR: USB SEND FAILED
[0953.500] Data recieve from host failure with error 0xfffffff3
[0953.506] Write command failed for APP partition
[0953.511] udc_priv_add_req_epOnly one req at a time for ep
[1003.513] nv3p_usbf_send transfer timedout
[1003.517] ERROR: USB SEND FAILED
[1003.520] ERROR: Error in USB Txfr
[1003.523] ERROR: USB RECEIVE FAILED
[1003.527] udc_priv_add_req_epOnly one req at a time for ep
[1053.523] nv3p_usbf_send transfer timedout
[1053.527] ERROR: USB SEND FAILED
[1053.530] Command receive is failed with error 0xffffffff
[1053.536] hanging in flashing server due to error: Command receive is failed

So looks like I am getting USB timeouts, possibly VBox/VM related. I’ll keep hacking at it…

… this was with v4.6.1, I’ve then tried flashing with 3.3.4 and got through the flashing process successfully.

Hi,

VM does not support well in most jetpack. It will have a unstable result in usb detection. If the board can get flashed now, then I think the board itself is still ok.

Some parts did flash correctly, so I have to agree that the Jetson itself is probably working correctly. To expand on VMs and why they are not supported, they usually do not pass through USB correctly for the case of USB which disconnects and reconnects during flash. It is common on many VMs that despite starting out ok with USB, that during the disconnect and reconnect the VM does not correctly pick up the USB again. A VM can work, but you would need to insure that USB always passes through even when there is a disconnect and reconnect.

In the cases I’ve heard of an Intel NUC failing this too was related to USB. I have no idea what might be different on a NUC.

I would avoid 3.3.4 since it is extremely out of date, but if this works, then you know the Jetson itself is working correctly.

Yes, success under VMs seems to be hit and miss. I tried on a faster macOS laptop and it worked a lot more reliably. It would be great if the ecosystem was more flexible with VMs in mind, but I understand there are HW limitations. In any case, the board seems to work - thank you for your help!

The above is up to the VM manufacturer. The Jetson software has no control over how USB might fail on the host PC. People tend to think a VM is an actual computer, but it isn’t…it is a computer emulation, and if the emulation is flawed, then there is nothing the Jetson software can do about it. This is in fact one of the points of a VM: One could run unsafe software in it and the software would have no ability to change anything outside of the VM. This is the very definition of “the Jetson has no control over it”.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.