How to kick watchdog enabled by enable_watchdog fuse?

When enabling secure boot on our AGX Xavier, we also set bit 5 (0x20) in SwReserved to set “enable_watchdog” so that we would have a watchdog running out of boot. This fuse is documented in several locations including:

  1. Jetson AGX Xavier Series Fuse Programming Application Note
  2. README_secureboot.txt in Jetson Platform Fuse Burning and Secure Boot Documentation and Tools
  3. Mentioned by NVIDIA in another thread, showing how to set enable_watchdog with odmfuse.sh. That thread also mentions that this should enable one of the two watchdogs mentioned here.

Unfortunately, neither the PMIC watchdog or the Tegra watchdog (CCPLEX watchdog) appear to be enabled after checking from Linux, and the board is now resetting every 15 seconds! Even worse, it appears that the recovery bootloader which is used to load the eMMC over USB (nvtboot_recovery_cpu_t194.bin) does not kick this watchdog either, so flash.sh fails if the loading takes more than 15 seconds.

Can you please provide information on which watchdog is causing this reset, if there is any way of kicking it or stopping it, and if there is a version of nvtboot_recovery_cpu_t194.bin which is capable of kicking or stopping this watchdog? It seems bad that this was mentioned in documentation if it isn’t supported by software, and my board is currently unusable due to this issue. Thanks.

Edit: I see that the PMC reset reason is TEGRA_BPMP_WATCHDOG, which doesn’t appear to be documented anywhere. I see references to the WDT_BPMP in the public cboot sources, but accessing those WDT registers triggers a Security Violation from cboot and Linux…

Hi,
The watchdog listed in developer guide is enabled by default and you don’t need to program enable_watchdog fuse.

Since proramming fuse is not reversible so we may not be able to recover the board back. Could you check if you can flash image by following the steps:
Unable to burn fuses (dev kit) / no more output (serial/hdmi) / bricked? - #89 by DaneLLL

Hi,
From what I can find, it doesn’t look like the PMIC Watchdog and/or Tegra Watchdog are enabled by default out of boot like you indicate here. I checked early in the CPU bootloader, and neither was enabled. It looks like cboot will enable the Tegra Watchdog from platform_early_init(), but nothing appears to be enabled before that point in the boot process (including BootROM/MB1/MB2) unless this fuse is set? It also looks like cboot will disable the Tegra Watchdog before booting the kernel (from platform_uninit), so there definitely are large parts of the boot process which are not being protected by a watchdog.

As the answer to the question you asked, I can successfully flash and boot the secured board, just as long as the flashing takes less than 15 seconds (before the BPMP watchdog expires).

It would seem that the easiest solution to allow software to work with this fuse set would to be to simply have the BPMP-FW disable the BPMP WDT that is enabled by this fuse after initialization completes. If the watchdog wasn’t enabled by the fuse, then it does no harm to disable it again too. Alternatively, it could be disabled by BPMP-FW after a mailbox command from the CPU-BL was received, guaranteeing that the CPU was up (and the CPU could have already started the Tegra Watchdog).

Is there a possibility of getting an official updated bpmp_t194.bin that just disables this BPMP watchdog? Thanks.

hello user71830,

could you please share your commands for burning the fuse, also, the command you used to flash the target.
you may also share UART console logs for reference, there’s port J501 you may used to gather bootloader logs,
thanks

Hi Jerry,
Here are the commands I used for burning the fuses. The most import part is the “-r 0x28” option to odmfuse.sh since that includes the enable_watchdog bit.

I first had to edit the odmfuse.sh script as there was a bug in it and it wasn’t generating the proper fuse values in the XML in offline / --noburn mode. On line 1071, there is a line that states bootauth="None" and I had to replace that with bootauth="NS" .

I then ran the following commands to get odmfuse_pkc.xml:

BOARDID=2888 BOARDSKU=0004 FAB=400 BOARDREV=M.0 ./odmfuse.sh --noburn -i 0x19 --auth SBKPKC -p -k pkc_priv.pem --KEK0 kek0 --KEK1 kek1 --KEK2 kek2 -S sbk --disable-jtag -r 0x28 jetson-agx-xavier-devkit
tar jxOf fuseblob.tbz2 bootloader/odmfuse_pkc.xml > odmfuse_pkc.xml

I noticed that odmfuse_pkc.xml was missing any programming for the FUSE_BOOT_DEVICE_INFO fuse as well, and the “Jetson AGX Xavier Series Fuse Specification” states “These fuses should be burned to 0x3”, so I added those in by adding this line to the XML:

<fuse name="BootDevInfo" size="4" value="0x3" />

Finally, I used the burnfuses.sh script in OE4T to actually burn the fuses. That essentially executes the OE4T helper script ./tegra194-flash-helper.sh -c "burnfuses odmfuse_pkc.xml" which in turn calls tegraflash.py with --cmd "burnfuses odmfuse_pkc.xml".

Fusing went just fine – I had no issues there. But you have my commands now if you wanted to reproduce what I did.

I subsequently flashed the board with the OE4T doflash.sh wrapper script. That just calls tegraflash.py with pre-signed/encrypted images and the cmd argument is “secureflash;reboot”. This is where I started to run into issues due to this BPMP watchdog. The flashing would fail while writing the APP partition always 15 seconds into boot, with this being the last that printed to the console:

...
[0058.658] I> Entering 3p server
[0058.658] I> USB configuration success
[0058.813] I> Populate storage info
[0058.870] I> Erasing device 0: 3
[0060.092] I> Writing device 0: 3.
[0060.200] I> Found 17 partitions in SDMMC_BOOT (instance 3)
[0060.201] I> Erasing device 1: 3
[0060.711] I> Writing device 1: 3.
[0060.723] I> Writing device 1: 3.
[0060.747] I> Found 44 partitions in SDMMC_USER (instance 3)
[0060.748] I> Writing device 1: 3.
[0060.768] I> Found 44 partitions in SDMMC_USER (instance 3)
[0060.771] I> Writing mb1 partition.
[0060.813] I> Writing mb1_b partition.
[0060.847] I> Writing spe-fw partition.
[0060.872] I> Writing spe-fw_b partition.
[0060.907] I> Writing mb2 partition.
[0060.937] I> Writing mb2_b partition.
[0060.967] I> Writing mts-preboot partition.
[0060.987] I> Writing mts-preboot_b partition.
[0061.009] I> Writing SMD partition.
[0061.023] I> Writing SMD_b partition.
[0061.037] I> Writing VER_b partition.
[0061.051] I> Writing VER partition.
[0061.066] I> Writing device 1: 3.
[0061.074] I> Writing APP partition.

After I made the APP partition image much smaller though, so that flashing completed within 15 seconds, I could subsequently boot the Linux kernel into the initramfs (but I was missing a root filesystem of course). I can see all of the prints that my images are being properly authenticated and decrypted based on the secure boot fuses as well, so fusing is working. However, the board only stays up for 15 seconds after power up now before this BPMP watchdog expires. I can also see that the PMC reset source printed by Linux was “TEGRA_BPMP_WATCHDOG”.

The problem is that there appears to be no way of kicking or disabling this BPMP_WATCHDOG from the CPU.

hello user71830,

I’ll arrange resources for checking this.
just double confirm you’re program FUSE_RESERVED_SW[23:0] as following, right?

Bit [3] If set, will boot using the device selected in Bits 2:0
Bit [5] Enable Watchdog

Yes, that is correct, bits 3 and 5 were set, making FUSE_RESERVED_SW=0x28. Thanks.

hello user71830,

in order to test watchdog reboot for 15-sec,
could you please modify $OUT/Linux_for_Tegra/bootloader/t186ref/BCT/tegra194-mb1-soft-fuses-l4t.cfg,
please adding DisableWdtGlobally = 1; then re-flash the target for confirmation.
thanks

hello user71830,

may I know your test results, are you able to workaround watchdog rebooting on AGX Xavier platforms?
you may see-also this thread, Topic 198557, which also fuse burning Xavier NX with -r 0x28 and had the same failure.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.