AGX Orin shutdown error loop
Test environment (DevKit):
- AGX Orin R39.2.1 devkit
- NVMe SSD Boot
- Connected to display through DP->HDMI adapter
- Monitor: Iiyama ProLite XB3294UHSCP
Test environment (Custom board):
- AGX Orin R39.2.1 custom board
- NVMe SSD Boot and SD boot
- Connected to display through HDMI
- Monitor: Iiyama ProLite XB3294UHSCP
Issue
Sometimes when we send a shutdown signal, during the shutdown sequence, the cpu throws an error, enters an infinite loop and the shutdown never completes. The only recovery seems to be cutting off the power.
It’s intermitent but the reproduction rate is quite high; for each test circumstances I tried rebooting at least 50 times and the result seems to be about 40% reproduction rate.
Sometimes the cpu just throws the error, but more often than not, it loops forever. The looping faults seems to have very consistent register address: first the cpu throws an error at 0x155c0124 and then loops with errors at 0x1380c410. The error looks like this:
[ 3594.534111] CPU:0, Error: cbb-fabric@0x13a00000, irq=33
[ 3594.539542] **************************************
[ 3594.544496] CPU:0, Error:cbb-fabric, Errmon:64
[ 3594.549091] Error Code : TIMEOUT_ERR
[ 3594.553154]
[ 3594.554695] Error Code : TIMEOUT_ERR
[ 3594.558745] MASTER_ID : TSECA_NONSECURE
[ 3594.563063] Address : 0x155c0124
[ 3594.566756] Cache : 0x3 -- Bufferable Modifiable
[ 3594.572059] Protection : 0x2 -- Unprivileged, Non-Secure, Data Access
[ 3594.579070] Access_Type : Read
[ 3594.582584] Access_ID : 0x0
[ 3594.582588] Fabric_Id : 0x0
[ 3594.589079] Fabric : cbb-fabric
[ 3594.592690] Slave_Id : 0x3
[ 3594.595848] Burst_length : 0x0
[ 3594.599360] Burst_type : 0x1
[ 3594.602698] Beat_size : 0x2
[ 3594.605945] VQC : 0x0
[ 3594.608751] GRPSEC : 0x7f
[ 3594.611824] FALCONSEC : 0x0
[ 3594.615071] HOST1X_SLV_TIMEOUT_STATUS : 0x1
[ 3594.619660] **************************************
[ 3594.624730] CPU:0, Error: cbb-fabric@0x13a00000, irq=33
[ 3594.630125] **************************************
[ 3594.635071] CPU:0, Error:cbb-fabric, Errmon:64
[ 3594.639664] Error Code : TIMEOUT_ERR
[ 3594.643711] Overflow : Multiple TIMEOUT_ERR
[ 3594.648396]
[ 3594.649933] Error Code : TIMEOUT_ERR
[ 3594.653987] MASTER_ID : TSECA_HEAVYSECURE
[ 3594.658485] Address : 0x1380c410
[ 3594.662178] Cache : 0x3 -- Bufferable Modifiable
[ 3594.667480] Protection : 0x2 -- Unprivileged, Non-Secure, Data Access
[ 3594.674488] Access_Type : Read
[ 3594.678000] Access_ID : 0x0
[ 3594.678003] Fabric_Id : 0x0
[ 3594.684489] Fabric : cbb-fabric
[ 3594.688095] Slave_Id : 0x37
[ 3594.691339] Burst_length : 0x0
[ 3594.694851] Burst_type : 0x1
[ 3594.698191] Beat_size : 0x2
[ 3594.701436] VQC : 0x0
[ 3594.704229] GRPSEC : 0x7b
[ 3594.707302] FALCONSEC : 0x2
[ 3594.710551] AXI2APB_5_BLOCK_TMO_STATUS : 0x1
[ 3594.715235] AXI2APB_5_BLOCK0_TMO : 0x8
[ 3594.719383] **************************************
[ 3594.724447] CPU:0, Error: cbb-fabric@0x13a00000, irq=33
[ 3594.729835] **************************************
[ 3594.734780] CPU:0, Error:cbb-fabric, Errmon:64
[ 3594.739372] Error Code : TIMEOUT_ERR
[ 3594.743426] Overflow : Multiple TIMEOUT_ERR
[ 3594.748113]
[ 3594.749650] Error Code : TIMEOUT_ERR
[ 3594.753695] MASTER_ID : TSECA_HEAVYSECURE
[ 3594.758197] Address : 0x1380c410
[ 3594.761895] Cache : 0x3 -- Bufferable Modifiable
[ 3594.767201] Protection : 0x2 -- Unprivileged, Non-Secure, Data Access
[ 3594.774209] Access_Type : Read
[ 3594.777724] Access_ID : 0x0
[ 3594.777727] Fabric_Id : 0x0
[ 3594.784212] Fabric : cbb-fabric
[ 3594.787822] Slave_Id : 0x37
[ 3594.791069] Burst_length : 0x0
[ 3594.794584] Burst_type : 0x1
[ 3594.797922] Beat_size : 0x2
[ 3594.801168] VQC : 0x0
[ 3594.803972] GRPSEC : 0x7b
[ 3594.807042] FALCONSEC : 0x2
[ 3594.810288] AXI2APB_5_BLOCK_TMO_STATUS : 0x1
[ 3594.814970] AXI2APB_5_BLOCK0_TMO : 0x8
[ 3594.819115] **************************************
... (The above error loops infinitely)
What we tried
- The error does not happen if there is not HDMI cable connected
- If we kill gdm before shutting down/rebooting, the error does not happen
- On the devkit, if plugged through DP cable directly, the error does not happen
- The error does not happen if we shutdown/reboot through the GUI
Previously we had an error related to HDMI HPD that seemed similar to this, but the HPD error should have been fixed in JP 6.X I believe, and the workarounds from then didn’t solve the issue right now.
So we tested a few other workarounds, mostly trying to figure out some timing during the shutdown sequence that the error happens and what to do at such moment. We can workaround this error by forcibly killing gdm earlier during shutdown or adding a sleep of about 3s. For example:
# [rootfs/usr/lib/systemd/system/jetson-display-shutdown-guard.service]
[Unit]
Description=Jetson display shutdown guard
After=nv_nvsciipc_init.service
Before=display-manager.service
[Service]
Type=oneshot
ExecStart=/usr/bin/true
RemainAfterExit=yes
ExecStop=-/usr/bin/logger -t jetson-shutdown "display-manager stopped; waiting 5 seconds"
ExecStop=/usr/bin/sleep 5
TimeoutStopSec=10
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now jetson-display-shutdown-guard.service
by enabling this service we can workaround the problem 100% of the time.
Our investigation suggests that asynchronous access from TSEC may be triggering this error.
Could you please investigate the root cause and provide a fix so that the workaround described above is no longer necessary?
I have attached 3 logs, one with a clean start-shutdown, one where the cpu throws the error but doesn’t loop and one where the loop happens and the machine cannot shutdown. These logs were collected from the devkit.
devkit-r39.2.1_boot_005_cpu_error_no_loop.log (117.4 KB)
devkit-r39.2.1_boot_008_error_loop.log (144.2 KB)
devkit-r39.2.1_boot_010_no_error.log (37.9 KB)