One of three thors powers off whenever it's under heavy load, even after reflashing

Hello,

NVIDIA Customer Care case: 261005-000268

I purchased three NVIDIA Jetson AGX Thor Developer Kits on August 21. NVIDIA Customer Care asked me to post the technical issue here so the Jetson/developer team can review it before they proceed with the RMA process.

All three units were flashed using NVIDIA SDK Manager with the same L4T image and configured for essentially identical workloads. Two are operating normally. The third has a persistent unit-specific issue where it becomes extremely hot under sustained GPU inference load and eventually hard-powers-off.

Problem

The affected Thor abruptly loses power after several minutes of heavy, sustained GPU utilization.

This is not a normal reboot or application crash. The entire system powers off, and the logs do not show a kernel panic, OOM event, orderly shutdown, or other normal OS-level explanation immediately before the failure.

This has now occurred three times:

  • September 24: hard power-off under sustained GPU inference load

  • September 25: hard power-off again under sustained GPU inference load

  • October 3: hard power-off again under sustained GPU inference load, after the unit had been completely reflashed

The September 28 reflash used the same L4T release used successfully on the other two Thor units. I then reinstalled the workload and reproduced the same failure.

Workload

The systems are being used primarily for local AI inference. The setup includes Docker and workloads such as Ollama, Whisper, and Flux.

The crashes occur during prolonged, sustained GPU utilization rather than during idle or light use.

The affected unit is running in the 120 W power mode.

Thermal behavior

Immediately before each hard power-off, the affected unit enters repeated GPU thermal-throttle states and logs:

thermal cooling_device9: gpu-throttle-alert cooling state: 0 -> 1

For the three failure events, I recorded approximately:

  • 3,615 throttle-state transitions before the first shutdown

  • 566 before the second shutdown

  • 1,855 before the post-reflash shutdown

During the post-reflash incident, GPU temperature reached approximately 109°C, with the last telemetry before shutdown at approximately 108°C.

The other two Thor units are performing the same class of sustained workload without exhibiting this extreme thermal behavior or shutting down.

GPU / driver errors

During one of the pre-reflash incidents, the kernel logged the following GPU memory-mapping error approximately 31 minutes into sustained GPU load:

Cannot map memory with base addr 0xfffce0000000 and size of 0x2aa08 pages

This is the only occurrence of this type of GPU memory-mapping fault that I found in the persisted kernel record.

The affected unit also produces the following GPU initialization signatures on every boot:

xidfc_flesa | 0x00000040 |

and repeated:

NVRM_RPC_DCE: Failed RM ctrl call cmd:0x731341

including both:

NV_ERR_GENERIC

and

NV_ERR_TIMEOUT

These signatures were present on 14/14 recorded boots before the reflash and are still present on 3/3 recorded boots after the reflash.

Reflash test

To eliminate the existing installation/configuration state as much as possible, I completely reflashed the affected Thor and reinstalled the environment.

The other two Thor units use the same software/image successfully.

After the reflash:

  1. The GPU initialization errors remained.

  2. The affected unit continued to run significantly hotter than the other systems.

  3. Under sustained GPU load, it again entered repeated gpu-throttle-alert states.

  4. The entire unit again abruptly powered off.

There was no kernel panic, OOM event, or normal shutdown associated with the post-reflash failure.

System information

Affected unit:

  • NVIDIA Jetson AGX Thor Developer Kit

  • GPU part number: 2B00--A1

  • L4T: R39.2

  • Kernel: 6.8.12-1021-tegra

  • Ubuntu 24.04.4 LTS / aarch64

  • NVIDIA driver: 595.78

  • CUDA: 13.2

  • NV Power Mode: 120 W

I am not posting the unit’s full serial number publicly, but I can provide it directly to NVIDIA through the support case if needed.

Host-visible ECC/retired-page counters appear to be unavailable on this platform; the relevant nvidia-smi fields report N/A.

Fleet comparison

I have three Thor Developer Kits configured for the same project and running the same general workload.

The other two systems are working normally.

Only this physical unit:

  • reaches these extreme GPU temperatures,

  • repeatedly enters prolonged GPU thermal throttling,

  • experiences the hard power losses,

  • and reproduced the same behavior after a complete reflash.

I have retained the kernel logs, system logs, thermal telemetry, and pre- and post-reflash diagnostic captures and can provide them to NVIDIA if useful.

At this point the unit is not usable for sustained workloads.

Could the NVIDIA team please advise whether there are any additional diagnostics you would like me to run, or whether the observed behavior is sufficient to proceed with hardware replacement/RMA through Customer Care?

Thank you.

  1. Could you share the dmesg for that SOM after boot up without running anything else?

  2. What is the power source you are using on the board?

  3. Have you tried to swap the SOM on that board to another one to crosscheck?

Hello,

Below are the GPU initialization messages from the affected unit’s October 3 boot, after the reflash. These are excerpts from the saved /var/log/kern.log, rather than a fresh standalone dmesg capture with all workloads disabled.

2026-10-03T12:05:27.076105-07:00 thor-1 kernel: Machine model: NVIDIA Jetson AGX Thor Developer Kit
2026-10-03T12:05:27.076758-07:00 thor-1 kernel: DMI: NVIDIA NVIDIA Jetson AGX Thor Developer Kit/Jetson, BIOS 39.2.1-gcid-46758480 08/07/2026
2026-10-03T12:05:27.080384-07:00 thor-1 kernel: 	| xidfc_flesa |0x00000040|
2026-10-03T12:05:27.080461-07:00 thor-1 kernel: 	| xidfc_flesa |0x00000040|
2026-10-03T12:05:27.080529-07:00 thor-1 kernel: 	| xidfc_flesa |0x00000040|
2026-10-03T12:05:27.080746-07:00 thor-1 kernel: 	| xidfc_flesa |0x00000040|
2026-10-03T12:05:27.644991-07:00 thor-1 kernel: NVRM: devm_reset_control_get failed, err: -2
2026-10-03T12:05:27.644992-07:00 thor-1 kernel: NVRM: devm_reset_control_get failed, err: -2
2026-10-03T12:05:27.644996-07:00 thor-1 kernel: NVRM: mipi_cal devm_reset_control_get failed, err: -2
2026-10-03T12:05:27.645014-07:00 thor-1 kernel: NVRM: loading NVIDIA UNIX Open Kernel Module for aarch64  TempVersion  Release Build  (r595_00)  (buildbrain@74e043fc-6c96-4a54-a660-50b30a72108d-97k4-j97dd)  Thu Aug  6 10:18:15 PM PDT 2026
2026-10-03T12:05:28.567760-07:00 thor-1 kernel: nvidia-modeset: WARNING: HW supports 8 heads. Limiting to 4 heads
2026-10-03T12:05:28.567783-07:00 thor-1 kernel: nvidia-modeset: WARNING: GPU:0: The head configuration (0xff) is inconsistent with the number of heads (4)
2026-10-03T12:05:28.772865-07:00 thor-1 kernel: NVRM: GPU1 rpcRmApiControl_dce: NVRM_RPC_DCE: Failed RM ctrl call cmd:0x731341 result 0xffff: Failure: Generic Error [NV_ERR_GENERIC]
2026-10-03T12:05:28.784820-07:00 thor-1 kernel: NVRM: GPU1 rpcRmApiControl_dce: NVRM_RPC_DCE: Failed RM ctrl call cmd:0x731341 result 0x65: Call timed out [NV_ERR_TIMEOUT]
2026-10-03T12:05:29.144835-07:00 thor-1 kernel: message repeated 31 times: [ NVRM: GPU1 rpcRmApiControl_dce: NVRM_RPC_DCE: Failed RM ctrl call cmd:0x731341 result 0x65: Call timed out [NV_ERR_TIMEOUT]]

The xidfc_flesa and RPC timeout messages also appear on the other recorded post-reflash boots. I have retained the complete kernel log if you need the full boot context.

I am using the USB-C power supply supplied with the Thor Developer Kit. I have tried swapping the USB-C power supplies. It didn’t make any difference.

I am not comfortable disassembling the units and swapping parts between them for testing. That seems like an unreasonable risk to take with such expensive and delicate hardware.

Hi,

  1. Could you directly reflash that problematic device with image again and share me the full dmesg right after boot? I mean not a log from like 3 days ago but a fresh one directly from the UART console instead of kern.log.

  2. What is the spec of the type C adapter from your NV devkit?

  3. Could you keep running tegrastats when doing test and share us its result right before issue happened?

Hello,

Thank you. Here is the update on your three requests:

1. I completed another reflash and saved UART output and a new dmesg capture as `first_boot.dmesg.txt`. I have prepared copies with private identifiers redacted for the public forum; the diagnostic error messages and their timestamps are preserved. This is a new capture rather than the previous October 3 `kern.log` excerpts. The saved file contains the following messages during initialization:

[ 19.534301] NVRM: GPU1 rpcRmApiControl_dce: NVRM_RPC_DCE: Failed RM ctrl call cmd:0x731341 result 0xffff: Failure: Generic Error [NV_ERR_GENERIC]

[ 19.547854] NVRM: GPU1 rpcRmApiControl_dce: NVRM_RPC_DCE: Failed RM ctrl call cmd:0x731341 result 0x65: Call timed out [NV_ERR_TIMEOUT]

There are also 96 RPC timeout messages and three generic RPC errors for this command, between approximately 19 and 25 seconds after boot.

2. I am using the supplied NVIDIA USB-C adapter, model ADP-240JB B. Swapping the supplied adapters between units did not change the behavior.

3. I ran the installed inference workload while recording tegrastats continuously. The latest run retained 25 hours 36 minutes 30 seconds of observation and 407 NVIDIA GPU0 `NV_ERR_NO_MEMORY` allocation-error entries, averaging 15.89 errors per hour. The mean interval between consecutive errors was 3 minutes 43 seconds, approximately 3.8 minutes; the errors were not evenly spaced.

At the nearest tegrastats samples to these errors, 27,692–35,989 MB of host RAM remained unallocated, with a median of 30,116 MB. These are total-minus-used measurements.

These problems began before sustained stress testing: initially the recurring failures prevented me from completing installation/setup and getting the inference tools running reliably. The retained Thor-1 records show failures at model loading and inference startup, including:

NVRM: GPU0 nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1336

An earlier attempt completed one long request, then failed during the next model-loading window. That window contained **41 NVIDIA allocation-error entries**; the Ollama `/api/ps` liveness check timed out and the request was canceled. Ollama logged `client connection closed before llama-server finished loading, aborting load` and `timed out waiting for llama-server to start: context canceled`. Separately, the VS Code client received an HTTP 500 reporting that the llama-server process had terminated with `GGML_ASSERT(stat == cudaSuccess)`. These records show the setup/loading and CUDA failures I encountered, although their timing alone does not prove a single cause for all of them.

The original hard power-off did not reproduce. The exceptionally high frequency of errors—an allocation-error entry approximately every 3.8 minutes on average—has made it impossible for me to keep the GPU reliably pegged and recreate the uninterrupted sustained-load conditions that caused the previous crashes. In practice, recurring allocation errors and provider loading failures disrupt the attempt before I can maintain that condition. This is an obstacle to reproducing the original failure, not evidence that the unit has recovered or that the previous crashes are resolved.

Some long inference requests do complete—64 completed during this run—but successful individual requests are not the continuously pegged GPU condition I am trying to reproduce. The test also unloads/reloads the model between requests, so the logs do not attribute every load gap to an allocation error; the inability to maintain the required condition is my operational observation.

The other two Thors run the same workload and inference-provider set installed through the same bootstrap. They do show occasional similar allocation errors, but their reported rates are **0.01 and 0.03 errors per hour**, compared with **15.89 per hour on this Thor**. This unit’s rate is therefore approximately **1,600 times** the lower-rate peer and **530 times** the other peer. The difference is substantial even though the peers are not entirely error-free.

I have retained the kernel and continuous tegrastats captures for your review, and removed any sensitive personal information before posting here, but I can also send you the complete files directly if you prefer.

Could you confirm whether the boot-time RPC errors above are expected on this developer kit, and advise what to collect next to distinguish a driver/allocation issue from a fault in this particular unit? The bottom line is that the frequent errors are obstructing reproduction of the original sustained-load power-off, and obstructing realistic use of the device in normal inference tasks.

ttyACM0.log (331.2 KB)

first_boot.dmesg.txt (127.3 KB)

Hi,

Want to confirm.

Do you see all these repeating NVRM error on all the 3 SOM you have on your side? or only on one SOM
Do you have a monitor connected during the test?
Where is the tegrastats log? That is what I want to check for your test environment but looks like it is not included in your log files.

Yes, NVRM errors are occuring on all three devices, but at rates of 0.1 per hour, 0.3 per hour, and 1600 per hour on the one that was powering off repeatedly. Because the errors are so high on that one, I can’t keep inference tasks running long enough to keep the GPU pegged and replicated the previous crashes at this point.

Yes monitor is connected.

Here is the tegrastats log;

tegrastats.log (52.4 MB)

Hi,

Just to confirm. So I assume the last few line from your tegrastats log is the data that before the power off happened?

Also, is it possible that your test to be done without monitor connected (just for an experiment). Those NVRM driver print is for display.

sudo nvpmodel -q
grep -H ‘’ /sys/class/hwmon/hwmon*/oc*_event_cnt

Please also share above info. The oc event cnt needs to be captured after your heavy load test started for a while.

Is this still an issue to support? Any result can be shared?

Yes, sorry. I need more time to work on this.