[THOR] Cannot run LLM - System Reboots

Robotics & Edge Computing Jetson Systems Jetson Thor
23

Hi

The file is most empy so we are not able to find any useful information from it.

Please connect the UART (no need to use demuxer).
After rebooting, please share the log before/after the reboot with us.

The information we want is the dmesg log, as the shared thor-boot.log below:

But we need the log right before the reboot happens.

We have tested the issue in our internal environment.
But not able to reproduce the rebooted issue with a upgraded Thor (from 38.4 to 39.2.1).

Thanks.

24

Thank you @whitesscott for your patience on this.

Based on the commands you noted, I have ran the commands from the top - and I recieved multiple reboots on the NVIDIA Thor that did not render the device able to login for more than a couple minutes maximum. I was attempting to run the commands via SSH but instead (while the demuxer was running) I simply attached to the demuxer session while the system was rebooting.

I am attaching two logs (from two attempts each, attempts meaning waiting for reboot):

Thank you!
E

25

Thank you @AastaLLL for the direction and the update. Additionally, thank you for spending the time and effort pushing for testing this in an internal environment.

I attempted to run the command (with updated filenames) to help capture data we are looking for:

sudo minicom -D /dev/ttyACM0 | tee -a <logname>.txt

During a reboot cycle, I attempted to run sudo dmesg > log.txt (inside the NVIDIA Thor via UART session - which ran successfully), and then the system rebooted. After the reboot, I reran the command sudo minicom -D /dev/ttyACM0 | tee -a <logname>.txt, which I believe shows in the respective log.

Please see logs attached:

I look forward to your updates and review!

Thank you!
E

26

Good day all,

I hope all is well with both of you and the Nvidia team.

Anything else we can think of to help troubleshoot?

Thanks,
E

27

In one terminal start and let this run:

sudo dmesg -H --nopager --follow| tee before_dmesg.txt

In second terminal start vllm or whatever method reliably causes reboot.

After Thor reboots and you have logged on, run:

sudo dmesg -H --nopager | tee after_dmesg.txt

Then post both txt documents here.

28

Good day @whitesscott !

Thank you for continuing to support this thread! I apologize if previous communications were mixed up on my side. I have run the commands you have noted and attached the output below:

Thank you!
E

29

Hi,

We check all the attached logs and somehow fail to find the issue you met.

Could you help us to collect the log again?

  1. Open a standalone console on the host and run
$ sudo minicom -D /dev/ttyACM0 -b 115200

  1. Test Thor and wait for it reboot

  2. Manually copy all the logs shown on the desktop console and share with us.

Thanks.

30

@AastaLLL ,

Thank you for continuing to troubleshoot with me. I have ran the following command:

sudo minicom -D /dev/ttyACM0 -b 115200 | tee -a 08282026console.log

I am attaching the log here: 08282026console.log (606.7 KB)

Thanks,
E

31

As far as I can tell that also didn’t capture the error. The following should at least capture the state of

TIME	SEQ	GPU%	GPU_MHz	GPU_W	GPU_TEMP	CPU_TEMP	VIN_mW	GPU_mW	CPU_SOC_mW	OC1	OC2	OC3

and that might be helpful.

Install jtop/jetson_stats on Thor.

curl -LsSf https://raw.githubusercontent.com/rbonghi/jetson_stats/master/scripts/install_jtop_torun_without_sudo.sh | bash

Download attached jtop_capture_log.py.txt to your home directory with name jtop_capture_log.py

Run this from the same directory where you saved jtop_capture_log.py

$HOME/.local/share/jtop/bin/python3 jtop_capture_log.py

In another terminal immediately start vllm or do whatever reliably causes reboot.

After Thor reboots:

cp jtop-capture.jsonl jtop-capture.jsonl.txt

Then post jtop-capture.jsonl.txt here.
jtop_capture_log.py.txt (8.1 KB)

32

Good day @whitesscott,

Thank you for the research and the support! I see the software you are referring to is hosted on GitHub. It looks like it’s maintained by a person / org that is not Nvidia.

Could Nvidia Support prove this software is NVIDIA approved?

Thanks,
E

34

Hi,

We are checking the possibility from the log you shared above.
In the meantime, could you capture the log with the above approach?

It can save log automatically so you don’t need to use the | tee ... to dump it.

$ sudo minicom -wD /dev/ttyACM0

image
image495×277 68.5 KB

Enter I > Enter filename > Enter Q to leave.

Thanks.

35

Good day @AastaLLL ,

Thank you for the response. I have run the commands you noted in your latest reply. Here is the attached log:

08312026minicom.log (218.9 KB)

If I am mistaken in your request, please send an update. Please let me know if there is anything else I can help to troubleshoot.

UPDATE: It appears that the Thor is not able to stay powered on for very long. I have only used the Nvidia Thor for troubleshooting since before starting this post. I have a video here of the same data we requested. I have updated the previous photo / video album here: https://photos.app.goo.gl/PyszcqPKEZ29bn8s6. As evident in this video, the Nvidia Thor does not respond to power on / restart requests. The Nvidia Thor only turns on if it is plugged in.

Thanks,
E

36

Hi,

We originally suspected the issue was related to the following error:

[   10.117988] dce: dce_ipc_channel_init_unlocked:248  Invalid Channel State [0x0] for ch_type [2]
[   10.128321] dce: dce_admin_send_cmd_ver:824  version : dcefw:[0x4] dcekmd:[0x4] err : [0x0]
[   10.129282] tegra-so135009] dce: dce_admin_setup_clients_ipc:1002 Channel Reset Complete for Type [1] ...

However, it looks more like a power stability issue.
Could you please try the following experiments to see if they help resolve the issue?

  1. Connect the power supply directly to a wall outlet (without using an extension cord).
  2. Try a different power supply (if available) to see if the issue persists.

Thanks.

37

Good day @AastaLLL,

Thank you for the information and feedback from you and the team’s research. Thank you and @whitesscott for doing incredible troubleshooting of this issue with me.

I am able to confirm a number of things:

  • Using a different NVIDIA Thor Power supply provided by a local MicroCenter results in the same issue (plugged in at MicroCenter, then later plugged in at home) renders no difference in result
  • Using a different outlet (many around the house) renders no difference in result
  • Using no surge protector renders no difference in result

Regardless of changes, there is no difference in result (Nvidia Thor continues to restart automatically).

Thanks,
E

38

Hi,

Thanks for your testing.
Are you able to reflash the system (either r38 or r39) to see if it helps?

Thanks.

39

Good day @AastaLLL ,

I do not believe device will stay powered on long enough for a reflash to occur. I can confirm when upgrading from 38 to 39, it took around an hour. Based on previous messages - I believe we have narrowed this down to hardware.

Am I mistaken in that assessment?

Thank you,
E

40

Good day @AastaLLL ,

I hope all is well with you and yours.

I also wanted to add some information to answer a previous response - I have attempted to use both USB Type-C (vertical, not the debug port) to power the unit. Additionally, I have tried all possible combinations of the above (12 of 12) - with no change in results (Nvidia Thor reboots all by itself).

Is there any way we could discuss other options to get to a resolution?

Thanks,
E

41

I apologize for the multiple pings @AastaLLL,

Simply put - I am wondering if you could confirm whether (based on everything done / collected so far) you believe the device is exhibiting a hardware fault, and you recommend an RMA?

Thanks,
E

42

Hi,

Would you mind manually reflash the device with r39.2.1:

Please share the log with us if the reflashing failed.

Thanks.

43

Hi,

To clarify, the power stability issue mentioned above refers to instability from the extension cord, not a hardware problem with the Thor device itself.
To help us further investigate and isolate the issue, could you please re-flash the device firmware and check if the problem still occurs?

Thanks.