Actually spoke to soon, still triggers the warning sometimes but much better.
Would be good to figure out what is causing the over-current?
To repeat the test install ollama (native)
Then run: ollama run gemma2
THen run a query, I used “tell me about the roman empire”
Or “tell me about the british empire”
Hi, both
Thanks for sharing the info.
There are two possible throttling: thermal and OC.
Below is the document to gather more event log for your reference:
https://docs.nvidia.com/jetson/archives/r36.4/DeveloperGuide/SD/PlatformPowerAndPerformance/JetsonOrinNanoSeriesJetsonOrinNxSeriesAndJetsonAgxOrinSeries.html#jetson-orin-nx-series-and-jetson-orin-nano-series
In order to reproduce the exact same behavior, could you share the step-by-step with us?
Including the container/setup/command that is required for reproducing the behavior.
Thanks.
Hi, all
Since we just released the JetPack 6.2 which also supports super mode.
Please try to reflash the system (rather than the SDcard image) to see if the issue can still be observed.
Thanks.
Will someone please provide exact terminal commands being suggested to “reflash the system” and I assume to get JetPack 6.2. Thanks in advance.
@AastaLLL There’s a bug with your SDK Manager installer for Jetpack 6.2.
16:07:51 ERROR: File System and OS - target_image: E: Unable to locate package netcat-openbsd-openbsd
16:07:51 ERROR: File System and OS - target_image: [exec_command]: /bin/bash -c /home/[user]/.nvsdkm/replays/scripts/JetPack_6.2_Linux/NV_L4T_FILE_SYSTEM_AND_OS_COMP.sh; [error]: E: Unable to locate package netcat-openbsd-openbsd
This should probably be “netcat-openbsd” instead of “netcat-openbsd-openbsd”. SDK Manager is overwriting this install script so I can’t modify in manually.
@energyideas they are referring to the SDK Manager flashing method listed on the jetson orin nano super homepage.
What do these 16:07:51 ERROR mean, what are you referring to? Are you suggesting something users who have persistent “Offline” errors for CV0, CV1 and CV2 should be doing. Again, please either provide exact terminal command suggestions or links to any webpages like “jetson orin nano super homepage”. Thanks in advance.
Same here.
The installation issue is caused by ‘netcat-openbsd’ in the Linux_for_Tegra/tools/l4t_flash_prerequisites.sh. The keyword
‘netcat’ in Jetpack 6.1 but modified to ‘netcat-openbsd’ in Jetpack 6.2.
Unfortunately, the sdk manager’s NV_L4T_FILE_SYSTEM_AND_OS_COMP.sh still replaces string in il4t_flash_prerequisites.sh with following pattern
sed -i ‘s/netcat/netcat-openbsd/’ ./tools/l4t_flash_prerequisites_.sh.
Either the SDK manage revises for it or revert ‘netcat-openbsd’ to ‘netcat’ in Jetpack 6.2, please Nvidia fix it ASAP.
@rockefellerlin In the end I was able to upgrade Jetpack on the device itself using the instructions here: How to Install and Configure JetPack SDK — JetPack 6.1 documentation
I think as long as you are already on Jetpack 6.x you no longer have to use an external device/SDK Manager.
Unfortunately, the over-current throttling issue remains with 6.2.
Thanks, I have successfully updated my Jetson Orin Nano from Jetpack 6.1 to 6.2 via ‘Software update’ in Ubuntu 22.04, and as same as you, still see the throttling issue.
I also wonder what test can make Jetson Orin nano touch 25W power consumption, here I only see about 21~22W.
Thanks to mfchin and rockefellerlin for confirming that upgrading to jetpack 6.2 does not fix the “offline” errors.
Hi, all
The upgrade command cannot change the rootfs from SDCard to NVMe.
Could you help set up the OS on the NVMe SSD?
You can find the command in the below link:
Jetson Orin Nano Developer Kit (NVMe):
We have tested super mode with the NVMe rootfs and didn’t see the OC throttling issue as reported.
In the meantime, we are trying to reproduce this issue on both SDCard and NVMe environments to gather the info.
Thanks.
Thanks. I didn’t need to change the rootfs from SD card to NVMe as I had flashed 6.1 directly to the NVMe when I got the device through SDK Manager. I have NVMe rootfs, am running Jetpack 6.2 and have the OC throttling.
Bringing this back to the original subject; “Offline” error is not fixed, regardless of Jetpack version. Many with this error.
My first day of ownership produced the error, two weeks now without any potential solutions. MAXN not possible.
Thanks for any solutions.
Hi, @energyideas
“offline” is not a bug.
Because Orin Nano doesn’t have PVA and DLA, these components (the temperature of PVA and DLA) are expected to be offline.
Please see the above comment:
Thanks.
Hi, all
Thanks for your patience.
We test Qwen2.5-1.5B VLM model withdustynv/llama-factory:r36.4.0 container.
Here are some finding:
1. Behavior for SDCard or NVMe are similar.
2. Throttling is triggered but does not impact performance much.
By running:
$ sudo grep "" /sys/class/hwmon/hwmon*/oc*
/sys/class/hwmon/hwmon3/oc1_event_cnt:0
/sys/class/hwmon/hwmon3/oc1_throt_en:1
/sys/class/hwmon/hwmon3/oc2_event_cnt:0
/sys/class/hwmon/hwmon3/oc2_throt_en:1
/sys/class/hwmon/hwmon3/oc3_event_cnt:194
/sys/class/hwmon/hwmon3/oc3_throt_en:1
There are some oc3 throttling events that occur.
Based on our doc here, oc3 indicates the device is over the “VDD_IN Instantaneous Power: 25W” limit.
The adjustment is to reduce CPU and GPU clocks to 50%.
But when you check the GPU clocks with tegrastats:
01-20-2025 10:20:45 RAM 5609/7620MB (lfb 9x1MB) SWAP 869/20194MB (cached 0MB) CPU [4%@1728,0%@1728,57%@1728,3%@1728,0%@1728,0%@1728] EMC_FREQ 0%@3199 GR3D_FREQ 0%@[1019] NVDEC off NVJPG off NVJPG1 off VIC off OFA off APE 200 cpu@51.562C soc2@51.093C soc0@50C gpu@50.875C tj@51.562C soc1@51.093C VDD_IN 7732mW/6981mW VDD_CPU_GPU_CV 2120mW/1677mW VDD_SOC 2524mW/2434mW
01-20-2025 10:20:46 RAM 5988/7620MB (lfb 1x256kB) SWAP 869/20194MB (cached 0MB) CPU [25%@1728,0%@1728,2%@1728,32%@1728,0%@1728,0%@1728] EMC_FREQ 15%@3199 GR3D_FREQ 98%@[1008] NVDEC off NVJPG off NVJPG1 off VIC off OFA off APE 200 cpu@52.031C soc2@52.031C soc0@50.5C gpu@54.437C tj@54.437C soc1@51.125C VDD_IN 17730mW/7006mW VDD_CPU_GPU_CV 9352mW/1694mW VDD_SOC 3711mW/2437mW
01-20-2025 10:20:47 RAM 6002/7620MB (lfb 1x4MB) SWAP 867/20194MB (cached 1MB) CPU [18%@1728,2%@1728,3%@1728,0%@1728,5%@1728,1%@1728] EMC_FREQ 26%@3199 GR3D_FREQ 99%@[1007] NVDEC off NVJPG off NVJPG1 off VIC off OFA off APE 200 cpu@52.531C soc2@52.875C soc0@50.968C gpu@55.593C tj@55.593C soc1@51.531C VDD_IN 19955mW/7035mW VDD_CPU_GPU_CV 10910mW/1715mW VDD_SOC 3936mW/2441mW
01-20-2025 10:20:48 RAM 6002/7620MB (lfb 1x4MB) SWAP 867/20194MB (cached 1MB) CPU [0%@1728,0%@1728,2%@1728,0%@1728,0%@1728,0%@1728] EMC_FREQ 30%@3199 GR3D_FREQ 99%@[1007] NVDEC off NVJPG off NVJPG1 off VIC off OFA off APE 200 cpu@52.812C soc2@53.187C soc0@51.437C gpu@55.75C tj@55.75C soc1@51.687C VDD_IN 19955mW/7064mW VDD_CPU_GPU_CV 10988mW/1736mW VDD_SOC 3890mW/2444mW
The impact is very slight (GPU clock from 1019 to 1007).
This indicates the duration of the clocks being throttled is not that long.
So we expect this won’t impact the performance too much.
Would you mind testing the same with your use case (including grep OC throttling event and monitor clocks with tegrastats).
So we can discuss further to see how to improve this issue.
Thanks.
Do you think repeating “not a bug” will make it more true? Because it isn’t true. Permanent “Offline” error does not allow MAXN.
Written on January 14th:
Hi, @energyideas
There are two issues discussed here.
1.
The offline message is because Orin Nano doesn’t have the PVA and DLA hardware. So it’s expected.
But in our understanding, the “offline” should be a status listed in the jtop.
Do you get any error messages or logs from the system (either dmesg or UI) about the cv0, cv1, and cv2 offline?
If so, please help to provide the log or screenshot to us.
2.
Since super mode increases the power consumption and clock rate, the device may hit throttling events more frequently (either OC or thermal throttling).
This is what we are trying to help.
In order to know more about the behavior on your device, would you mind helping us collect the system status and throttling data?
Please help to run the below command concurrently:
$ sudo tegrastats
And collect the throttling event after the app is terminated:
$ sudo grep "" /sys/class/hwmon/hwmon*/oc*
We try to provide a custom nvpmodel mode for the LLM-related use case. (GPU usage >> CPU)
To get more info about the CPU/GPU utilization can help us design the custom mode.
Thanks.
Hi,
We test a custom nvpmodel with 2x CPU@1728, GPU@1020 and EMC@3199.
Under the heavy load GPU tasks (ex. LLM inference), we can still observe the OC throttling behavior.
But similar to the previous testing, the impact on the performance is limited as the clocks don’t decrease much. (1019->1003)
01-22-2025 06:53:47 RAM 7119/7620MB (lfb 4x4MB) SWAP 580/16384MB (cached 155MB) CPU [0%@1728,2%@1728,off,off,off,off] EMC_FREQ 43%@3199 GR3D_FREQ 99%@[1008] NVDEC off NVJPG off NVJPG1 off VIC off OFA off APE 200 cpu@63.25C soc2@63.812C soc0@62.218C gpu@66.25C tj@66.25C soc1@62.218C VDD_IN 19844mW/14802mW VDD_CPU_GPU_CV 10675mW/6251mW VDD_SOC 4008mW/3850mW
01-22-2025 06:53:48 RAM 7119/7620MB (lfb 4x4MB) SWAP 580/16384MB (cached 155MB) CPU [1%@1728,3%@1728,off,off,off,off] EMC_FREQ 38%@3199 GR3D_FREQ 99%@[1007] NVDEC off NVJPG off NVJPG1 off VIC off OFA off APE 200 cpu@63.343C soc2@63.968C soc0@62.218C gpu@66.531C tj@66.531C soc1@62.343C VDD_IN 20080mW/14843mW VDD_CPU_GPU_CV 11048mW/6288mW VDD_SOC 3890mW/3850mW
01-22-2025 06:53:49 RAM 7119/7620MB (lfb 4x4MB) SWAP 580/16384MB (cached 155MB) CPU [0%@1728,2%@1728,off,off,off,off] EMC_FREQ 36%@3199 GR3D_FREQ 98%@[1004] NVDEC off NVJPG off NVJPG1 off VIC off OFA off APE 200 cpu@63.5C soc2@64.031C soc0@62.531C gpu@66.718C tj@66.718C soc1@62.531C VDD_IN 20040mW/14882mW VDD_CPU_GPU_CV 10970mW/6323mW VDD_SOC 3929mW/3851mW
01-22-2025 06:53:50 RAM 7119/7620MB (lfb 4x4MB) SWAP 580/16384MB (cached 155MB) CPU [0%@1728,1%@1728,off,off,off,off] EMC_FREQ 35%@3199 GR3D_FREQ 99%@[1003] NVDEC off NVJPG off NVJPG1 off VIC off OFA off APE 200 cpu@63.593C soc2@64.156C soc0@62.593C gpu@66.625C tj@66.625C soc1@62.468C VDD_IN 20080mW/14921mW VDD_CPU_GPU_CV 10931mW/6358mW VDD_SOC 3968mW/3852mW
01-22-2025 06:53:51 RAM 7119/7620MB (lfb 4x4MB) SWAP 580/16384MB (cached 155MB) CPU [1%@1728,2%@1728,off,off,off,off] EMC_FREQ 34%@3199 GR3D_FREQ 99%@[1005] NVDEC off NVJPG off NVJPG1 off VIC off OFA off APE 200 cpu@63.843C soc2@64.156C soc0@62.687C gpu@66.718C tj@66.718C soc1@62.656C VDD_IN 19844mW/14958mW VDD_CPU_GPU_CV 10736mW/6391mW VDD_SOC 3929mW/3853mW
OC throttling is the essential mechanism to protect the device.
But you can turn off the notification and let it work on the backend.
Please find the below two comments to turn off the notification. (either one is fine).
Thanks.
That works for me. I also see very limited throttling, especially in Jetpack 6.2, so it appears that in reality none of this is really an issue, especially after the annoying notification is disabled.
Thanks for your help.
@mfchin seems satisfied, I am not. I would still like to see the “offline” error eliminated not just hidden, then I wouldn’t feel like the product is damaged. Cars in accidents get Carfax reports. This computer is permanently damaged as far as I can see, meaning that cv0, cv1 and cv2 are not reporting like they used to report.