We are working on a heavy load GPU application that involves inference on an Xavier NX with Jetpack 4.6. To obtain maximum performance we are using the 20W 6CORE predefined mode. When running our application we see several times the “System throttled due to overcurrent” message.
We’ve seen in other posts such as this and this that one workaround is to set the current maximum limit to 5A, however it has been reported that with high GPU loads it won’t work. That is our case, increasing the maximum limit is not enough to solve the problem. So, we would like to understand it better with the following questions:
Is this warning related to temperature or power consumption? If not then, could you explain what triggers it?
Is it harmful for the device to just run the application and ignore the warnings?
What are the consequences of the system being throttled? Are the clocks frequencies lowered? Will the performance drop?
I think The error message "System throttling due to over-current" appears when running YOLOv4 - #27 by WayneWWW alraedy explained this.
This is related to power consumption. Really indicating that there is a over current happened.
If you don’t change the maximum limit to another higher value, then it won’t harm the device.
The performance will be dropped.
This mechanism is to protect the device from being over-current. It means the performance of your NX has been pushed to limit.
Thanks for your answers @WayneWWW, that is really helpful. A couple of questions more:
- You mentioned that it is related to power consumption, so is there a known limit of power consumption before the warning is triggered? Something like maybe we could monitor with the tegrastats tool?
Also, to elaborate more on how is the performance dropped:
- Does the system switch power modes to a lower one? I’ve noticed that when the warning shows and we query the current model with nvpmodel -q it remains unchanged.
- Are the clock frequencies lowered for the CPU or the GPU? And if so, do you know at what values?