I recently was observing the CPU/GPU temperatures and found out that the CPU was running way too hot (up to 92-95c) on the head node. Turns out that the dgx spark fan is mainly controlled by the GPU temperature, even if the CPU is getting melted
The underclocking of the GPU caused such low GPU temperature that the CPU was starved of cooling. AI confirmed those findings, but maybe others have different opinions
GPU underclocking needs to be done carefully it seems on the dgx spark. After removing the underclock I am still hitting 86c on the head node. I am using the dual spark 3d print housing
I don’t have a cluster setup but on a single node with heavy inferencing I don’t see the CPU going past 70C EVER, normally it hovers at 50-60s .
If you can swap your head with your secondary node to test, I would try that and see if it’s actual heavy CPU utilization causing that temp, or maybe one of your GB10s needs a thermal paste job.
Yes but first get a good FAN - I first used a 10 years old PC fan that max out around 1300 rpm. It probably bring down the 3-6 degrees. Nice but GPU underclocking does better.
I’ve since replace the fan with a ARTIC P12 PRO (max 3000 rpm) ,there are much more expensive and better fans but this was in the right price range for me. It’s a world of differences in cooling and I can really feel the differences and hear the differences on max speed.
For fan speed control - 1. Get a PWN , these are a few dollars on Aliexpress or you can pay alot more on Amazon. Manual control only, I have not found a BLE PWN controller. If you find a BLE PWN, then that solves the external fan problem.
Someone posted a simple script/patch that write to the firmware smem for the fan, that will allow you to change the fan profile . I’ve not tried it, but it couldn’t be difficult to extend the script to monitor CPU temp and switch the fan profile. Assuming the script works the way I think it does.
I’m not running an underclock but I also often observe CPU temps ~10°C hotter than GPU during inference workloads (CPU utlisation at ~5%-10%) on Gigabyte AI TOP ATOM. Even with a fan positioned at the exhaust drawing away hot air I’m getting into the 90s on CPU which is higher than I’d like - seems the cooling solution is probably weighted a bit too heavily towards the GPU.
