Hi all, may I tap your collective wisdom about an odd performance issue on one of our deep-learning nodes?
First, does the hardware profile itself raise any red flags? The box runs 8 × NVIDIA A40s (40 GB each) on PCIe, dual EPYC CPUs giving 64 physical cores, and a hefty 4 TB of DDR4-3200 ECC RAM. The software stack is Ubuntu 20.04 LTS, NVIDIA driver 550.*, CUDA 12.4, and PyTorch 2.2 built for that CUDA line. Everything screams along at expected speed for about a week.
Then, why does the very same training job—identical data, batch size, and code—suddenly slow to roughly one-quarter of its original throughput after 7–14 days of continuous uptime? GPU clocks stay at boost, temps hover in the 60 °C range, nvidia-smi shows no throttle flags or ECC errors, and the PCIe links remain x16 Gen4. CPU usage, I/O wait, and memory pressure all look perfectly normal. Yet a single reboot snaps performance back to normal, only for the slowdown to re-appear a week or two later.
What could possibly accumulate over time to throttle GPU throughput when no obvious counter (clocks, temps, ECC, power, PCIe) reports distress? Could it be a kernel or driver resource leak? Might long-lived CUDA contexts, NCCL communicators, or MIG remnants be decaying performance behind the scenes? Is there any known issue with the 550 driver line or CUDA 12.4 that matches this symptom?
Which live metrics or traces would you capture to catch the moment the slowdown begins? Would an Nsight Systems 30-second sweep, a rotating nvidia-smi dmon log, or kernel ftrace reveal a culprit that basic monitoring misses? Is there a way to reset the GPUs, unload the driver, or re-initialise NCCL without performing a full system reboot, just to confirm where the bottleneck lives?
Finally, has anyone here faced—and solved—a similar “runs-fast-for-a-week, then crawls until reboot” pattern on multi-GPU EPYC boxes? Any pointers or war stories would be hugely appreciated, because weekly scheduled reboots are becoming a real productivity drain.
Thanks in advance for any insight!