Nvidia-smi not working while driver is installed


I recently experienced some failure that nvidia-smi gives “No devices found” but I have two 1080Ti GPU mounted, lspci also showed my gpu cards. Before, a system reboot would solve this problem but recently the gpu cards didn’t show up even after the reboot. I doubt if this is a software or hardware failure?
The complete log is attached. Thanks so much for the support!
nvidia-bug-report.log.gz (2.5 MB)