Honestly, I’m very surprised that the CUDA documentation tells me to reboot the whole server to finish the CUDA upgrade. I’m using a single A100 on a Ubuntu server, without the need of graphic output, but do I still have to reboot the whole server? It’s 2024 now! The server is running and some people are using it. I cannot just kill their tasks and reboot!
I tried the sudo rmmod approach, which were discussed 8 years ago on the Internet, but the result, “rmmod: ERROR: Module nvidia_drm is in use” and lsof showed no process was using it, is really something beyond my knowledge.