How to fix NVML: Driver/library version mismatch _without_ rebooting

[remi.coulom1]

My Ubuntu server performed unattended upgrades, and I cannot use nvidia libraries any more. I know that rebooting will fix the problem, but I am not able to reboot the machine remotely, so I’d like to find a way to undo the unattended upgrades instead.

Can anyone suggest a solution?

This is a similar question, but no answer was given:

Nvidia-smi failed to initialize NVML (driver/library version mismatch)

Hi, I tried to upgrade my Cuda version using commands : wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pinsudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600wget https://developer.download.nvidia.com/compute/cuda/11.7.0/local_installers/cuda-repo-ubuntu2004-11-7-local_11.7.0-515.43.04-1_amd64.debsudo dpkg -i cuda-repo-ubuntu2004-11-7-local_11.7.0-515.43.04-1_amd64.debsudo cp /var/cuda-repo-ubuntu2004-11-7-local/cuda-*-keyr…

This seems to offer a solution for downgrading:

Ubuntu install specific old cuda drivers combo

So I did some testing, and it does look like there is ever so slightly a way around this with the right combination. I did a test of all the versions of cuda and all versions of cuda-drivers in the repo as of this morning, after the latest drivers were added. for cudaver in $(apt list cuda -a 2>/dev/null | awk '{print $2}' | xargs) ; do for driverver in $(apt list cuda-drivers -a 2>/dev/null | awk '{print $2}' | xargs) ; do echo $cudaver $driverver ; sudo apt-get install cuda=$cudaver cuda-dri…

But I cannot get it to work. My current running driver is 535.104.05, as found by running dmesg:

[1214031.466153] NVRM: API mismatch: the client has the version 535.104.12, but
                 NVRM: this kernel module has the version 535.104.05.  Please
                 NVRM: make sure that this kernel module and all NVIDIA driver
                 NVRM: components have the same version.

Trying to re-install this specific driver does not work:

sudo apt install nvidia-driver-535 cuda-drivers=535.104.05-1
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 cuda-drivers : Depends: cuda-drivers-535 (= 535.104.05-1) but 535.104.12-1 is to be installed
E: Unable to correct problems, you have held broken packages.

What can I do? Any suggestion is very welcome.

[Mart]

Decide which driver you need. Only one of them cuda-driver, or nvidia-driver, can be installed.

sudo apt purge DRIVER_TO_DELETE
sudo apt reinstall DRIVER_OF_CHOICE

sudo systemctl isolate multi-user.target

lsmod | grep nvidia - to identify the modules needing to be reloaded.
sudo moprobe --remove ALL_MODULES_IDENTIFIED_ABOVE
sudo insmod ALL_MODULES_IDENTIFIED_ABOVE

sudo systemctl isolate graphical.target

[remi.coulom1]

Thanks! you saved my life.

I had to stop all running calculations first.

insmod did not work, but “modprobe nvidia” did.