Ubuntu "unattended-upgrades" for CUDA drivers

Graphics / Linux Linux
1

We have a long running deployment (it’s deep learning workload and it needs to be very persistent). Also there are some auto updates for Nvidia drivers and when this auto update happens, the below error happens:
“Failed to initialize NVM: Driver/library version mismatch”
so reboot is needed. So there comes two problem:

  1. we should lock/freeze updates by
    sudo apt-mark hold
    but I don’t know which packages should be hold or not? (I’ve done

sudo apt-mark hold libnvidia*
sudo apt-mark hold nvidia*

and also editing this file and added this config:

/etc/apt/apt.conf.d/50unattended-upgrades

Unattended-Upgrade::Package-Blacklist

{ “nvidia-”; “libnvidia-”; }

this but still there are packages that have nvidia in their name such as:

linux-headers-nvidia
linux-image-nvidia
linux-modules-6.8.0--nvidia
linux-modules-extra-6.8.0--nvidia
linux-modules-nvidia-server-
linux-modules-nvidia-fs-6.8.0
linux-nvidia
linux-nvidia-headers*
linux-nvidia-tools*
linux-objects-nvidia--server
linux-signatures-nvidia*
linux-tools-nvidia

Should I hold them too?)

  1. When this update happens, the drivers become updated but the fabric manager won’t update at all (so there is a mismatch and you should update the fabric manager manually). Is it normal? Should I consider something else?
2

That’s an issue for us too, we would like to stay with 570.124.06 but there is nothing to do, it updates to 570.133.20 and since we need to have the fabricmanager, i need to use : apt install nvidia-driver-open-570 but doing so, forces to update to 570.133.20, now the problem is i’m not sure if we can keep benchmarking if driver versions and a bunch of dependencies change. I tried apt-mark hold *nvidia* but it doesn’t seem to help.

3

Maybe I’m missing something here: if you folks want to stick with a particular version of Nvidia drivers and as they sit in a completely separate Nvidia apt repo, can’t you just remove Nvidia repo from your apt sources list after installing the desired driver version?