vGPU and CUDA

Hi,

How can I install CUDA on the HOST with vGPU driver installed.
vGPU is working fine, I can assign profiles to the VMs etc… but I need to run some CUDA stuff on the host too.
Using Debian 12 with Tesla T4 (550.127.06). According to all manuals I should be able to install CUDA Toolkit 12.4 without any issues, but that’s not the case. I can install cuda-toolkit-12 just fine, but when I try to install libcuda1=550.127.05 (the closest version to my driver) (because, libcuda1 is being installed with the driver package, and by default it will install 570xx which BREAKS my vGPU driver) I’m getting a dependency hell

The following packages have unmet dependencies:
libcuda1 : Depends: nvidia-alternative (= 550.127.05-1)
Depends: libnvidia-ptxjitcompiler1 (= 550.127.05-1) but 570.86.15-1 is to be installed
Depends: libnvidia-nvvm4 (= 550.127.05-1) but 570.86.15-1 is to be installed
Recommends: nvidia-kernel-dkms (= 550.127.05-1) but it is not going to be installed or
nvidia-kernel-550.127.05
Recommends: nvidia-smi
Recommends: libnvidia-cfg1 (= 550.127.05-1) but it is not going to be installed
Recommends: nvidia-persistenced but it is not going to be installed
Recommends: libcuda1-i386 (= 550.127.05-1) but it is not installable
E: Unable to correct problems, you have held broken packages.

I also tried to install CUDA 12.4 from a runfile, excluding the drivers, and again libcuda1 is not present in that package and I cannot seem to install it in any way

So what’s the logic behind this and how can I install cuda to run on my host with libcuda1 libs ?