I have a small number of multi-GPU (P100) nodes in a multi-user environment. I am considering switching the compute mode from ‘default’ mode to ‘exclusive process’ mode to reinforce our fair-share and queueing (Univa) model - one job per GPU. Users can log in to the nodes directly which allows the possibility of GPU contention, which I want to avoid, without shutting off direct access completely.
Will the compute mode change apply only to new jobs or will running jobs be affected ? Should the GPUs be idle when I make the change ?