Nvidia-smi really slow to execute

You’ll need to have the persistence daemon (nvidia-persistenced) started on boot and have it continuously running. Otherwise, the driver gets unloaded and the Teslas deinit so when nvidia-smi is run it needs a full reinit. Failing to have the persistenced running may also lead to more serious issue like gpus crashing, depending on workloads.