Spark: cudaErrorNotPermitted in comfyui - but only after Docker sits idle for hours

[DannyTup]

This sounds a lot like the issue I reported here:

"Failed to initialize NVML: Unknown Error" running nvidia-smi in a docker container only after some hours/days

I created my own dashboard which shows some metrics and docker containers on my Spark. The full source is at GitHub - DanTup/dgx_dashboard: A simple dashboard for the DGX Spark. but it basically runs: nvidia-smi --query-gpu=utilization.gpu,temperature.gpu,power.draw --format=csv,noheader,nounits -l=5 To get metrics streamed every 5s. It starts the process only when there is a browser connected to the backend, and then terminates the process after a timeout when there are no connections. I ran…

It appears that whenever something triggers systemctl daemon-reload the GPUs disappear from the containers.

This is documented along with a workaround here:

--gpus - Troubleshooting — NVIDIA Container Toolkit

That page suggests that the issue has been resolved, however it definitely hadn’t for me. The workaround (use cgroupfs as the cgroup driver for containers) has solved the issue though.