We have a server in our research center with 4 GeForce GTX TITAN X (running on ubuntu 16.04, Driver Version: 418.39, CUDA Version: 10.1) and from the past few weeks all processes running on GPU started to be automatically killed by the system, after few seconds they start, without any reason (even simple models such as the mnist code sample provided with keras: https://keras.io/examples/mnist_cnn/). There is a lot of available memory and resources, as the server is practically not in use because of this (so this may not be a problem of high memory usage). After some research, I found and ran the following command “sudo journalctl -xb” to check some logs, and it seems the system is continuously reporting the following message:
– Unit nvidia-persistenced.service has begun shutting down.
Jul 17 11:01:34 cudahpc02 nvidia-persistenced[11027]: Socket closed.
Jul 17 11:01:34 cudahpc02 nvidia-persistenced[11027]: PID file unlocked.
Jul 17 11:01:34 cudahpc02 nvidia-persistenced[11027]: PID file closed.
Jul 17 11:01:34 cudahpc02 nvidia-persistenced[11027]: Shutdown (11027)
Jul 17 11:01:34 cudahpc02 systemd[1]: Stopped NVIDIA Persistence Daemon.
– Subject: Unit nvidia-persistenced.service has finished shutting down
– Defined-By: systemd
Everything was working perfectly few weeks ago… Could this be a problem related to driver (anyway, we didn’t change the driver)? Any idea about how we could fix this?
Thanks in advance and best regards