Reset dedicated GPU after it gets stuck

I have a setup with a dedicated NVIDIA for CUDA computing and another NVIDIA for display. From time to time, the CUDA dedicated NVIDIA gets stuck - the fan keeps running at 100% and nothing can be really done with it anymore. I can “solve” it by rebooting but I’d much prefer to solve it by resetting the CUDA dedicated card.

Here is the nvidia-smi output when the issue happens (the CUDA card is #0 but #1 is set as primary):

±----------------------------------------------------------------------------+
| NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version: 11.6 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce … Off | 00000000:41:00.0 Off | N/A |
|ERR! 41C P2 ERR! / 250W | 2MiB / 11264MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
| 1 NVIDIA GeForce … Off | 00000000:42:00.0 On | N/A |
| 40% 47C P8 10W / 75W | 479MiB / 4096MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 1 N/A N/A 4517 G /usr/libexec/Xorg 199MiB |
| 1 N/A N/A 4861 G /usr/bin/kwin_x11 1MiB |
| 1 N/A N/A 5515 G …akonadi_archivemail_agent 1MiB |
| 1 N/A N/A 5523 G …/akonadi_mailfilter_agent 17MiB |
| 1 N/A N/A 5526 G …n/akonadi_sendlater_agent 1MiB |
| 1 N/A N/A 5527 G …nadi_unifiedmailbox_agent 1MiB |
| 1 N/A N/A 1923084 G /usr/bin/plasmashell 71MiB |
| 1 N/A N/A 3052928 G …449555690282580582,131072 123MiB |

Thus, no processes are reported as running on the CUDA card. Yet, trying to reset the card returns:

vidia-smi --gpu-reset -i 0
GPU 00000000:41:00.0 is currently in use by another process.

1 device is currently being used by one or more other processes (e.g., Fabric Manager, CUDA application, graphics application such as an X server, or a monitoring application such as another instance of nvidia-smi). Please first kill all processes using this device and all compute applications running in the system.

nvidia-persistenced is not running (so that is not the blocking process)

Is there a way to reset the card without reboot?

I have found that I can reset the card after killing Xorg (actually I used 'systemctl isolate multi-user.target). So, obviously, somehow Xorg still interferes with the dedicated card despite no processes are listed as running on the GPU.

After the Xorg stop and card reset, I am able to use the card as usual. However, I am still hoping there is a solution that would not require Xorg restart - that’s why I have the setup with CUDA dedicated GPU in the first place…

Please check if this helps:
Check whether

sudo cat /sys/module/nvidia_drm/parameters/modeset

Returns ‘Y’, if so, run
run

grep nvidia /etc/modprobe.d/* /lib/modprobe.d/*

to find a file containing

options nvidia-drm modeset=1

and change 1 to 0
then run

sudo update-initramfs -u

and reboot.

sudo cat /sys/module/nvidia_drm/parameters/modeset

should return ‘N’ if done right.
Furthermore, you should monitor gpu temperatures and correctly set up nvidia-persistenced in order to prevent running into the error state.

cat /sys/module/nvidia_drm/parameters/modeset
cat: /sys/module/nvidia_drm/parameters/modeset: No such file or directory

ls /sys/module/ | grep nvidia
nvidia
nvidia_modeset
nvidia_uvm

lsmod | grep nvidia
nvidia_uvm 1191936 0
nvidia_modeset 1163264 23
nvidia 39108608 1215 nvidia_uvm,nvidia_modeset
drm 630784 1 nvidia

find /sys -iname drm
/sys/kernel/btf/drm
/sys/kernel/tracing/events/drm
/sys/kernel/debug/tracing/events/drm
/sys/class/drm
/sys/module/drm

So I guess nvidia-drm is not the culprit.

I did not know that nvidia-persistenced could prevent running into the error state though, I thought the gain was just performance-wise. I have started it, lets see whether it helps indeed.

The issue still happens also if nvidia-persistenced is running. So still no way to either prevent the issue nor to sort it out without Xorg restart.

I have managed to remove all processes but Xorg from accessing the GPU:

  • lsof /dev/nvidia* returns nothing
  • fuser -v /dev/nvidia* returns just Xorg on /dev/nvidia0, /dev/nvidia1, /dev/nvidiactl, /dev/nvidia-modeset
  • nvidia-smi only shows a single process - Xorg running on card 1

Yet, reset it still not possible. So, the problem seems to be that despite nvidia-smi does not show Xorg running on card 0, it is still somehow connected to it.

Is there really no way to properly dedicate an NVIDIA card for CUDA computing? Would I have to stop using at least one NVIDIA card to achieve that? The current setup with two NVIDIA cards is almost unusable as I have to restart almost every day due to this issue.