Cuda-gdb hanging and locking /dev/nvidia1

I have run into a problem where it appears the system locks /dev/nvidia1 and I can’t launch any cuda applications. This happens with the following sequence of events:

– I am running cuda-gdb
– cuda-gdb hangs while running and the system has to be rebooted (system unresponsive)
– after reboot I get the following error message.

NVIDIA: could not open the device file /dev/nvidia1 (Input/output error).

error: The CUDA driver failed initialization (error=20).

(cuda-gdb)

Any insight you can give would be appreciated. What is the best method to clear the error and allow cuda to use the /dev/nvidia1 driver again?

Thanks.

looking into it more, it looks like the crash changed something fundamental. Only one of my graphics cards shows up in nvidia-settings.

I am wondering if I am missing something in my setup that did not show up before.
I re-installed the 256.53 nvidia driver and that did not help.

my xorg.conf contains

Section “Device”
Identifier “Device0”
Driver “nvidia”
VendorName “NVIDIA Corporation”
EndSection

Section “Device”
Identifier “Device1”
Driver “nvidia”
VendorName “NVIDIA Corporation”
EndSection

looking into it more, it looks like the crash changed something fundamental. Only one of my graphics cards shows up in nvidia-settings.

I am wondering if I am missing something in my setup that did not show up before.
I re-installed the 256.53 nvidia driver and that did not help.

my xorg.conf contains

Section “Device”
Identifier “Device0”
Driver “nvidia”
VendorName “NVIDIA Corporation”
EndSection

Section “Device”
Identifier “Device1”
Driver “nvidia”
VendorName “NVIDIA Corporation”
EndSection