Do you have up to date drivers (get version from nvidia-smi)?
Are you running on an supported GPU (pascal or higher for the Next-Gen debugger)?
Do you have multiple GPUs (if so, is CUDA_VISIBLE_DEVICES set to the correct GPU)?
What version of Nsight VSE are you using?
Does the program run properly without the debugger (or with the Legacy debugger if on a Kepler or Maxwell GPU)?
What version of win10 are you running (must be Redstone 3 or later for Next-Gen debugger)?
Although, there a mechanisms in place to clean up or run multiple instances, it is possible that an orphaned NvDebugAgent.exe is not terminated. Search for this in the Task Manager and manually kill it if you find it while the debugger is not running.