After some time newly launched app instance begins to produce invalid data w/o reporting any errors at kernel launches.
The single way to repair broken GPU/runtime I know is to reboot whole host.
Maybe it’s possible to reset GPU/CUDA runtime/driver w/o rebooting?
And why garbage leaks into app memory buffers w/o any errors reported by CUDA runtime?..
Usually this happen when you have a memory bug in your program that is not significant enough to crash the card. An out of bounds memory access usually will cause this behaviour for me. I would suggest trying to fix any memory errors in your program as these might segfault or worse on another gpu.
are you running the latest 190.xx? there were some ctrl-c/app crash issues that were fixed in that release