The link you gave in #3 results in a “page not found”. I assume a devbox is some pre-configured machine NVIDIA made available to certain developers, and that that was some while ago (since the link is dead). Although if the machine contains a Titan X it can’t be that old.
In my experience, there are no false positives for that particular error message from nvidia-smi. I cannot give you a 100% guarantee that an incorrect error message could not result from an improper software upgrade. Maybe an incorrect error message could be caused by installing the wrong driver package, or invoking an outdated nvidia-smi executable with a new driver package.
Does NVIDIA provide guidance on how to go about upgrading the system software for these pre-configured machines? Does NVIDIA support user-upgrades to devboxes?
Was the physical machine transported recently? Vibrations during transport can have a negative impact on connectors of all kinds.
It is also possible (though very unlikely) that damage to the PSU occurred while power cycling the system, e.g. because the machine was not properly shut down before external power was removed.
If this were my machine, I would systematically exclude potential sources of error, starting with the hardware configuration. Then proceed to checking the software configuration. E.g. is the installed driver package sufficient for the installed CUDA version; are drivers and CUDA version listed as supported for this version of Ubuntu; are paths to executables and libraries correct and consistent (avoiding a mix of of newly installed and previously installed software).
It seems you may be referring to a “DIGITS devbox”? NVIDIA seems to have a bunch up software updates available for that, did you make use of those? https://docs.nvidia.com/deeplearning/digits-devbox-sw-release-notes/index.html. If NVIDIA sells (or sold) these machines directly to end users, I would expect there to be a dedicated support channel for them. Have you checked for relevant information? These CUDA forums are more of a “users helping users” platform, not an official contact point for NVIDIA support. I wonder whether these boxes might rely on a customized OS image, resulting in issues when a generic Ubuntu image is used.