NVIDIA-SMI can not detect GPU-1070 but lspci can see this PCIe deivce. There is no PCIe AER error at all. This is intermittent failures. It initially passed the test, after a couple of weeks, SMI can not see the GPU.
It happened to multiple devices
What does this mean: “It happened to multiple devices”? Multiple different GPUs in the same system, or one GPU tried in multiple different systems? What kind of system(s) are these?
Did you acquire this GPU new (factory fresh, in original packaging) from a reputable dealer? Is it a second-hand item (possibly previously used for crypto-currency mining)?
Are you running Linux? does dmesg (/var/log/syslog) indicate any errors regarding the GPUs? Do they come back after reboot or power cycle or are they gone permanently?
I’ve had a couple 1060s and 1070s die in crypto currency mining. They dropped off the bus and were never seen again. And I wasn’t even overclocking.
Consumer parts are generally not designed for 24/7 load, so that is not unexpected. Typical design parameters for consumer-grade components in PCs would be 20% to 30% duty cycle and life expectancy of 5 years. Under a simplistic view this means we should see many parts fail after 1.0 to 1.5 years of 24/7 load (i.e. 100% duty cycle). The reality is of course much more complicated, but I am lacking the necessary statistical background for such detailed models. I am curious to know (for my in-brain database) how long these particular cards lasted before they failed.
Were these GPUs running at NVIDIA-specified clock rates, or vendor overclocked (e.g. “SC” variants)?