Not all graphic cards can be detected by nvidia-smi

I got 4 graphic cards, GeForce RTX 2080 ti in Ubuntu 18.04. All of them can be detected by lshw, but only 3 of them can be detected by nvidia-smi

Please run nvidia-bug-report.sh as root and attach the resulting nvidia-bug-report.log.gz file to your post.

posted!
nvidia-bug-report.log.gz (1.7 MB)

Might be broken

[    8.330028] NVRM: GPU 0000:67:00.0: RmInitAdapter failed! (0x26:0xffff:1290)
[    8.330054] NVRM: GPU 0000:67:00.0: rm_init_adapter failed, device minor number 2

The GPU fails to initialize, please check for properly connected power connectors, reseat card in its slot, check in another system for a general hardware failure.

Your are getting a lot of this error:

ar 14 15:43:35 ChengLabLambda kernel: [162742.663898] NVRM: GPU 0000:67:00.0: RmInitAdapter failed! (0x24:0x65:1248)
Mar 14 15:43:35 ChengLabLambda kernel: [162742.663937] NVRM: GPU 0000:67:00.0: rm_init_adapter failed, device minor number 2

The driver cannot initialize the adapter.
Could be hardware failure.
Try to re-seat that gpu and check power cabling.
If that won’t work, try switching with another card in a working slot, to check if the faulty card is also misbehaving in another slot.