How to know which pci slot the GPU is located in?

There are 4 GTX295 cards in my computer, which has totally 8 GPUs.
I found third GPU(cuda device id is 2 ) was not stable. But I don’t know which graphic card the bad GPU is located in.
I inversed the sequence of the graphic card in computer, the device id of the bad GPU becomed 4.

How can i find the corresponding graphic card without open the computer?

thank you

I don’t think you can. There is no fixed or guaranteed relationship between CUDA enumeration order and PCI device enumeration, at least as far as I know. nvidia-smi can give GPU serial numbers and CUDA device numbers, that is probably going to be the only possible way to be sure, short of pulling cards until you find the bad one.

There is indeed currently no way to get the relationship between the bus id and the cuda device id, I whined quite a lot about this and was told we should get it at some point in the future … stay tuned :)

In any case, a common solution is to benchmark the bandwith between each NUMA memory bank and the different GPUs, you should very easily detect which GPUs are the closest to the different NUMA nodes.

Cédirc