It may be possible that you are selecting two GPUs under two different IOHs.
The order in the driver and the order in the lspci may be different.
nvidia-smi -q should give you the info of the PCI slot for each card.
Could you try to play with different combinations of devices to see if you find a working one?
Before starting the test, set the variable CUDA_VISIBLE_DEVICES.
If you have 4 cards, these should cover all the possible combination.
Apparently the second parameter in this function call: cutilSafeCall(cudaDeviceEnablePeerAccess(gpuid_tesla, gpuid_tesla)) should be 0 and not gpuid_tesla. The error was thrown because I set gpuid_tesla to 1. It’s a bug in the SDK example.