Only 5 of 6 Titan X GPUs detected by NVidia Driver 352.41 using Ubuntu Server 15.04

The title is a pretty decent summary, basically when I run lspci -v | grep nvidia I get 5 GPUs returned instead of 6.

It says “Kernel Driver in use : nvidia” 5 times.

Running the Cuda 7 deviceQuery returns 5 devices also (as expected at this point)

However if I “ls /dev/nvidia*” I can see that all 6 cards are in fact present and there is no issue. All cards have been tested individually and found working also.

Anything I’m doing wrong? Everything is updated as of 5 minutes ago to the latest versions, the 5 cards are running benchmarks at expected or above expected rates - just the 6th card that is currently an issue.

Kind Regards,
Nick
nvidia-bug-report.log.gz (142 KB)

To add to this, I’ve done more testing, the PCI-e ports are listed 1-6, and are running 1x adapters with their speed limited to Gen 1 for stability.

If I run any 4 cards, it’s fine, if I run 5 it’s fine as long as the card removed is port 4 or 6 (otherwise I lose the card on 6), if I run 6 I simply lose it.