Multiple cards and linux drivers

I’m trying to install four GTX285 cards on a MSI k9a2 motherboard. Allegedly, this has been done before. It seems I can get it to work with two cards (in arbitrary slots) but as soon as I put in three or all four, I get problems with the nvidia drivers. I can still get the X-server up and running but if I try a simple cuda program that enumerates devices, it only finds one, fails to read its properties (cc. 9999.9999) and gives me an error of the kind:

NVIDIA: could not open the device file /dev/nvidia2 (Input/output error).

and similarly from Xorg:

NVIDIA: could not open the device file /dev/nvidia2 (Input/output error).
(EE) Sep 02 02:51:53 NVIDIA(GPU-2): Failed to initialize the NVIDIA graphics device PCI:4:0:0.
(EE) Sep 02 02:51:53 NVIDIA(GPU-2): Please check your system’s kernel log for additional error
(EE) Sep 02 02:51:53 NVIDIA(GPU-2): messages and refer to Chapter 8: Common Problems in the
(EE) Sep 02 02:51:53 NVIDIA(GPU-2): README for additional information.
(EE) Sep 02 02:51:53 NVIDIA(GPU-2): Failed to initialize the NVIDIA graphics device!
NVIDIA: could not open the device file /dev/nvidia3 (Input/output error).
(EE) Sep 02 02:51:53 NVIDIA(GPU-2): Failed to initialize the NVIDIA graphics device PCI:5:0:0.
(EE) Sep 02 02:51:53 NVIDIA(GPU-2): Please check your system’s kernel log for additional error
(EE) Sep 02 02:51:53 NVIDIA(GPU-2): messages and refer to Chapter 8: Common Problems in the
(EE) Sep 02 02:51:53 NVIDIA(GPU-2): README for additional information.
(EE) Sep 02 02:51:53 NVIDIA(GPU-2): Failed to initialize the NVIDIA graphics device!

The readme was unhelpful, disabling acpi and trying related kernel parameters did not affect the situation with the gpus.

If I run nvidia-settings, I can usually see two cards listed. All four devices show up as they should with “lspci”, but there are some messages in the kernel and Xorg log’s that may indicate the problem although I don’t know what to do about it… The logs are here:

http://odinlake.net/kernel.log.txt
http://odinlake.net/Xorg.0.log

I’ll gladly give more output on request. I run Arch but tried briefly with SUSE 11.1 and got the same kind of problem. Tried both the 185.18 and 190.18-Beta drivers with no change. I’d like to point out again that everything works fine with two devices but as soon as more are added, none works with cuda.

Any help I can get would be most appreciated!

(SOLVED) Turns out the driver wants to allocate a big chunk of virtual memory for each device and since my kernels seem to reserve only ~122MB per default, the it runs out after the second device. The solution was to reserve more at boot: Add “vmalloc=256m” to the kernel line in /boot/grub/menu.lst or whatever you use.