Getting CUDA to run with ATI primary graphics

I’m working on getting this application to run on CentOS 5.1 (RHEL 5.1 equivalent) and I’ve installed the drivers and the CUDA toolkit (with appropriate linkers).

At the moment, I’m using an onboard ATI graphics controller with 6 GeForce 8800 GPUs. I’ve found that Linux sees all 6 GPUs as PCI devices, but the problem I’m having is making them CUDA available.

I’ve decided to compile and run the multiGPU application and I get the following error:

NVIDIA: could not open the device file /dev/nvidia2 (Input/output error).

There is no device supporting CUDA.

It seems odd to me that it does this with /dev/nvidia2.

Here is the output of ‘ls -al /dev | grep nv’

[cheier@localhost release]$ ls -al /dev | grep nv

crw-rw-rw-  1 root root 195,    0 Dec 12 20:12 nvidia0

crw-rw-rw-  1 root root 195,    1 Dec 12 20:12 nvidia1

crw-rw-rw-  1 root root 195,    2 Dec 12 20:12 nvidia2

crw-rw-rw-  1 root root 195,    3 Dec 12 21:02 nvidia3

crw-rw-rw-  1 root root 195,    4 Dec 12 21:02 nvidia4

crw-rw-rw-  1 root root 195,    5 Dec 12 21:02 nvidia5

crw-rw-rw-  1 root root 195,  255 Dec 12 20:12 nvidiactl

crw-rw----  1 root root  10,  144 Dec 12 20:09 nvram

Also, this is the contents of my xorg.conf file as well:

# nvidia-xconfig: X configuration file generated by nvidia-xconfig

# nvidia-xconfig:  version 1.0  (buildmeister@builder3)  Wed Nov 14 17:10:54 PST 2007

# Xorg configuration created by system-config-display

Section "ServerLayout"

    Identifier     "single head configuration"

    Screen      0  "Screen0" 0 0

    InputDevice    "Mouse0" "CorePointer"

    InputDevice    "Keyboard0" "CoreKeyboard"

EndSection

Section "Files"

EndSection

Section "Module"

    Load           "glx"

EndSection

Section "InputDevice"

   # generated from default

    Identifier     "Mouse0"

    Driver         "mouse"

    Option         "Protocol" "auto"

    Option         "Device" "/dev/input/mice"

    Option         "Emulate3Buttons" "no"

    Option         "ZAxisMapping" "4 5"

EndSection

Section "InputDevice"

    Identifier     "Keyboard0"

    Driver         "kbd"

    Option         "XkbModel" "pc105"

    Option         "XkbLayout" "us"

EndSection

Section "Monitor"

### Comment all HorizSync and VertSync values to use DDC:

    Identifier     "Monitor0"

    ModelName      "LCD Panel 1680x1050"

    HorizSync       31.5 - 90.0

    VertRefresh     59.9 - 60.1

    Option         "dpms"

EndSection

Section "Device"

    Identifier     "Videocard0"

    Driver         "radeon"

EndSection

Section "Device"

    Identifier "Videocard1"

    Driver "nvidia"

    VendorName "NVIDIA Corporation"

    BoardName "GeForce 8800 GTS"

    BusID "PCI:10:0:0"

    Option "UseDisplayDevice" "none"

EndSection

Section "Device"

    Identifier "Videocard2"

    Driver "nvidia"

    VendorName "NVIDIA Corporation"

    BoardName "GeForce 8800 GTS"

    BusID "PCI:12:0:0"

    Option "UseDisplayDevice" "none"

EndSection

Section "Device"

    Identifier "Videocard3"

    Driver "nvidia"

    VendorName "NVIDIA Corporation"

    BoardName "GeForce 8800 GTS"

    BusID "PCI:13:0:0"

    Option "UseDisplayDevice" "none"

EndSection

Section "Device"

    Identifier "Videocard4"

    Driver "nvidia"

    VendorName "NVIDIA Corporation"

    BoardName "GeForce 8800 GTS"

    BusID "PCI:22:0:0"

    Option "UseDisplayDevice" "none"

EndSection

Section "Device"

    Identifier "Videocard5"

    Driver "nvidia"

    VendorName "NVIDIA Corporation"

    BoardName "GeForce 8800 GTS"

    BusID "PCI:24:0:0"

    Option "UseDisplayDevice" "none"

EndSection

Section "Device"

    Identifier "Videocard6"

    Driver "nvidia"

    VendorName "NVIDIA Corporation"

    BoardName "GeForce 8800 GTS"

    BusID "PCI:25:0:0"

    Option "UseDisplayDevice" "none"

EndSection

Section "Screen"

    Identifier     "Screen0"

    Device         "Videocard0"

    Monitor        "Monitor0"

    DefaultDepth    24

    SubSection     "Display"

        Viewport    0 0

        Depth       24

    EndSubSection

EndSection

I’m at a bit of a loss as to what to do at this point. Any direction in the matter would be helpful, since I’ve now exhausted all I can think of. I just can’t run any CUDA applications at this point.

Are you working on a 32bit or 64bit system?

Try to use this script to load the nvidia kernel module and make the proper /dev entries instead of using X:

#!/bin/bash

modprobe nvidia

if [ "$?" -eq 0 ]; then

 # Count the number of NVIDIA controllers found.

  N3D=`/sbin/lspci | grep -i NVIDIA | grep "3D controller" | wc -l`

  NVGA=`/sbin/lspci | grep -i NVIDIA | grep "VGA compatible controller" | wc -l`

 N=`expr $N3D + $NVGA - 1`

  for i in `seq 0 $N`; do

    mknod -m 666 /dev/nvidia$i c 195 $i;

  done

 mknod -m 666 /dev/nvidiactl c 195 255

else

  exit 1

fi

You may need to pass some kernel options at boot time, something similar to this

/boot/grub/grub.conf:

title Red Hat 

root (hd0,0)

uppermem 524288

kernel /vmlinuz-2.6.9-42.ELsmp ro root=LABEL=/1 rhgb quiet vmalloc=256MB

pci=nommconf

initrd /initrd-2.6.9-42.ELsmp.img

The system is a 64bit capable system but I’m currently using the 32bit Linux and drivers. I had some issues going with the 64bit Linux, but I think the BIOS update I did to get the 32bit Linux working might pass over to the 64bit version.

I’ll test the script before I head to bed tonight and see how it goes.

Thanks.

EDIT: Also, should I keep my xorg.conf as is or just comment out the additional devices?

If it is a 32bit, I would try the boot options before using the script.

What is the output of “lsmod |grep nv”?

[root@localhost ~]# lsmod | grep nv

nvidia               7866400  0 

i2c_core               23745  2 nvidia,i2c_i801

EDIT:

Also just a shot in the dark, I know that Windows automatically allocates system memory to each GPU, does the vmalloc essentially do the same thing, but explicitly?

Looks like a combination of the script and the grub.conf file was necessary to getting it working. I just did a modification to the multiGPU app and set the max threads to 8 instead of 4 and got an output that shows 6 GPUs in the system, so it seems to be working. Now I just need to test to make sure I’m getting the performance I should be getting given the systems specifications.

Alright, now now that I have CUDA operating properly, how do I enable the GeForce as a primary display device with stability? I had to run VMD in a text mode to test the CUDA functionality, which worked beautifully, but I can’t run it normally because I can’t get an OpenGL context (the ATI chip doesn’t support 3D acceleration). How would I go about getting one of the GeForce cards to be a primary display device?

What do you mean by “primary display device”? Also, please generate and attach an nvidia-bug-report.log.

Primary display device in this case would be the device that is primarily being used for graphics display, which at the moment is the ATI chip. This is because I couldn’t get Linux installed using a secondary display device (the NVIDIA GPUs) because the installer would lock up almost immediately after the CentOS CD was booted.

How do I generate an nvidia-bug-report.log file?

Hey Chris,
Sounds like you’d now like to disable the integrated graphics so you can get good 3d display? There may be a bios setting to disable it and use the PCIe card instead.

ah yes… that seems to work… it’s always the simple things a person will tend to overlook… :D