Unable to enable CUDA - Double Precision on GeForce GTX Titan Black Ubuntu 16.04

Hello,

My configuration:
Ubuntu 16.04
Xeon E5 2680 v2 (2.8 GHz 10 core)
ASRock mobo EPC602D8A
32 GB DDR3 1866 MHz
Tesla K20
GeForce GTX Titan Black
CUDA 8.0 GA2
Nvidia 384.11 driver

I am trying to use the GeForce Titan Black for double precision compute purposes (SuiteSparse) since it is capable of 1.7 TFLOPS. However, when I run ./nbody -benchmark -fp64 (from cuda/samples), I see 1/10 of that performance:

Windowed mode
Simulation data stored in video memory
Double precision floating point simulation
1 Devices used for simulation
gpuDeviceInit() CUDA Device [1]: "GeForce GTX TITAN
Compute 3.5 CUDA device: [GeForce GTX TITAN]
12288 bodies, total time for 10 iterations: 326.616 ms
= 4.623 billion interactions per second
= 138.690 double-precision GFLOP/s at 30 flops per interaction

My Tesla K20 is capable of 1.1 TFLOPS and at manages about 7/10s of that:

Windowed mode
Simulation data stored in video memory
Double precision floating point simulation
1 Devices used for simulation
gpuDeviceInit() CUDA Device [0]: "Tesla K20c
Compute 3.5 CUDA device: [Tesla K20c]
13312 bodies, total time for 10 iterations: 69.488 ms
= 25.502 billion interactions per second
= 765.067 double-precision GFLOP/s at 30 flops per interaction

I spent time researching the activation of double precision performance on the Titan Black, they recommended I use:

sudo nvidia-settings

And check the box that says “CUDA - Double precision”, in the GeForce GTX Titan tab.

However, when I check the box, it fails to set double precision - it reads out:

“Failed to set CUDA - Double precision configuration!”

I also tried to set the double precision attribute via terminal with:

sudo nvidia-settings -a [gpu:1]/GPUDoublePrecisionBoostImmediate=1

But the benchmark results posted above for the Titan Black remain the same. That same thread (Turning on DP for Nvidia Titan on Headless Server? - Linux - NVIDIA Developer Forums) mentions the source code for nvidia-settings, so I tried to track this error message down in the ctkpowermizer.c, but I was unable to find the true origin of the failure:

Does anyone have any idea why I am unable to set the Double Precision Boost for my Titan Black?

Cheers,

Robert

I succeeded getting Nvidia Titan, Titan Black, and Titan Z doing double precision FP64 computations working for Milkyway@home (1/3 not 1/24) using Ubuntu 16.04 on Intel i7-4790K CPU’s (4 cores - 8 threads, 4 GHz, 8 GB RAM) by taking the following steps:

  1. Install Ubuntu 16.04 clean disk reformat
    a. Select updates at install time.
    b. Select include proprietary additions at install time.
    c. Install the Nvidia GPU hardware (if not already installed)
  2. Use Ubuntu 16.04 to install the following:
    a. Install Ubuntu App “Synaptic”
    a.1 Use Synaptic to “Apply” and install all dependent Boinc software (Client & Manager)
    a.2 Use Synaptic to “Apply” and install all dependent Nvidia software
    a.3 Use Boinc Manager to add grid project such as Milkyway@home. It requires FP64, not FP32.
    a.4 Run Milkyway@home in Boinc which will create directories to add app_config.xml later.
    b. User an editor to create and save the following text file named app_config.xml.
    <app_config>

    milkyway
    <gpu_versions>
    <gpu_usage>0.20</gpu_usage>
    <cpu_usage>0.12</cpu_usage>
    </gpu_versions>

    </app_config>
    c. Install Ubuntu App “Nvidia X Server” (File: nvidia-settings)
    d. Install XFE as follows because it is required for installing app_config.xml later as root admin.
    d.1 Open a new Terminal window
    d.2 Type in console window: sudo apt-get update
    d.3 Type in console window: sudo apt-get install xfe
    d.4 Type in console window: sudo xfe (this opens the xfe file explorer as Root - Required!)
  3. Use XFE to copy and paste the file app_config.xml (Incidentally, this will cause the GPU to do fp64).
    a. XFE is open as root from d.4 above, otherwise you will not be able to modify the destination folder.
    b. Use XFE to browse to and “copy” the app_config.xml file created in 2-b above.
    c. Use XFE to browse to and “paste” the app_config.xml file into the destination folder in this example:
    /var/lib/boinc-client/projects/milkyway.cs.rpi.edu_milkyway/
  4. Stop and restart Boinc by Resetting each computer in order to invoke the new app_config.xml file.
    a. Observe in Boinc Manager the increase in “Active Tasks” is now more than one (1) Milkyway task:
    Titan is now running 4 Milkyway task on the GPU.
    Titan Black is now running 4 Milkyway task on the GPU.
    Titan Z is now running 10 Milkyway task on the GPU.
    b. Confirm double precision is enabled. In the console window type: sudo nvidia-settings
    c. Observe under “Power Mizer” that the Double Precision box is now checked.

It works! NH