Unable to set up cuda-8.0 on RHEL 7.4

I have the kernel-dev and headers, gcc 4.8 installed on a RHEL 7.4, 3.10 kernel.
I installed cuda-8.0 using rpm local repo. when I run nvidia-smi it gives me the following error

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

Can some one help me? Here are lot of details about my system.

[AzrRootAdminUser@hazr0217b84292 ~]$ lspci | grep -i nvidia
3eb348e1:00:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)
3eb3694f:00:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)

[root@hazr0217b84292 AzrRootAdminUser]# uname -r
3.10.0-693.11.6.el7.x86_64

[root@hazr0217b84292 AzrRootAdminUser]# cat /etc/*-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.4 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.4"
PRETTY_NAME="Red Hat Enterprise Linux"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.4:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.4
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.4"
Red Hat Enterprise Linux Server release 7.4 (Maipo)
Red Hat Enterprise Linux Server release 7.4 (Maipo)

[root@hazr0217b84292 AzrRootAdminUser]# yum list installed | grep -E 'nvidia|cuda'
cuda.x86_64                    8.0.61-1               @cuda-8-0-local-ga2       
cuda-8-0.x86_64                8.0.61-1               @cuda-8-0-local-ga2       
cuda-command-line-tools-8-0.x86_64
                               8.0.61-1               @cuda-8-0-local-ga2       
cuda-core-8-0.x86_64           8.0.61-1               @cuda-8-0-local-ga2       
cuda-cublas-8-0.x86_64         8.0.61-1               @cuda-8-0-local-ga2       
cuda-cublas-dev-8-0.x86_64     8.0.61-1               @cuda-8-0-local-ga2       
cuda-cudart-8-0.x86_64         8.0.61-1               @cuda-8-0-local-ga2       
cuda-cudart-dev-8-0.x86_64     8.0.61-1               @cuda-8-0-local-ga2       
cuda-cufft-8-0.x86_64          8.0.61-1               @cuda-8-0-local-ga2       
cuda-cufft-dev-8-0.x86_64      8.0.61-1               @cuda-8-0-local-ga2       
cuda-curand-8-0.x86_64         8.0.61-1               @cuda-8-0-local-ga2       
cuda-curand-dev-8-0.x86_64     8.0.61-1               @cuda-8-0-local-ga2       
cuda-cusolver-8-0.x86_64       8.0.61-1               @cuda-8-0-local-ga2       
cuda-cusolver-dev-8-0.x86_64   8.0.61-1               @cuda-8-0-local-ga2       
cuda-cusparse-8-0.x86_64       8.0.61-1               @cuda-8-0-local-ga2       
cuda-cusparse-dev-8-0.x86_64   8.0.61-1               @cuda-8-0-local-ga2       
cuda-demo-suite-8-0.x86_64     8.0.61-1               @cuda-8-0-local-ga2       
cuda-documentation-8-0.x86_64  8.0.61-1               @cuda-8-0-local-ga2       
cuda-driver-dev-8-0.x86_64     8.0.61-1               @cuda-8-0-local-ga2       
cuda-drivers.x86_64            375.26-1               @cuda-8-0-local-ga2       
cuda-license-8-0.x86_64        8.0.61-1               @cuda-8-0-local-ga2       
cuda-misc-headers-8-0.x86_64   8.0.61-1               @cuda-8-0-local-ga2       
cuda-npp-8-0.x86_64            8.0.61-1               @cuda-8-0-local-ga2       
cuda-npp-dev-8-0.x86_64        8.0.61-1               @cuda-8-0-local-ga2       
cuda-nvgraph-8-0.x86_64        8.0.61-1               @cuda-8-0-local-ga2       
cuda-nvgraph-dev-8-0.x86_64    8.0.61-1               @cuda-8-0-local-ga2       
cuda-nvml-dev-8-0.x86_64       8.0.61-1               @cuda-8-0-local-ga2       
cuda-nvrtc-8-0.x86_64          8.0.61-1               @cuda-8-0-local-ga2       
cuda-nvrtc-dev-8-0.x86_64      8.0.61-1               @cuda-8-0-local-ga2       
cuda-repo-rhel7-8-0-local-ga2.x86_64
cuda-runtime-8-0.x86_64        8.0.61-1               @cuda-8-0-local-ga2       
cuda-samples-8-0.x86_64        8.0.61-1               @cuda-8-0-local-ga2       
cuda-toolkit-8-0.x86_64        8.0.61-1               @cuda-8-0-local-ga2       
cuda-visual-tools-8-0.x86_64   8.0.61-1               @cuda-8-0-local-ga2       
nvidia-kmod.x86_64             1:375.26-2.el7         @cuda-8-0-local-ga2       
xorg-x11-drv-nvidia.x86_64     1:375.26-1.el7         @cuda-8-0-local-ga2       
xorg-x11-drv-nvidia-devel.x86_64
                               1:375.26-1.el7         @cuda-8-0-local-ga2       
xorg-x11-drv-nvidia-gl.x86_64  1:375.26-1.el7         @cuda-8-0-local-ga2       
xorg-x11-drv-nvidia-libs.x86_64
                               1:375.26-1.el7         @cuda-8-0-local-ga2       

[root@hazr0217b84292 AzrRootAdminUser]# gcc --version
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[root@hazr0217b84292 AzrRootAdminUser]# yum list installed|grep 'kernel'
kernel.x86_64                  3.10.0-693.5.2.el7     @rhel-7-server-rpms       
kernel.x86_64                  3.10.0-693.11.6.el7    @rhel-7-server-rpms       
kernel-devel.x86_64            3.10.0-693.5.2.el7     @rhel-7-server-rpms       
kernel-devel.x86_64            3.10.0-693.11.6.el7    @rhel-7-server-rpms       
kernel-headers.x86_64          3.10.0-693.11.6.el7    @rhel-7-server-rpms       
kernel-tools.x86_64            3.10.0-693.11.6.el7    @rhel-7-server-rpms       
kernel-tools-libs.x86_64       3.10.0-693.11.6.el7    @rhel-7-server-rpms 

[root@hazr0217b84292 AzrRootAdminUser]# nvidia-smi 
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
      
[root@hazr0217b84292 AzrRootAdminUser]#

you might just need to reboot.

if that doesn’t work, provide the output of:

dmesg |grep NVRM

reboot did not work.
dmesg |grep NVRM comes empty.

then you have no driver actually loaded.

If you have a NVIDIA driver loaded, it will put at least one message in the system log with NVRM in it that says it is starting up the GPU.

So your driver install did not work. There may be a number of reasons for this, especially if you had any previous NVIDIA GPU drivers loaded on the machine. We would need to see the exact output from your rpm install to diagnose.

I checked installation log, it points to a failure in

/var/lib/dkms/nvidia/375.26/build/make.log

.
Here I see

DKMS make.log for nvidia-375.26 for kernel 3.10.0-693.11.6.el7.x86_64 (x86_64)
...
compilation terminated.
In file included from /var/lib/dkms/nvidia/375.26/build/nvidia-drm/nvidia-drm-encoder.c:27:0:
/var/lib/dkms/nvidia/375.26/build/nvidia-drm/nvidia-drm-priv.h:37:25: fatal error: linux/fence.h: No such file or directory
 #include <linux/fence.h>

Try installing the latest driver (instead of 375.26) for K80 separately. After that install CUDA. Using the runfile installer, deselect the option to install the driver. Using the package manager method, install the cuda-toolkit package instead of cuda. Refer to the linux install guide for more details.