Every CUDA installation requires a “graphics driver” to be installed as well.
You can run CUDA and graphics simultaneously, although you may notice some limitations around the running time of CUDA kernels, for example.
CUDA 6 requires an R334 driver, so make sure you are either using the graphics driver that comes with the CUDA 6 toolkit or an R334 (or newer branch) driver.
During the installation of a graphics driver, normally you cannot have X running on that GPU, so it may be necessary to shut down X while you are installing the graphics driver. And be sure to remove the nouveau driver if it is present on your system