Cloud instance with nVidia driver and desktop manager

I have access to a server with a GPU via an Ubuntu VM.

I would like to do some CUDA computations (rendering with Blender) on that server thus taking advantage of the GPU.

However, for installing the Nvidia toolkit some dependencies on the Nvidia drivers exist. When I do install everything, I am not able to connect to the machine (the machine is running Ubuntu 16.04) when the machine is running gnome or unity as desktop environments (it immediately logs me out after logging me in).

The problem does not appear when I use xfce4 as the desktop environment. In this case, I can render using the GPU. Is there any known issue/limitation I am hitting here or is there a solution to this problem so I can use a more convenient desktop manager for me (preferable gnome) than xfce4?

Thanks a lot!