You don’t need to uninstall the Xserver, you only has to stop the server if you try to install the nvidia .run file from nvidia. But it is easier to use the Ubuntu nvidia package.
Can you run
sudo apt-get update
sudo apt-get install nvidia-340
and post the output if something went wrong?