Same problem here. Xubuntu 14.04.1, GeForce GTX 780 Ti, driver 331.38 (the one packaged by Ubuntu). I installed the machine this morning. The graphics work fine when booted natively, but when I reboot with Xen and Linux as Dom0 it fails. As soon as the X server starts, the screen goes blank and none of the text consoles are accessible. It reports the same out-of-memory error message:
...
[ 21.757] (II) NVIDIA: Using 3072.00 MB of virtual memory for indirect memory
[ 21.757] (II) NVIDIA: access.
[ 22.155] (II) NVIDIA(0): Setting mode "DFP-0:nvidia-auto-select,DFP-3:nvidia-auto-select,DFP-4:nvidia-auto-select"
[ 22.311] (EE) NVIDIA(0): Failed to allocate sofware rendering cache surface: out of
[ 22.311] (EE) NVIDIA(0): memory.
[ 22.311] (EE) NVIDIA(0): *** Aborting ***
[ 22.393] (EE)
Fatal server error:
[ 22.393] (EE) AddScreen/ScreenInit failed for driver 0
I was able to compare this to a log from a successful run when booted without Xen:
...
[ 2009.901] (II) NVIDIA: Using 3072.00 MB of virtual memory for indirect memory
[ 2009.901] (II) NVIDIA: access.
[ 2010.292] (II) NVIDIA(0): Setting mode "DFP-0:nvidia-auto-select,DFP-3:nvidia-auto-select,DFP-4:nvidia-auto-select"
[ 2010.458] Loading extension NV-GLX
[ 2010.550] (==) NVIDIA(0): Disabling shared memory pixmaps
[ 2010.550] (==) NVIDIA(0): Backing store enabled
[ 2010.550] (==) NVIDIA(0): Silken mouse enabled
[ 2010.550] (==) NVIDIA(0): DPMS enabled
...
The logs are otherwise basically identical up to that point. Dom0 throws an error while the native boot instead loads NV-GLX and continues setting things up.