Also consider that on the desktop the GPU has its own RAM, and is not shared with the rest of the world (but of course running a CPU-only version does not use much GPU RAM :P ).
Someone else will probably be able to suggest how to reserve RAM (at boot) for the GPU. I’m pretty sure it can be done with nothing more than kernel command line arguments, but I do not know the specifics.