This is expected behavior, because the X server grabs the virtual terminal when it starts so that it can control access to the console.
You can sort of work around that by starting X with the -novtswitch (and maybe -sharevts) option, but that has a tendency to not work well because the X driver intercepts keyboard and mouse events by default. You could probably avoid that problem by carefully crafting xorg.conf to avoid having the server grab the input devices, but I don’t know offhand how to do that.