The design of X is that events are sent to the X server, and rendering or use of the GPU driver is an interpretation of the X events. If you are logged in locally, then the events are automatically bound to the X server you are running in. If you run remotely, without forwarding, then X events will fail since it doesn’t know which server. If you run remotely, with forwarding, then what you are forwarding are the events to your local computer’s X server.
Note that once you forward events to the PC’s X server it is no longer the Jetson’s driver/GPU doing the work. If the host PC is instead doing the work, then that PC must have the right CUDA libraries, the right OpenGL libraries, so on.
The purpose of a virtual X server is to have the interface and bindings locally on a system without actually rendering to a monitor. CUDA and other X applications will happily talk to a virtual X server and won’t care about whether or not there is a monitor attached to that server.
Remote desktop software differs from forwarding software in that remote desktop software does not forward events…remote desktop software instead sends an actual compressed desktop image (changes to the buffer representing the desktop) instead of events being redirected. The ultimate headless Jetson consists of a virtual X server and remote desktop software.
Even if you have a regular user desktop on a Jetson you can still also have a remote desktop software installed…the best of both worlds. Even more interesting is the fact that you can have more than one X server, and so one can have both a virtual X server and an actual local monitor attached via a separate X server. The local physical monitor X server could use remote desktop software to view the virtual X server if desired.
NOTE: In all cases forwarding X events has many security restrictions. Special setups are required if one X session is to interact with a different X server. This is such a common practice under ssh that the “ssh -X” and “ssh -Y” options handle all of this for you. Even so, this is still just event forwarding, and not the GPU buffer being forwarded. Virtual desktop setup does something much like “ssh -X” or “ssh -Y”, but it does so with the results of events and not the actual events.
Think of the X server as an ABI for various events triggering GPU functionality.