Just FYI, this isn’t specific to a Jetson or Ubuntu. What you are asking about is the nature of the X server and how any software talks to the GPU.
Most GPU-related commands (including CUDA) are associated to the GPU through the X server. The X server is not a desktop and is not graphical software the way most people think…it is an API for talking to a a specialized buffer. Windowing and most graphics effects are actually via a desktop manager which runs as an argument to X (CUDA doesn’t need a window manager). When you log in locally through the GUI you create the environment variable “DISPLAY”. When you log in from outside of X no such environment variable is created.
“DISPLAY” associates an instance of X to your software (the window manager and shells running from a window manager inherit this DISPLAY setting). X then acts as the glue between that software and the GPU. No DISPLAY, no GPU association. Running without DISPLAY being set is the same as running without a server, which in turn is the same as missing major components to what you are doing (including the GPU). If “DISPLAY=:0”, then this is the first instance of the X server…if “DISPLAY=:1”, then this is the second instance of an X server…if “DISPLAY=:2”, then this is the third instance of an X server…so on. The numbering doesn’t have to actually start at “:0” and go up in order, but it is tradition (one could manually start an X session and tell it to use “DISPLAY=:42” even if it is the only instance). If you see something like a virtual desktop environment they usually start higher up, e.g., “:10”.
Remote display where X events are forwarded does not do what you want. If you were to use “ssh -Y” or “ssh -X” to remote connect, then X events would forward to your host PC…and the GPU associated and the CUDA libraries would have to exist on the host PC instead. X events are eventually interpreted as something for the GPU to execute. Manually setting “export DISPLAY=:0” from any non-GUI login to the same physical machine as the GPU is quite different than using forwarding (you can log in via ssh without special arguments and then export DISPLAY…and this is the same as logging in at a text console local to the GPU and setting DISPLAY). You need an X server running on the Jetson, and you need the DISPLAY environment variable set to the X server which associates to that GPU.
If you don’t want to attach a monitor, then you need a virtual X server. If you don’t log in locally (such as with ssh), then you need either a logged in GUI session on the Jetson or a virtual X server session to associate with via the DISPLAY variable. A virtual X server does everything a regular X server does, but does not require a monitor to be attached. Virtual desktops also usually offer a remote display function (the remote protocols are not X forwarding).
Examples of virtual desktop servers are VirtualGL and TurboVNC. I can’t give any recommendations on what to use if you don’t want to attach a monitor to the Jetson, but other people may have advice.