The Xorg server ABI is how the GPU is accessed if direct hardware access is used (software framebuffer is a different story). One does not have to run a graphical desktop, nor a login manager, but the Xorg X11 server must exist to use hardware acceleration (X11 takes a window manager or login manager as an argument in a default Linux install, but this could be modified to instead run some other X11-compliant application directly). The Tegra 4 did not have a GPU, but starting at Tegra K1 a GPU exists…so this is when Xorg becomes mandatory.
Btw, running Qt in a framebuffer without a desktop is still running X11…you’ve bypassed the window manager, not X11.