The problem is that the video driver must be tied into something else which understands its ABI to own the actual rendering context. Anything you build would be a substitute for software such as X or Wayland…more or less re-inventing the wheel. X, when not using any window manager, and when not configured for remote display, is actually not too intrusive.
There was this announcement which talks about EGLstreams, but the primary focus of the announcement was centered on Vulkan:
https://devtalk.nvidia.com/default/topic/925605/linux/nvidia-364-12-release-vulkan-glvnd-drm-kms-and-eglstreams/
Basically it sounds like you’d want to do the equivalent of Vulkan (which would be a substitute for X, and has an implementation useful with the existing Jetson nVidia video driver for hardware acceleration)…this might be the route to go for your use case. I believe the newer L4T releases support Vulkan, and there is likely a method to run this without X, though I couldn’t guarantee that. The steps for setting up just Vulkan instead of X will probably be more trial-and-error since X is already the default and merely requires configuration to not use a window manager or login manager. If I were to personally do this I’d first get it to work with X going directly to your application without window manager or login manager; once that worked, I’d then try replacing X with Vulkan (I’ve never worked on Vulkan, this would be new territory). Half of this is setting up Linux init (simplifying until X runs directly with your app does this), the other half is setting up graphics (substituting Vulkan for X does this). Should you be unable to get Vulkan as the X substitute you want, then you’d still have pure-X as the fallback.