Help with multi user headless VNC with OpenGL support

Hi I am facing some issues to get openGL acceleration over VNC.

My Goal: Have multi user VNC/SSH Sessions with support to run openGL applications over remote VNC sessions with low latency. (openGL accelerated X server to support local display as well as multi user VNC sessions)

Background Info: The server is having an AMD EPYC 64 core processor with NVIDIA Quadro RTX 4000 and running Ubuntu 20.04. NVIDIA Propietary drivers version 440.82 is installed.

-> lightdm as desktop manager

Here’s what I tried:

  1. VirtualGL+TigerVNC (Works but the openGL application launched keeps on crashing repeatedly)
  2. x11vnc with monitor forwarding (Doesnt work with multiple users)

I did see this nvidia forum: Headless Remote Server Setup (but I have a single GPU)

But some help in this regard with Ubuntu 20.04 would be appreciated