Viewer GUI with virtual display

Since I don’t own a local desktop machine, I rely on remote cluster for GPUs. The only way is headless docker + ssh. I have installed virtual display and can access the GUI via VNC. However, I’m unable to use IsaacGym’s Viewer GUI. It keeps getting segfault. Otherwise, I’m able to train with --headless mode on. I built my image upon the Dockerfile provided in Preview 2. The setup is RTX 2080Ti and Ubuntu 20.04.

Does IsaacGym require a physical display, or is there any workaround for virtual display? I can get the other simulator GUIs work in docker, like Mujoco, so the virtual display should be working correctly. Thank you so much for any help!

1 Like

same seg fault when using X11 forwarding

1 Like

Hi @JimJarvis @zipengfu

This post may be helpful

1 Like

Thanks for the link! I just replied in that post as well - I followed the instructions but didn’t have any success. It seems extremely difficult to configure everything correctly. It’d be great if IsaacGym team can provide a Dockerfile that supports remote Viewer GUI.

Hi @JimJarvis,

Unfortunately, the Docker configuration doesn’t allow remote rendering of the viewer - it only supports rendering to offscreen buffers for the training side (see python/examples/graphics.py).

To get remote access for the viewer, you would have to log into a machine that had some local graphics rendering capacity, hence the RealVNC suggestion.

Take care,
-Gav

2 Likes

Hi Gav,

Thank you so much for your timely reply! My remote servers do not have physical monitors attached. I have to use Docker to render the virtual display. Are you suggesting that it’s impossible? Would it become possible with newer IsaacGym releases in the future?

Hi @JimJarvis,

I’m afraid that’s correct as far as I know, unless you are somehow able to get a full remote display set up, running vulkan and a VNC client in your container.

There are some things you can do to support faking a physical monitor so that you can directly run desktop X11 and VNC into your box (CustomEDID and such), but it’s not for the faint of heart.

Another alternative is to simply train in your docker environment, and then periodically test the output network on a local machine. We do this all the time internally.

Take care,
-Gav

Hi Gav,

Thanks for your reply! I have actually attempted to fake a physical monitor to run a full desktop, but couldn’t get it to work - there are simply too many obscure errors here and there.

The workflow you mentioned makes sense. However, my local machine is a Macbook Pro with no Nvidia GPU support. Is there any other way to see the View GUI? In your experience, does remote VirtualBox (or other full VMs) work?

Hi @JimJarvis,

Yeah, it can definitely be a challenge to get the fake physical monitor setup going. Some cards also may not support it.

I’m afraid that beyond getting access to a machine that can run Vulkan locally with an NVIDIA GPU to test with, I don’t have any other good suggestions. You may be able to get something going via a cloud provider.

We do have a streaming option available from inside a docker container with Omniverse and Isaac Sim, and once we have Gym features fully integrated there that could be an option for you, but we aren’t ready yet to announce when that will be available.

Take care,
-Gav

1 Like

These are great to know. You’ve been extremely helpful. Thanks again Gav!

No problem. For a cloud approach I found this with a quick search, but can’t directly vouch for whether it would work:

So long as Vulkan is runnable you should be able to get rendering output. You would probably want to use a more recent GPU - T4, for example.

Take care,
-Gav