How to get camera input when VR is running?

Omniverse Isaac Sim
,
1

I am using Meta Quest 2 for robot teleoperation and I have added a tiled camera in the environment scene. When I try to take input of that camera while running my VR like env.scene[“tiled_camera”].data.output[“rgb”][0], the program gets frozen. Now on the other hand when I turn off my VR, I am getting the tiled camera inputs but obviously not able to use the VR’s inputs for teleoperation. I have arrived to an understanding that this is caused due to a rendering thread getting blocked, but have struggled to understand the exact working of this thread. Please help me understand how to concurrently read camera input as well as get VR inputs for teleoperation.

PS- I need that set of camera inputs for training a VLA model for that robot.

3

Hm, it sounds like you’re blocking the Isaac Sim render thread with the syhcronous read. Can you try using asynchronous reader or callback?

4

Hi, @hking thanks for the reply, tried async read, didn’t seem to work, also found this description in one of the configuration files, can this somehow be bypassed safely?

/isaac-sim/IsaacLab/blob/main/source/isaaclab/isaaclab/devices/openxr/xr_cfg.py#L112-L126

def remove_camera_configs(env_cfg: Any) → Any:
“”"Removes cameras from environments when using XR devices.

XR does not support additional cameras in the environment as they can cause
rendering conflicts and performance issues. This function scans the environment
configuration for camera objects and removes them, along with any associated
observation terms that reference these cameras.

Args:
    env_cfg: The environment configuration to modify.

Returns:
    The modified environment configuration with cameras removed.
"""

/isaac-sim/IsaacLab/blob/main/scripts/environments/teleoperation/teleop_se3_agent.py#L111-L113

if args_cli.xr:
env_cfg = remove_camera_configs(env_cfg)
env_cfg.sim.render.antialiasing_mode = “DLSS”

5

That remove_camera_configs call means extra scene cameras are intentionally removed when XR is enabled, so even non‑blocking reads from tiled_camera will not be reliable in VR. There is no simple or officially supported “bypass” for this right now.​

To help find a supported workaround, could you share:

  • Your Isaac Sim / Isaac Lab version, and

  • A minimal script showing how you create and read from env.scene["tiled_camera"] in VR?​

6

Hello!

We noticed that this topic hasn’t received any recent updates from anyone reporting this issue, so we are closing it for now to help keep the forum organized.

If you are still experiencing this issue or have related questions, please create a GitHub Discussion or Issue in the Isaac Sim repository and include a link to this topic along with updated details. Mentioning or linking to this original topic provides helpful context and makes it easier for others to assist you.

Thank you for being part of the NVIDIA Isaac Sim community.

Best regards,
The NVIDIA Isaac Sim Forum Team