Get camera image from gym in headless mode

fira · March 10, 2023, 7:38pm

I’m attempting to obtain the camera image from a robot-mounted camera that I have imported into my project in USD format. When I run the program in normal mode, I am able to successfully retrieve the camera feed. However, when running in headless mode, I encounter the error message below.

It’s worth noting that this issue pertains to the Omni Isaac Gym environments, which are a new gym integrated into Isaac Sim. To provide context, I have placed my robot inside the cartpole example. @toni.sm do you have any idea what could be causing the issue? Or what could be a way around it.

The code in set_up_scene

    def set_up_scene(self, scene) -> None:

        self.get_cartpole()
        super().set_up_scene(scene)
        self._cartpoles = ArticulationView(prim_paths_expr="/World/envs/.*/Cartpole", name="cartpole_view", reset_xform_properties=False)

        self.camera = Camera(prim_path="/World/envs/env_0/Cartpole/test/chassi/camera", resolution=(100,100))

        self.camera.initialize()

        self.camera.add_motion_vectors_to_frame()
        scene.add(self._cartpoles)
        return

I feel this is the important part of the error, but all error is bellow to.

    self.camera = Camera(prim_path="/World/envs/env_0/Cartpole/test/chassi/camera", resolution=(100,100))
  File "/home/oscsa287/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.isaac.sensor/omni/isaac/sensor/scripts/camera.py", line 109, in __init__
    self._rgb_annotator.attach([self._render_product_path])
  File "/home/oscsa287/.local/share/ov/pkg/isaac_sim-2022.2.0/extscache/omni.replicator.core-1.6.4+104.1.lx64.r.cp37/omni/replicator/core/scripts/annotators.py", line 379, in attach
    self.template_name, 0, render_products, attributes=self._init_params
  File "/home/oscsa287/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extscore/omni.syntheticdata/omni/syntheticdata/scripts/SyntheticData.py", line 1385, in activate_node_template
    SyntheticData._add_rendervar(rp, rv, stage)
  File "/home/oscsa287/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extscore/omni.syntheticdata/omni/syntheticdata/scripts/SyntheticData.py", line 694, in _add_rendervar
    vp_iface = omni.kit.viewport_legacy.get_viewport_interface()
  File "/home/oscsa287/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extscore/omni.kit.window.viewport/omni/kit/viewport_legacy/scripts/viewport.py", line 16, in get_viewport_interface
    get_viewport_interface.viewport = acquire_viewport_interface()
RuntimeError: Failed to acquire interface: omni::kit::IViewport (pluginName: nullptr)

Error executing job with overrides: ['task=Cartpole', 'headless=True']
Traceback (most recent call last):
  File "scripts/rlgames_train.py", line 109, in parse_hydra_configs
    task = initialize_task(cfg_dict, env)
  File "/home/oscsa287/.local/share/ov/pkg/isaac_sim-2022.2.0/OmniIsaacGymEnvs/omniisaacgymenvs/utils/task_util.py", line 70, in initialize_task
    env.set_task(task=task, sim_params=sim_config.get_physics_params(), backend="torch", init_sim=init_sim)
  File "/home/oscsa287/.local/share/ov/pkg/isaac_sim-2022.2.0/OmniIsaacGymEnvs/omniisaacgymenvs/envs/vec_env_rlgames.py", line 51, in set_task
    super().set_task(task, backend, sim_params, init_sim)
  File "/home/oscsa287/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.isaac.gym/omni/isaac/gym/vec_env/vec_env_base.py", line 80, in set_task
    self._world.reset()
  File "/home/oscsa287/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.isaac.core/omni/isaac/core/world/world.py", line 275, in reset
    task.set_up_scene(self.scene)
  File "/home/oscsa287/.local/share/ov/pkg/isaac_sim-2022.2.0/OmniIsaacGymEnvs/omniisaacgymenvs/tasks/cartpole.py", line 85, in set_up_scene
    self.camera = Camera(prim_path="/World/envs/env_0/Cartpole/test/chassi/camera", resolution=(100,100))
  File "/home/oscsa287/.local/share/ov/pkg/isaac_sim-2022.2.0/exts/omni.isaac.sensor/omni/isaac/sensor/scripts/camera.py", line 109, in __init__
    self._rgb_annotator.attach([self._render_product_path])
  File "/home/oscsa287/.local/share/ov/pkg/isaac_sim-2022.2.0/extscache/omni.replicator.core-1.6.4+104.1.lx64.r.cp37/omni/replicator/core/scripts/annotators.py", line 379, in attach
    self.template_name, 0, render_products, attributes=self._init_params
  File "/home/oscsa287/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extscore/omni.syntheticdata/omni/syntheticdata/scripts/SyntheticData.py", line 1385, in activate_node_template
    SyntheticData._add_rendervar(rp, rv, stage)
  File "/home/oscsa287/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extscore/omni.syntheticdata/omni/syntheticdata/scripts/SyntheticData.py", line 694, in _add_rendervar
    vp_iface = omni.kit.viewport_legacy.get_viewport_interface()
  File "/home/oscsa287/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extscore/omni.kit.window.viewport/omni/kit/viewport_legacy/scripts/viewport.py", line 16, in get_viewport_interface
    get_viewport_interface.viewport = acquire_viewport_interface()
RuntimeError: Failed to acquire interface: omni::kit::IViewport (pluginName: nullptr)

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Exception ignored in: <function _make_registry.<locals>._Registry.__del__ at 0x7ff6e54da050>
Traceback (most recent call last):
  File "/home/oscsa287/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extscore/omni.kit.viewport.registry/omni/kit/viewport/registry/registry.py", line 103, in __del__
  File "/home/oscsa287/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extscore/omni.kit.viewport.registry/omni/kit/viewport/registry/registry.py", line 98, in destroy
TypeError: 'NoneType' object is not callable
Exception ignored in: <function _make_registry.<locals>._Registry.__del__ at 0x7ff6e54da050>
Traceback (most recent call last):
  File "/home/oscsa287/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extscore/omni.kit.viewport.registry/omni/kit/viewport/registry/registry.py", line 103, in __del__
  File "/home/oscsa287/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extscore/omni.kit.viewport.registry/omni/kit/viewport/registry/registry.py", line 98, in destroy
TypeError: 'NoneType' object is not callable
Exception ignored in: <function SettingChangeSubscription.__del__ at 0x7ff758c8d9e0>
Traceback (most recent call last):
  File "/home/oscsa287/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/kernel/py/omni/kit/app/_impl/__init__.py", line 114, in __del__
AttributeError: 'NoneType' object has no attribute 'get_settings'
2023-03-10 19:25:44 [9,017ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-03-10 19:25:44 [9,017ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-03-10 19:25:44 [9,018ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-03-10 19:25:44 [9,018ms] [Error] [omni.usd] TF_PYTHON_EXCEPTION: in TfPyConvertPythonExceptionToTfErrors at line 114 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/base/tf/pyError.cpp -- Tf Python Exception

2023-03-10 19:25:45 [9,103ms] [Warning] [carb.audio.context] 1 contexts were leaked

toni.sm · March 11, 2023, 10:19am

Hi @fira

The SimulationApp instance, in headless, use a simplified experience configuration (apps/omni.isaac.sim.python.gym.headless.kit) that doesn’t include several viewport/rendering-related extensions.

Although I don’t get the same error as you, I had to perform the following steps to be able to launch a headless simulation with a camera (like your code snippet) and without errors

comment the lines 33-34 in exts/omni.isaac.gym/omni/isaac/gym/vec_env/vec_env_base.py to avoid loading omni.isaac.sim.python.gym.headless.kit experience in headless mode
```
    experience = ""
    # if headless:
    #     experience = f'{os.environ["EXP_PATH"]}/omni.isaac.sim.python.gym.headless.kit'
```

run the script with the following arguments:

 headless=True +task.sim.enable_viewport=True task.sim.enable_cameras=True

Note: Even with that, I am not sure everything with works. I think we need to wait for the integration of camera tensor API in futures releases to do massive parallel RL with visual information successfully

kellyg · March 13, 2023, 2:13pm

Thanks @toni.sm ! Yes, the headless app that the gym environments use does not currently support headless rendering. We will be modifying the logic for this slightly in the upcoming release and adding support for headless rendering by specifying the enable_cameras parameter in the task config.

nzamani · June 7, 2023, 9:49pm

@kellyg has this been resolved in 2022.2.1 version?

rthaker · June 21, 2023, 4:33pm

Yes. You can try it in the Isaac Sim 2022.2.1 and let us know if you have any other questions.

psh9002 · June 22, 2023, 8:01am

Hi, @rthaker

I tried but I cannot get any images with headless mode.
I wrote two topics but cannot get any answers.

Should I had to try with another method for getting images with headless mode?

Topic		Replies	Views
How can I handle images at headless mode? Isaac Sim	7	1499	July 11, 2023
How to get some image data from Isaac Sim Isaac Gym camera	1	1244	October 11, 2023
How to handle camera images with headless mode at SKRL? Isaac Sim	6	639	May 19, 2023
Running Python code with headless container Isaac Sim headless , python	7	1728	April 5, 2024
Error occurring when trying reinforcement learning examples in Isaac gym Isaac Sim	15	1326	March 8, 2023
Omniverse ISAAC Gym - Access camera stream Isaac Gym	10	3226	October 26, 2023
Problem with images I get from cameras Isaac Sim	5	1189	May 17, 2023
Camera example and headless mode Isaac Gym camera , headless	11	6500	July 12, 2023
OmniIsaacGymEnv porevious version (Isaac Sim 2022.2.0) Isaac Sim	6	561	June 10, 2023
AttributeError in Isaac Sim 2022.2.1: 'omni.kit' has no attribute 'viewport_legacy' - Debugging and Resolution Isaac Sim	8	958	September 5, 2023

Get camera image from gym in headless mode

Related topics