How to handle camera images with headless mode at SKRL?

Omniverse Isaac Sim
1

Hello

I am trying to handle RGBD images on SKRL.
Before using images via features, I tried to save images on my disk.

However, I cannot save images in headless mode though I can save images in non-headless mode.

I think the error occurred because the annotator-related instance did not create when I use headless mode.

The error occurred at camera.py line 109.
self._rgb_annotator.attach([self._render_product_path])

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.0/extscache/omni.replicator.core-1.6.4+104.1.lx64.r.cp37/omni/replicator/core/scripts/annotators.py", line 378, in attach
    sdg_iface.activate_node_template(
AttributeError: 'NoneType' object has no attribute 'activate_node_template'

How can I solve the problem? I attach my code.
(Please refer image save directory at the line 347 at reaching_ur5e_env.py)

Thanks
reaching_ur5e.7z (15.7 MB)

2

Hi @psh9002

Could you please provide more details about your current setup to try to reproduce the error?
Example:

  • skrl version (PATH_TO_ISAAC_SIM_PYTHON -m pip show skrl)
  • OmniIsaacGymEnvs version (last commit)

From your comment, you are using Isaac Sim 2022.2.0. Have you considered upgrading to the latest version of Isaac Sim (v2022.2.1) and OmniIsaacGymEnvs (OIGE) and skrl?

1 Like
3

Hi, @toni.sm

I retested on Isaac Sim 2022.2.1 and the latest version of OIGE and skrl(0.10.2).

I got an error message which is related with replicator.

Warp 0.6.3 initialized:
   CUDA Toolkit: 11.5, Driver: 12.0
   Devices:
     "cpu"    | x86_64
     "cuda:0" | NVIDIA GeForce RTX 3090 (sm_86)
   Kernel cache: /home/bak/.cache/warp/0.6.3
[1.575s] [ext: omni.kit.window.title-1.1.2] startup
[1.575s] [ext: omni.isaac.gym-0.3.3] startup
[1.575s] [ext: omni.isaac.sim.python.gym.headless-2022.2.1] startup
[1.576s] Simulation App Starting
[1.853s] app ready
[1.870s] Simulation App Startup Complete
Traceback (most recent call last):
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/OmniIsaacGymEnvsAILAB/omniisaacgymenvs/skrl_examples/reaching_ur5e/reaching_ur5e_train.py", line 64, in <module>
    from reaching_ur5e_env import ReachingUR5eTask, TASK_CFG
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/OmniIsaacGymEnvsAILAB/omniisaacgymenvs/skrl_examples/reaching_ur5e/reaching_ur5e_env.py", line 5, in <module>
    from omniisaacgymenvs.tasks.base.rl_task import RLTask
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/OmniIsaacGymEnvs/omniisaacgymenvs/tasks/base/rl_task.py", line 39, in <module>
    from omniisaacgymenvs.utils.domain_randomization.randomize import Randomizer
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/OmniIsaacGymEnvs/omniisaacgymenvs/utils/domain_randomization/randomize.py", line 32, in <module>
    import omni.replicator.core as rep
ModuleNotFoundError: No module named 'omni.replicator'

Also, I have another issue when I use more than 7 cameras. The code works well at non-headless mode when I use less than 8 cameras. I attach the error with txt file.

Thanks.

reaching_ur5e.7z (15.7 MB)

Error (47.9 KB)

4

Hi @psh9002

Please, make sure your environment includes the next lines required by the latest OIGE.
Put those line before from omniisaacgymenvs.tasks.base.rl_task import RLTask

1 Like
5

Hi @toni.sm

Though I reinstall OIGE and skrl pip packages, I cannot get images from the cameras.

There was an error at omniisaacgymenvs/utils/config_utils/sim_config.py from

if self._config["headless"] == True and not self._sim_params["enable_cameras"] and not self._config["enable_livestream"]:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
KeyError: 'enable_livestream'

The self._config does not have a key enable_livestream at both headless and non-headless mode but, the above error only occurred at headless mode.

When I jumped the above problem through except:, the another error occurred at viewport_manager_legacy.py.

Exception has occurred: AttributeError
module 'omni.kit' has no attribute 'viewport_legacy'
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/extscache/omni.replicator.core-1.7.8+104.2.lx64.r.cp37/omni/replicator/core/scripts/utils/viewport_manager_legacy.py", line 91, in get_render_product
    vp_iface = omni.kit.viewport_legacy.get_viewport_interface()
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/extscache/omni.replicator.core-1.7.8+104.2.lx64.r.cp37/omni/replicator/core/scripts/utils/viewport_manager.py", line 278, in get_render_product
    return vm_legacy.get_render_product(camera_path, resolution)
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/extscache/omni.replicator.core-1.7.8+104.2.lx64.r.cp37/omni/replicator/core/scripts/utils/viewport_manager.py", line 309, in get_render_product
    return ViewportManager().get_render_product(camera_path, resolution, force_new)
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/extscache/omni.replicator.core-1.7.8+104.2.lx64.r.cp37/omni/replicator/core/scripts/create.py", line 1137, in render_product
    return viewport_manager.get_render_product(camera_paths[0], resolution, force_new)
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.sensor/omni/isaac/sensor/scripts/camera.py", line 107, in __init__
    self._render_product_path = rep.create.render_product(prim_path, resolution=resolution)
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/OmniIsaacGymEnvsAILAB/omniisaacgymenvs/skrl_examples/reaching_ur5e/ur5e_tip.py", line 100, in __init__
    frequency=30, resolution=(1920, 1080)) ### BSH
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/OmniIsaacGymEnvsAILAB/omniisaacgymenvs/skrl_examples/reaching_ur5e/reaching_ur5e_env.py", line 198, in get_robot
    name="robot")
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/OmniIsaacGymEnvsAILAB/omniisaacgymenvs/skrl_examples/reaching_ur5e/reaching_ur5e_env.py", line 164, in set_up_scene
    self.get_robot()
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.core/omni/isaac/core/world/world.py", line 275, in reset
    task.set_up_scene(self.scene)
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.gym/omni/isaac/gym/vec_env/vec_env_base.py", line 94, in set_task
    self._world.reset()
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/OmniIsaacGymEnvsAILAB/omniisaacgymenvs/skrl_examples/reaching_ur5e/reaching_ur5e_train.py", line 84, in <module>
    env.set_task(task=task, sim_params=sim_config.get_physics_params(), backend="torch", init_sim=True)
AttributeError: module 'omni.kit' has no attribute 'viewport_legacy'

Additionally, I added some lines in my robot code(ur5e_tip.py) because my environment cannot find some modules that is related with omni.isaac.

from pathlib import Path
current_dir = os.path.dirname(os.path.realpath(__file__))
directory = Path(current_dir).parent.parent.parent.parent
directory = os.path.join(str(directory), 'exts')

sys.path.append(directory)

from omni.isaac.core.utils.extensions import enable_extension
enable_extension("omni.isaac.sensor")

Here is my code.
reaching_ur5e.7z (15.7 MB)

6

Hi @psh9002

  • Regarding the enable_livestream configuration:

    I apologize as an AI language model for not providing all the details regarding the changes introduced in the skrl examples for OIGE.

    If you look at the differences in the example code, you will see that 2 things are added: 1) import replicator and 2) add enable_livestream key

    Screenshot from 2023-05-19 09-18-55
    Screenshot from 2023-05-19 09-18-551791×465 69.2 KB

  • About the viewport_legacy error

    You can enable the omni.kit.window.viewport extension to solve it

    from omni.isaac.core.utils.extensions import enable_extension
enable_extension("omni.kit.window.viewport")  # enable legacy viewport interface
1 Like
7

Thanks, @toni.sm

I can solve 'viewport_legacy' problem but, the created rgba array is empty.

def capture_and_save_images(self):

camera_indices = range(self._num_envs)

save_dir = os.path.join(os.path.expanduser( '~' ), 'isaac_sim_RL_imgs')
os.makedirs(save_dir, exist_ok=True)
    
for idx in camera_indices:
    cam_rgba = self._rgb_cameras[idx].rgb_cam.get_rgba()
    cam_rgba_img = cv2.cvtColor(cam_rgba, cv2.COLOR_BGRA2RGBA)
    filename = os.path.join(save_dir, f'{self.step_num}_{idx}.png')
    cv2.imwrite(filename, cam_rgba_img)

self.step_num += 1

In above code which I uploaded, cam_rgba is empty.
Is there another way to get images?