How can I handle images at headless mode?

Omniverse Isaac Sim
,
1

Hello,

I am trying to handle RGBD images.
Before using images via features, I tried to save images on my disk.

However, I cannot save images in headless mode though I can save images in non-headless mode.

[skrl:INFO] Environment class: omni.isaac.gym.vec_env.vec_env_base.VecEnvBase
[skrl:INFO] Environment wrapper: Omniverse Isaac Gym
Traceback (most recent call last):
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/OmniIsaacGymEnvsAILAB/omniisaacgymenvs/skrl_examples/reaching_ur5e/reaching_ur5e_train.py", line 146, in <module>
    trainer.train()
  File "/home/bak/anaconda3/envs/isaac-sim_22.2.1/lib/python3.7/site-packages/skrl/trainers/torch/sequential.py", line 75, in train
    self.single_agent_train()
  File "/home/bak/anaconda3/envs/isaac-sim_22.2.1/lib/python3.7/site-packages/skrl/trainers/torch/base.py", line 155, in single_agent_train
    states, infos = self.env.reset()
  File "/home/bak/anaconda3/envs/isaac-sim_22.2.1/lib/python3.7/site-packages/skrl/envs/torch/wrappers.py", line 250, in reset
    self._obs_dict = self._env.reset()
  File "/home/bak/anaconda3/envs/isaac-sim_22.2.1/lib/python3.7/site-packages/skrl/utils/omniverse_isaacgym_utils.py", line 226, in reset
    return self.step(actions)[0]
  File "/home/bak/anaconda3/envs/isaac-sim_22.2.1/lib/python3.7/site-packages/skrl/utils/omniverse_isaacgym_utils.py", line 218, in step
    observations, rewards, dones, info = self._task.post_physics_step()
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/OmniIsaacGymEnvsAILAB/omniisaacgymenvs/tasks/base/rl_task.py", line 254, in post_physics_step
    self.get_observations()
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/OmniIsaacGymEnvsAILAB/omniisaacgymenvs/skrl_examples/reaching_ur5e/reaching_ur5e_env.py", line 246, in get_observations
    self.capture_and_save_images()
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/OmniIsaacGymEnvsAILAB/omniisaacgymenvs/skrl_examples/reaching_ur5e/reaching_ur5e_env.py", line 374, in capture_and_save_images
    cam_rgba_img = cv2.cvtColor(cam_rgba, cv2.COLOR_BGRA2RGBA)
cv2.error: OpenCV(4.7.0) /io/opencv/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

In my function def capture_and_save_images(self):, there is a creating image function from simulation instance to image that I used by cam_rgba = self._rgb_cameras[idx].rgb_cam.get_rgba().
However cam_rgba is empty.
Is there another way to get images?

Here is my code.
reaching_ur5e.7z (15.7 MB)

Thanks

Actually, I asked before but, there is no more reply. I think the cause of the problem is not come from SKRL so I rewrite the question and change the title.

1 Like
3

Dear, @kellyg

Can’t we still get images at headless mode?
I read a comment at March 13.

5

Hi there, rendering in headless mode should work if the enable_cameras flag in the task config file is set to True. Without this flag, OmniIsaacGymEnvs will by default disable all rendering extensions for performance purposes.

1 Like
6

Thanks, @kellyg

I tired "enable_cameras": True but, the same error messages still exist.

I changed the config in env file such as

                     "sim": {"dt": 0.0083,  # 1 / 120
                             "use_gpu_pipeline": True,
                             "gravity": [0.0, 0.0, -9.81],
                             "add_ground_plane": True,
                             "use_flatcache": True,
                             "enable_scene_query_support": False,
                             "enable_cameras": True,
                             "default_physics_material": {"static_friction": 1.0,
                                                         "dynamic_friction": 1.0,
                                                         "restitution": 0.0},

The error messages is same as above.

[skrl:INFO] Environment class: omni.isaac.gym.vec_env.vec_env_base.VecEnvBase
INFO:skrl:Environment class: omni.isaac.gym.vec_env.vec_env_base.VecEnvBase
[skrl:INFO] Environment wrapper: Omniverse Isaac Gym
INFO:skrl:Environment wrapper: Omniverse Isaac Gym
Traceback (most recent call last):
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.0/OmniIsaacGymEnvs/omniisaacgymenvs/skrl_examples/reaching_ur5e/reaching_ur5e_train.py", line 148, in <module>
    trainer.train()
  File "/home/bak/anaconda3/envs/tool-manipulation/lib/python3.7/site-packages/skrl/trainers/torch/sequential.py", line 75, in train
    self.single_agent_train()
  File "/home/bak/anaconda3/envs/tool-manipulation/lib/python3.7/site-packages/skrl/trainers/torch/base.py", line 155, in single_agent_train
    states, infos = self.env.reset()
  File "/home/bak/anaconda3/envs/tool-manipulation/lib/python3.7/site-packages/skrl/envs/torch/wrappers.py", line 250, in reset
    self._obs_dict = self._env.reset()
  File "/home/bak/anaconda3/envs/tool-manipulation/lib/python3.7/site-packages/skrl/utils/omniverse_isaacgym_utils.py", line 216, in reset
    return self.step(actions)[0]
  File "/home/bak/anaconda3/envs/tool-manipulation/lib/python3.7/site-packages/skrl/utils/omniverse_isaacgym_utils.py", line 208, in step
    observations, rewards, dones, info = self._task.post_physics_step()
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.0/OmniIsaacGymEnvs/omniisaacgymenvs/tasks/base/rl_task.py", line 254, in post_physics_step
    self.get_observations()
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.0/OmniIsaacGymEnvs/omniisaacgymenvs/skrl_examples/reaching_ur5e/reaching_ur5e_env.py", line 234, in get_observations
    self.capture_and_save_images()
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.0/OmniIsaacGymEnvs/omniisaacgymenvs/skrl_examples/reaching_ur5e/reaching_ur5e_env.py", line 362, in capture_and_save_images
    cam_rgba_img = cv2.cvtColor(cam_rgba, cv2.COLOR_BGRA2RGBA)
cv2.error: OpenCV(4.2.0) /io/opencv/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

Exception ignored in: <function _make_registry.<locals>._Registry.__del__ at 0x7fb3d3db1950>
Traceback (most recent call last):
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extscore/omni.kit.viewport.registry/omni/kit/viewport/registry/registry.py", line 103, in __del__
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extscore/omni.kit.viewport.registry/omni/kit/viewport/registry/registry.py", line 98, in destroy
TypeError: 'NoneType' object is not callable
Exception ignored in: <function _make_registry.<locals>._Registry.__del__ at 0x7fb3d3db1950>
Traceback (most recent call last):
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extscore/omni.kit.viewport.registry/omni/kit/viewport/registry/registry.py", line 103, in __del__
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extscore/omni.kit.viewport.registry/omni/kit/viewport/registry/registry.py", line 98, in destroy
TypeError: 'NoneType' object is not callable
Exception ignored in: <function SettingChangeSubscription.__del__ at 0x7fb3ce9137a0>
Traceback (most recent call last):
  File "/home/bak/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/kernel/py/omni/kit/app/_impl/__init__.py", line 114, in __del__
AttributeError: 'NoneType' object has no attribute 'get_settings'
7

Could you please try with the Isaac Sim 2022.2.1 release? The fix should be in that release.

1 Like
8

Thanks, @kellyg !
It works.

However, when I increase the number of environments. The error appeared.

2023-06-29 07:05:33 [6,235ms] [Error] [carb.events.python] AttributeError: 'PlayButtonGroup' object has no attribute '_play_button'

At:
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/kit/exts/omni.kit.widget.toolbar/omni/kit/widget/toolbar/builtin_tools/play_button_group.py(129): _on_timeline_event
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py(549): play
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py(387): initialize_physics
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py(408): reset
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.core/omni/isaac/core/world/world.py(281): reset
  /home/bak/.local/share/ov/pkg/isaac_sim-2022.2.1/exts/omni.isaac.gym/omni/isaac/gym/vec_env/vec_env_base.py(94): set_task
  reaching_ur5e_train.py(84): <module>

2023-06-29 07:05:34 [7,218ms] [Warning] [omni.physx.plugin] The rigid body at /World/envs/env_0/robot/base has a possibly invalid inertia tensor of {1.0, 1.0, 1.0} and a negative mass, small sphere approximated inertia was used. Either specify correct values in the mass properties, or add collider(s) to any shape(s) that you wish to automatically compute mass properties for. If you do not want the objects to collide, add colliders regardless then disable the 'enable collision' property.
2023-06-29 07:05:34 [7,218ms] [Warning] [omni.physx.plugin] The rigid body at /World/envs/env_0/robot/flange has a possibly invalid inertia tensor of {1.0, 1.0, 1.0} and a negative mass, small sphere approximated inertia was used. Either specify correct values in the mass properties, or add collider(s) to any shape(s) that you wish to automatically compute mass properties for. If you do not want the objects to collide, add colliders regardless then disable the 'enable collision' property.
2023-06-29 07:05:34 [7,218ms] [Warning] [omni.physx.plugin] The rigid body at /World/envs/env_0/robot/tip has a possibly invalid inertia tensor of {1.0, 1.0, 1.0} and a negative mass, small sphere approximated inertia was used. Either specify correct values in the mass properties, or add collider(s) to any shape(s) that you wish to automatically compute mass properties for. If you do not want the objects to collide, add colliders regardless then disable the 'enable collision' property.
2023-06-29 07:05:34 [7,218ms] [Warning] [omni.physx.plugin] The rigid body at /World/envs/env_0/robot/world has a possibly invalid inertia tensor of {1.0, 1.0, 1.0} and a negative mass, small sphere approximated inertia was used. Either specify correct values in the mass properties, or add collider(s) to any shape(s) that you wish to automatically compute mass properties for. If you do not want the objects to collide, add colliders regardless then disable the 'enable collision' property.
~*~*~*
~*~*~* Direct GPU Helper:
~*~*~*   16 rigid dynamics
~*~*~*   16 articulations, maxLinks=13
~*~*~*
2023-06-29 07:05:37 [10,457ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2023-06-29 07:05:37 [10,457ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2023-06-29 07:05:37 [10,457ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2023-06-29 07:05:37 [10,457ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2023-06-29 07:05:37 [10,457ms] [Error] [gpu.foundation.plugin] Failed to update params for RenderOp 1165
2023-06-29 07:05:37 [10,457ms] [Error] [gpu.foundation.plugin] Failed to update params for RenderOp Cached PT ClearAll. Will not execute this or subsequent RenderGraph operations. Aborting RenderGraph execution
2023-06-29 07:05:37 [10,457ms] [Error] [carb.scenerenderer-rtx.plugin] Failed to execute RenderGraph on device 0. Error Code: 7
[skrl:INFO] Environment class: omni.isaac.gym.vec_env.vec_env_base.VecEnvBase
[skrl:INFO] Environment wrapper: Omniverse Isaac Gym
2023-06-29 07:05:38 [10,847ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2023-06-29 07:05:38 [10,847ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed.
2023-06-29 07:05:38 [10,847ms] [Error] [gpu.foundation.plugin] Texture creation failed for the device: 0.
2023-06-29 07:05:38 [10,847ms] [Error] [gpu.foundation.plugin] Failed to update params for RenderOp 239
2023-06-29 07:05:38 [10,880ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2023-06-29 07:05:38 [10,880ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed.
2023-06-29 07:05:38 [10,880ms] [Error] [gpu.foundation.plugin] Texture creation failed for the device: 0.
2023-06-29 07:05:38 [10,880ms] [Error] [carb.scenerenderer-rtx.plugin] Failed to allocate 1920x1080 ldrColor resource for device mask 0x1
Segmentation fault (core dumped)

Is it due to the limitation of GPU computing power?

9

Hi @psh9002 ,

Yes, this error means that the machine has ran out of memory. We are continuing to work on memory and speed performance issues related to rendering. We are aware that currently, only a few cameras can be defined in the scene.

2 Likes
10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.