I am trying to record a video of the agent, while testing:
python train.py task=ShadowHand num_envs=1 test=True checpoint=path capture_video=True
The code crashes at the render() call in the vec_task.py script when a function is called to grab the image and store it in the img variable:
img=self.virtual_display.grab()
How do I fix this?
I have tried updating Pillow
I am on Ubuntu 22.04
I met the same problem in Ubuntu 20.04. I solved it by reinstall PIL with XCB support following the instructions in Pillow website (Installation - Pillow (PIL Fork) 10.0.1 documentation).
More specifically, first install the external libraries: sudo apt-get install libtiff5-dev libjpeg8-dev libopenjp2-7-dev zlib1g-dev \ libfreetype6-dev liblcms2-dev libwebp-dev tcl8.6-dev tk8.6-dev python3-tk \ libharfbuzz-dev libfribidi-dev libxcb1-dev. Then reinstall Pillow with python3 -m pip install --upgrade Pillow --no-binary :all:.
The answer is probably too late now, but hopefully this will help future users.
Hi,
I tried but still fail…
I followed this link for the gym,
I can run
python train.py task=Cartpole
but if I want to capture the video, it failed
python train.py task=Ant capture_video=True capture_video_freq=1500 capture_video_len=100 force_render=False
I wonder how to solve the problem? If not, my plan B is to use my phone to video tape it. :P
Thanks.
E: Unable to locate package libfreetype6-dev
E: Unable to locate package libharfbuzz-dev
OSError: Pillow was built without XCB support
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
(IsaacGym) ak@ak-MS-7D99:/media/ak/HD/IsaacGym_Preview_4_Package/isaacgym/python/examples/IsaacGymEnvs/isaacgymenvs$ sudo apt-get install libtiff5-dev libjpeg8-dev libopenjp2-7-dev zlib1g-dev \ libfreetype6-dev liblcms2-dev libwebp-dev tcl8.6-dev tk8.6-dev python3-tk \ libharfbuzz-dev libfribidi-dev libxcb1-dev
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package libfreetype6-dev
E: Unable to locate package libharfbuzz-dev
(IsaacGym) ak@ak-MS-7D99:/media/ak/HD/IsaacGym_Preview_4_Package/isaacgym/python/examples/IsaacGymEnvs/isaacgymenvs$ python3 -m pip install --upgrade Pillow --no-binary :all:
Requirement already satisfied: Pillow in /home/ak/anaconda3/envs/IsaacGym/lib/python3.8/site-packages (10.3.0)
(IsaacGym) ak@ak-MS-7D99:/media/ak/HD/IsaacGym_Preview_4_Package/isaacgym/python/examples/IsaacGymEnvs/isaacgymenvs$ python train.py task=Ant capture_video=True capture_video_freq=1500 capture_video_len=100 force_render=False
/home/ak/anaconda3/envs/IsaacGym/lib/python3.8/site-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
ret = run_job(
Importing module 'gym_38' (/media/ak/HD/IsaacGym_Preview_4_Package/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_38.so)
Setting GYM_USD_PLUG_INFO_PATH to /media/ak/HD/IsaacGym_Preview_4_Package/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
/home/ak/anaconda3/envs/IsaacGym/lib/python3.8/site-packages/torch/utils/cpp_extension.py:25: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
from pkg_resources import packaging # type: ignore[attr-defined]
/home/ak/anaconda3/envs/IsaacGym/lib/python3.8/site-packages/pkg_resources/__init__.py:2832: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
PyTorch version 1.13.1
Device count 1
/media/ak/HD/IsaacGym_Preview_4_Package/isaacgym/python/isaacgym/_bindings/src/gymtorch
Using /home/ak/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Emitting ninja build file /home/ak/.cache/torch_extensions/py38_cu117/gymtorch/build.ninja...
Building extension module gymtorch...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module gymtorch...
2024-06-01 21:42:48,993 - INFO - logger - logger initialized
<unknown>:3: DeprecationWarning: invalid escape sequence \*
Error: FBX library failed to load - importing FBX data will not succeed. Message: No module named 'fbx'
FBX tools must be installed from https://help.autodesk.com/view/FBX/2020/ENU/?guid=FBX_Developer_Help_scripting_with_python_fbx_installing_python_fbx_html
/media/ak/HD/IsaacGym_Preview_4_Package/isaacgym/python/isaacgym/torch_utils.py:135: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
def get_axis_params(value, axis_idx, x_value=0., dtype=np.float, n_dims=3):
/home/ak/anaconda3/envs/IsaacGym/lib/python3.8/site-packages/networkx/classes/graph.py:23: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
from collections import Mapping
/home/ak/anaconda3/envs/IsaacGym/lib/python3.8/site-packages/networkx/classes/reportviews.py:95: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
from collections import Mapping, Set, Iterable
/home/ak/anaconda3/envs/IsaacGym/lib/python3.8/site-packages/networkx/readwrite/graphml.py:346: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
(np.int, "int"), (np.int8, "int"),
/home/ak/anaconda3/envs/IsaacGym/lib/python3.8/site-packages/torch/utils/tensorboard/__init__.py:4: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
if not hasattr(tensorboard, "__version__") or LooseVersion(
task:
name: Ant
physics_engine: physx
env:
numEnvs: 4096
envSpacing: 5
episodeLength: 1000
enableDebugVis: False
clipActions: 1.0
powerScale: 1.0
controlFrequencyInv: 1
headingWeight: 0.5
upWeight: 0.1
actionsCost: 0.005
energyCost: 0.05
dofVelocityScale: 0.2
contactForceScale: 0.1
jointsAtLimitCost: 0.1
deathCost: -2.0
terminationHeight: 0.31
plane:
staticFriction: 1.0
dynamicFriction: 1.0
restitution: 0.0
asset:
assetFileName: mjcf/nv_ant.xml
enableCameraSensors: False
sim:
dt: 0.0166
substeps: 2
up_axis: z
use_gpu_pipeline: True
gravity: [0.0, 0.0, -9.81]
physx:
num_threads: 4
solver_type: 1
use_gpu: True
num_position_iterations: 4
num_velocity_iterations: 0
contact_offset: 0.02
rest_offset: 0.0
bounce_threshold_velocity: 0.2
max_depenetration_velocity: 10.0
default_buffer_size_multiplier: 5.0
max_gpu_contact_pairs: 8388608
num_subscenes: 4
contact_collection: 0
task:
randomize: False
randomization_params:
frequency: 600
observations:
range: [0, 0.002]
operation: additive
distribution: gaussian
actions:
range: [0.0, 0.02]
operation: additive
distribution: gaussian
actor_params:
ant:
color: True
rigid_body_properties:
mass:
range: [0.5, 1.5]
operation: scaling
distribution: uniform
setup_only: True
dof_properties:
damping:
range: [0.5, 1.5]
operation: scaling
distribution: uniform
stiffness:
range: [0.5, 1.5]
operation: scaling
distribution: uniform
lower:
range: [0, 0.01]
operation: additive
distribution: gaussian
upper:
range: [0, 0.01]
operation: additive
distribution: gaussian
train:
params:
seed: 42
algo:
name: a2c_continuous
model:
name: continuous_a2c_logstd
network:
name: actor_critic
separate: False
space:
continuous:
mu_activation: None
sigma_activation: None
mu_init:
name: default
sigma_init:
name: const_initializer
val: 0
fixed_sigma: True
mlp:
units: [256, 128, 64]
activation: elu
d2rl: False
initializer:
name: default
regularizer:
name: None
load_checkpoint: False
load_path:
config:
name: Ant
full_experiment_name: Ant
env_name: rlgpu
multi_gpu: False
ppo: True
mixed_precision: True
normalize_input: True
normalize_value: True
value_bootstrap: True
num_actors: 4096
reward_shaper:
scale_value: 0.01
normalize_advantage: True
gamma: 0.99
tau: 0.95
learning_rate: 0.0003
lr_schedule: adaptive
schedule_type: legacy
kl_threshold: 0.008
score_to_win: 20000
max_epochs: 500
save_best_after: 200
save_frequency: 50
grad_norm: 1.0
entropy_coef: 0.0
truncate_grads: False
e_clip: 0.2
horizon_length: 16
minibatch_size: 32768
mini_epochs: 4
critic_coef: 2
clip_value: True
seq_len: 4
bounds_loss_coef: 0.0001
pbt:
enabled: False
task_name: Ant
experiment:
num_envs:
seed: 42
torch_deterministic: False
max_iterations:
physics_engine: physx
pipeline: gpu
sim_device: cuda:0
rl_device: cuda:0
graphics_device_id: 0
num_threads: 4
solver_type: 1
num_subscenes: 4
test: False
checkpoint:
sigma:
multi_gpu: False
wandb_activate: False
wandb_group:
wandb_name: Ant
wandb_entity:
wandb_project: isaacgymenvs
wandb_tags: []
wandb_logcode_dir:
capture_video: True
capture_video_freq: 1500
capture_video_len: 100
force_render: False
headless: False
Setting seed: 42
Using rl_device: cuda:0
Using sim_device: cuda:0
{'name': 'Ant', 'full_experiment_name': None, 'env_name': 'rlgpu', 'multi_gpu': False, 'ppo': True, 'mixed_precision': True, 'normalize_input': True, 'normalize_value': True, 'value_bootstrap': True, 'num_actors': 4096, 'reward_shaper': {'scale_value': 0.01}, 'normalize_advantage': True, 'gamma': 0.99, 'tau': 0.95, 'learning_rate': 0.0003, 'lr_schedule': 'adaptive', 'schedule_type': 'legacy', 'kl_threshold': 0.008, 'score_to_win': 20000, 'max_epochs': 500, 'save_best_after': 200, 'save_frequency': 50, 'grad_norm': 1.0, 'entropy_coef': 0.0, 'truncate_grads': False, 'e_clip': 0.2, 'horizon_length': 16, 'minibatch_size': 32768, 'mini_epochs': 4, 'critic_coef': 2, 'clip_value': True, 'seq_len': 4, 'bounds_loss_coef': 0.0001, 'device': 'cuda:0', 'population_based_training': False, 'pbt_idx': None}
self.seed = 42
Started to train
/home/ak/anaconda3/envs/IsaacGym/lib/python3.8/site-packages/gym/spaces/box.py:84: UserWarning: WARN: Box bound precision lowered by casting to float32
logger.warn(f"Box bound precision lowered by casting to {self.dtype}")
[Warning] [carb.gym.plugin] useGpu is set, forcing single scene (0 subscenes)
Not connected to PVD
+++ Using GPU PhysX
Physics Engine: PhysX
Physics Device: cuda:0
GPU Pipeline: enabled
num envs 4096 env spacing 5
/media/ak/HD/IsaacGym_Preview_4_Package/isaacgym/python/examples/IsaacGymEnvs/isaacgymenvs/tasks/ant.py:151: DeprecationWarning: an integer is required (got type isaacgym._bindings.linux-x86_64.gym_38.DofDriveMode). Implicit conversion to integers using __int__ is deprecated, and may be removed in a future version of Python.
asset_options.default_dof_drive_mode = gymapi.DOF_MODE_NONE
Box(-1.0, 1.0, (8,), float32) Box(-inf, inf, (60,), float32)
WARNING: seq_len is deprecated, use seq_length instead
seq_length: 4
current training device: cuda:0
build mlp: 60
RunningMeanStd: (1,)
RunningMeanStd: (60,)
/home/ak/anaconda3/envs/IsaacGym/lib/python3.8/site-packages/gym/wrappers/monitoring/video_recorder.py:43: DeprecationWarning: WARN: `env.metadata["render.modes"] is marked as deprecated and will be replaced with `env.metadata["render_modes"]` see https://github.com/openai/gym/pull/2654 for more details
logger.deprecation(
/home/ak/anaconda3/envs/IsaacGym/lib/python3.8/site-packages/gym/wrappers/monitoring/video_recorder.py:115: DeprecationWarning: WARN: `env.metadata["video.frames_per_second"] is marked as deprecated and will be replaced with `env.metadata["render_fps"]` see https://github.com/openai/gym/pull/2654 for more details
logger.deprecation(
Error executing job with overrides: ['task=Ant', 'capture_video=True', 'capture_video_freq=1500', 'capture_video_len=100', 'force_render=False']
Traceback (most recent call last):
File "train.py", line 210, in launch_rlg_hydra
runner.run({
File "/home/ak/anaconda3/envs/IsaacGym/lib/python3.8/site-packages/rl_games/torch_runner.py", line 133, in run
self.run_train(args)
File "/home/ak/anaconda3/envs/IsaacGym/lib/python3.8/site-packages/rl_games/torch_runner.py", line 116, in run_train
agent.train()
File "/home/ak/anaconda3/envs/IsaacGym/lib/python3.8/site-packages/rl_games/common/a2c_common.py", line 1307, in train
self.obs = self.env_reset()
File "/home/ak/anaconda3/envs/IsaacGym/lib/python3.8/site-packages/rl_games/common/a2c_common.py", line 531, in env_reset
obs = self.vec_env.reset()
File "/media/ak/HD/IsaacGym_Preview_4_Package/isaacgym/python/examples/IsaacGymEnvs/isaacgymenvs/utils/rlgames_utils.py", line 250, in reset
return self.env.reset()
File "/home/ak/anaconda3/envs/IsaacGym/lib/python3.8/site-packages/gym/wrappers/record_video.py", line 58, in reset
self.start_video_recorder()
File "/home/ak/anaconda3/envs/IsaacGym/lib/python3.8/site-packages/gym/wrappers/record_video.py", line 75, in start_video_recorder
self.video_recorder.capture_frame()
File "/home/ak/anaconda3/envs/IsaacGym/lib/python3.8/site-packages/gym/wrappers/monitoring/video_recorder.py", line 157, in capture_frame
frame = self.env.render(mode=render_mode)
File "/media/ak/HD/IsaacGym_Preview_4_Package/isaacgym/python/examples/IsaacGymEnvs/isaacgymenvs/tasks/base/vec_task.py", line 511, in render
img = self.virtual_display.grab()
File "/home/ak/anaconda3/envs/IsaacGym/lib/python3.8/site-packages/pyvirtualdisplay/smartdisplay.py", line 44, in grab
img = grab(xdisplay=self.new_display_var)
File "/home/ak/anaconda3/envs/IsaacGym/lib/python3.8/site-packages/PIL/ImageGrab.py", line 69, in grab
raise OSError(msg)
OSError: Pillow was built without XCB support
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
(IsaacGym) ak@ak-MS-7D99:/media/ak/HD/IsaacGym_Preview_4_Package/isaacgym/python/examples/IsaacGymEnvs/isaacgymenvs$