# Open AI Gym + ISAAC SIM **URL:** \ **Category:** Isaac Sim\ **Created:** [October 1, 2020, 8:40am UTC](https://forums.developer.nvidia.com/t/open-ai-gym-isaac-sim/156028 "2020-10-01T08:40:18Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Laurito](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/laurito/32/41732_2.png) [@Laurito](https://forums.developer.nvidia.com/u/Laurito)\ **Post date:** [October 1, 2020, 8:40am UTC](https://forums.developer.nvidia.com/t/open-ai-gym-isaac-sim/156028/1 "2020-10-01T08:40:18Z")
Hi guys! Is there a way to do some Reinforcement Learning by using Open AI gym in combination with Isaac Sim? Is there some information about it? Or where would you start? Best Regards ---
**Author:** ![toni.sm](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/toni.sm/32/27559_2.png) [@toni.sm](https://forums.developer.nvidia.com/u/toni.sm)\ **Post date:** [October 1, 2020, 10:17am UTC](https://forums.developer.nvidia.com/t/open-ai-gym-isaac-sim/156028/2 "2020-10-01T10:17:51Z")
Hi, I would like to share my experience with reinforcement learning and Isaac Sim with the hope that it will be useful ( **Note:** I use [RLlib](https://docs.ray.io/en/latest/rllib.html) for Reinforcement Learning, but the approach works for other libraries) First, to use a non-include python package (like Gym, OpenCV, TensorFlow, etc…) you must use the Isaac Sim’s python (not the system python) to install them. The Isaac Sim’ python path is `/isaac-sim/_build/target-deps/kit_sdk_release/_build/target-deps/python` Then, you can use the python application (`bin/python3`) like the next example: ```auto cd /isaac-sim/_build/target-deps/kit_sdk_release/_build/target-deps/python bin/python3 -m pip install --upgrade pip bin/python3 -m pip install gym bin/python3 -m pip install pydot graphviz bin/python3 -m pip install tensorflow bin/python3 -m pip install ray[rllib] ray[debug] ``` Then, for RLlib, it is necessary to start the Ray cluster together with the simulator once (so, you can add the next code at the end of the python bindings for Isaac Sim extensions) Edit ```auto nano /isaac-sim/_build/target-deps/kit_sdk_release/_build/linux-x86_64/release/plugins/bindings-python/omni/ext/ __init__.py ``` and add the next code at the end (modifying the CPUs and GPUs) ```auto try: print("") print("+++++++++++++++++++++++++++") print("[RAY][TRY] ray.init...") import ray ray.init(num_cpus=20, num_gpus=4, local_mode=True, temp_dir="/isaac-sim/ray", configure_logging=True) print("[RAY][DONE] ray") except Exception as e: import traceback print("[RAY][ERROR] traceback:", traceback.format_exc()) print("---------------------------") print("") ``` After that, I created my own Reinforcement Learning sample (inside omni.isaac.samples: `/isaac-sim/_build/linux-x86_64/release/exts/omni.isaac.samples/omni/isaac/samples/scripts`) with the following structure (Gym environment, my robots and my RL architectures/models): ```auto │ │ rl_preview.py │ ├───rl_scenarios │ │ quillen2018deep.py │ │ scenario.py │ │ __init__.py │ │ │ ├───gym_scenarios │ │ │ __init__.py │ │ │ │ │ ├───envs │ │ │ │ quillen2018deep_env.py │ │ │ │ __init__.py │ │ │ ├───models │ │ │ ddpg_quillen2018deep.py │ │ │ __init__.py │ │ │ ├───robots │ │ │ iiwa.py │ │ │ pr2.py │ │ │ robot.py │ │ │ __init__.py │ │ ``` ![RL_quillen_.wmv_20201001_121147.229](https://global.discourse-cdn.com/nvidia/original/3X/a/0/a030aadb3e5a889ed60d64b30765576c9a6aea3d.jpeg) ---
**Author:** ![Laurito](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/laurito/32/41732_2.png) [@Laurito](https://forums.developer.nvidia.com/u/Laurito)\ **Post date:** [October 1, 2020, 11:10am UTC](https://forums.developer.nvidia.com/t/open-ai-gym-isaac-sim/156028/3 "2020-10-01T11:10:54Z")
Thanks for your reply @toni.sm That’s a great starting point! I will try it out. Do you have also a small example of this somewhere on github? (if allowed) I think many people would be interested that =) ---
**Author:** ![toni.sm](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/toni.sm/32/27559_2.png) [@toni.sm](https://forums.developer.nvidia.com/u/toni.sm)\ **Post date:** [October 1, 2020, 12:17pm UTC](https://forums.developer.nvidia.com/t/open-ai-gym-isaac-sim/156028/4 "2020-10-01T12:17:22Z")
Hi, I have not uploaded it to GitHub yet. But, I have plans to document my experience and upload it to discuss it with my supervisor next week :) ---
**Author:** ![Laurito](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/laurito/32/41732_2.png) [@Laurito](https://forums.developer.nvidia.com/u/Laurito)\ **Post date:** [October 1, 2020, 12:20pm UTC](https://forums.developer.nvidia.com/t/open-ai-gym-isaac-sim/156028/5 "2020-10-01T12:20:59Z")
great man! Keep me us up to date :) ---
**Author:** ![AshWang](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/ashwang/32/21110_2.png) [@AshWang](https://forums.developer.nvidia.com/u/AshWang)\ **Post date:** [October 7, 2020, 5:11am UTC](https://forums.developer.nvidia.com/t/open-ai-gym-isaac-sim/156028/6 "2020-10-07T05:11:09Z")
Hi @toni.sm, I’m also very interested in this project. Is the project approved for uploading to GitHub? ---
**Author:** ![ltorabi](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/ltorabi/32/14043_2.png) [@ltorabi](https://forums.developer.nvidia.com/u/ltorabi)\ **Post date:** [October 9, 2020, 2:40am UTC](https://forums.developer.nvidia.com/t/open-ai-gym-isaac-sim/156028/7 "2020-10-09T02:40:42Z")
Hello all, Unfortunately according to the agreement, Toni can’t share his code on github. But we truly appreciate his helps and useful answers here. Kindly, Liila ---
**Author:** ![Andrey1984](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/andrey1984/32/14130_2.png) [@Andrey1984](https://forums.developer.nvidia.com/u/Andrey1984)\ **Post date:** [November 16, 2020, 5:25pm UTC](https://forums.developer.nvidia.com/t/open-ai-gym-isaac-sim/156028/8 "2020-11-16T17:25:15Z")
I can see that the robot at the image is iiwa what is the full list f supported robots? are custom arms supported? Thanks ---
**Author:** ![toni.sm](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/toni.sm/32/27559_2.png) [@toni.sm](https://forums.developer.nvidia.com/u/toni.sm)\ **Post date:** [November 17, 2020, 8:17am UTC](https://forums.developer.nvidia.com/t/open-ai-gym-isaac-sim/156028/9 "2020-11-17T08:17:15Z")
Hi, The default robots in Isaac Sim 2020.2 are Carter, Franka panda, Kaya, UR10, and STR (Smart Transport Robot). You can find them in `Isaac Robotics > URDF` and the STR in `Isaac Robotics > Samples > Simple Robot Navigation` menu I added the Kuka iiwa (as shown in the image) from a URDF description. So, it is posiible to add custom robots if you have a very well formatted URDF… ---
**Author:** ![Andrey1984](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/andrey1984/32/14130_2.png) [@Andrey1984](https://forums.developer.nvidia.com/u/Andrey1984)\ **Post date:** [November 17, 2020, 8:31pm UTC](https://forums.developer.nvidia.com/t/open-ai-gym-isaac-sim/156028/10 "2020-11-17T20:31:04Z")
Thank you for followiing up!! Previously we tried to import urdf but without much luck > [@Urdf not visible in isaac sim](https://forums.developer.nvidia.com/t/urdf-not-visible-in-isaac-sim/143889/3): > > Hi Prateek Are there any errors or warnings in the terminal output when importing? Converting meshes to .obj can sometimes help as well. We are making improvements to our urdf importer for the next release, if you can provide a minimal urdf file + mesh that reproduces the issue that would allow us to add it to our list of validation tests. ---
**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\ **Post date:** [April 5, 2024, 3:36pm UTC](https://forums.developer.nvidia.com/t/open-ai-gym-isaac-sim/156028/11 "2024-04-05T15:36:18Z")
Closed on April 5, 2024, 3:36pm UTC