Open AI Gym + ISAAC SIM

Omniverse Isaac Sim
1

Hi guys!
Is there a way to do some Reinforcement Learning by using Open AI gym in combination with Isaac Sim? Is there some information about it? Or where would you start?

Best Regards

2

Hi,

I would like to share my experience with reinforcement learning and Isaac Sim with the hope that it will be useful (Note: I use RLlib for Reinforcement Learning, but the approach works for other libraries)

First, to use a non-include python package (like Gym, OpenCV, TensorFlow, etc…) you must use the Isaac Sim’s python (not the system python) to install them. The Isaac Sim’ python path is /isaac-sim/_build/target-deps/kit_sdk_release/_build/target-deps/python

Then, you can use the python application (bin/python3) like the next example:

cd /isaac-sim/_build/target-deps/kit_sdk_release/_build/target-deps/python

bin/python3 -m pip install --upgrade pip

bin/python3 -m pip install gym
bin/python3 -m pip install pydot graphviz
bin/python3 -m pip install tensorflow
bin/python3 -m pip install ray[rllib] ray[debug]

Then, for RLlib, it is necessary to start the Ray cluster together with the simulator once (so, you can add the next code at the end of the python bindings for Isaac Sim extensions)

Edit

nano /isaac-sim/_build/target-deps/kit_sdk_release/_build/linux-x86_64/release/plugins/bindings-python/omni/ext/__init__.py

and add the next code at the end (modifying the CPUs and GPUs)

try:
    print("")
    print("+++++++++++++++++++++++++++")
    print("[RAY][TRY] ray.init...")
    import ray
    ray.init(num_cpus=20,
             num_gpus=4,
             local_mode=True,
             temp_dir="/isaac-sim/ray",
             configure_logging=True)
    print("[RAY][DONE] ray")
except Exception as e:
    import traceback
    print("[RAY][ERROR] traceback:", traceback.format_exc())
print("---------------------------")
print("")

After that, I created my own Reinforcement Learning sample (inside omni.isaac.samples: /isaac-sim/_build/linux-x86_64/release/exts/omni.isaac.samples/omni/isaac/samples/scripts) with the following structure (Gym environment, my robots and my RL architectures/models):

│   
│   rl_preview.py
│
├───rl_scenarios
│   │   quillen2018deep.py
│   │   scenario.py
│   │   __init__.py
│   │
│   ├───gym_scenarios
│   │   │   __init__.py
│   │   │
│   │   ├───envs
│   │   │   │   quillen2018deep_env.py
│   │   │   │   __init__.py
│   │
│   ├───models
│   │   │   ddpg_quillen2018deep.py
│   │   │   __init__.py
│   │
│   ├───robots
│   │   │   iiwa.py
│   │   │   pr2.py
│   │   │   robot.py
│   │   │   __init__.py
│   │

RL_quillen_.wmv_20201001_121147.229
RL_quillen_.wmv_20201001_121147.2291276×757 165 KB

2 Likes
3

Thanks for your reply @toni.sm

That’s a great starting point! I will try it out.
Do you have also a small example of this somewhere on github? (if allowed) I think many people would be interested that =)

4

Hi,

I have not uploaded it to GitHub yet. But, I have plans to document my experience and upload it to discuss it with my supervisor next week :)

2 Likes
5

great man! Keep me us up to date :)

6

Hi @toni.sm,
I’m also very interested in this project. Is the project approved for uploading to GitHub?

7

Hello all,

Unfortunately according to the agreement, Toni can’t share his code on github.
But we truly appreciate his helps and useful answers here.

Kindly,
Liila

8

I can see that the robot at the image is iiwa
what is the full list f supported robots?
are custom arms supported?
Thanks

9

Hi,

The default robots in Isaac Sim 2020.2 are Carter, Franka panda, Kaya, UR10, and STR (Smart Transport Robot). You can find them in Isaac Robotics > URDF and the STR in Isaac Robotics > Samples > Simple Robot Navigation menu

I added the Kuka iiwa (as shown in the image) from a URDF description. So, it is posiible to add custom robots if you have a very well formatted URDF…

10

Thank you for followiing up!!
Previously we tried to import urdf but without much luck