SKRL: a modular reinforcement learning library with Isaac Gym environments support

Dear community,

I would like to share, in this topic and in a more official way, the RL library (previously mentioned in this post) that we are developing/using in our lab…

skrl is an open-source modular library for Reinforcement Learning written in Python (using PyTorch) and designed with a focus on readability, simplicity, and transparency of algorithm implementation. In addition to supporting the Gym interface, it allows loading and configuring NVIDIA Isaac Gym environments, enabling agents’ simultaneous training by scopes (subsets of environments among all available environments), which may or may not share resources, in the same run."

Please, visit the documentation for usage details and examples

https://skrl.readthedocs.io/en/latest/

Notes:

  • This project is under active continuous development (please make sure you always have the latest version).
  • Bug detection and/or correction, feature requests and everything else are more than welcome :)
  • Please refresh your browser (Ctrl + R) if the API is not displayed correctly in the documentation
8 Likes

Dear community,

skrl version 0.3.0 is now available (it is under active development. Bug detection and/or correction, feature requests and everything else are more than welcome: Open a new issue on GitHub!)

Added:

  • DQN and DDQN agents
  • Export memory to files
  • Postprocessing utility to iterate over memory files
  • Model instantiator utility to allow fast development
  • More examples and contents in the documentation
3 Likes

Hi @toni.sm,

Thanks for sharing your RL library and for the direct support of Isaac Gym environments!

1 Like

Glad to contribute 😁

1 Like

Dear community,

skrl version 0.4.0 is now available (it is under active development. Bug detection and/or correction, feature requests and everything else are more than welcome: Open a new issue on GitHub! ). Please refresh your browser (Ctrl + Shift + R) if the documentation is not displayed correctly

Added

  • CEM, SARSA and Q-learning agents
  • Tabular model
  • Parallel training using multiprocessing
  • Isaac Gym utilities

Changed

  • Initialize agents in a separate method
  • Change the name of the networks argument to models

Fixed

  • Reset environments after post-processing

As part of the Isaac Gym utilities, a lightweight web viewer is available for development without X server

3 Likes

Dear community,

skrl version 0.5.0 is now available (it is under active development. Bug detection and/or correction, feature requests and everything else are more than welcome: Open a new issue on GitHub! ). Please refresh your browser (Ctrl + Shift + R or Ctrl + F5) if the documentation is not displayed correctly

Added

  • TRPO agent
  • DeepMind environment wrapper
  • KL Adaptive learning rate scheduler
  • Handle gym.spaces.Dict observation spaces (OpenAI Gym and DeepMind Gym environments)
  • Forward environment info to agent record_transition method
  • Expose and document the random seeding mechanism
  • Define rewards shaping function in agents’ config
  • Define learning rate scheduler in agents’ config
  • Improve agent’s algorithm description in documentation (PPO and TRPO at the moment)

Changed

  • Compute the Generalized Advantage Estimation (GAE) in agent _update method
  • Move noises definition to resources folder
  • Update the Isaac Gym examples

Removed

  • compute_functions for computing the GAE from memory base class

Now, with the implementation of the learning rate scheduler and the reward shaper (both adapted from rl_games), a comparable performance with rl_games is achieved…

For example, for the Cartpole environment (RTX 3080)

Hi,

Thanks for sharing your RL library! I am researching different options and I am wondering how skrl compares to for example Stable-Baselines. Why did you decide to develop a new library instead of writing support for IsaacGym envs in Stable-Baselines?

The web viewer you developed looks great and will be very useful!

Hi @mjvanderboon

Nice question…

The decision on the creation of a new library is described in the skrl statement:

skrl is an open-source modular library for Reinforcement Learning written in Python (using PyTorch) and designed with a focus on readability, simplicity, and transparency of algorithm implementation. In addition to supporting the OpenAI Gym and DeepMind environment interfaces, it allows loading and configuring NVIDIA Isaac Gym environments, enabling agents’ simultaneous training by scopes (subsets of environments among all available environments), which may or may not share resources, in the same run


  • I wanted to have an RL library where the implementation of the algorithms would be readable, transparent, and uncomplicated. This with the goal of allowing me, and anyone else who wants to give the library a chance to understand and access the code without having to navigate a deep hierarchy of files

    For example in Stable-Baselines, if you want to understand the DDPG you need to inspect:

    ddpp/ddpg.py -> td3/td3.py -> common/off_policy_algorithm.py -> common/base_class.py

  • I wanted each algorithm to be implemented independently, even when they might share most of their code with other algorithms. For example the case of DDPG and TD3 or DQN and DDQN

  • I wanted to have a modular and a well-organized library, where each file was separated and classified according to its functionality and not stacked in a common folder

  • I wanted to have a library written end-to-end using the same machine learning framework and with the possibility of scaling to other frameworks (like Tensorflow) in a simple and organized way

  • I wanted to exploit the amazing features offered by Isaac Gym without neglecting the classics

  • I wanted to learn from the experience of developing something meaningful to me and share it with the community in the hope that it might be useful to someone else in some way or another.

I wanted to do all that and much more, and at that moment the best I can give is skrl, which is not even close to being half good, but it has all my effort and dedication. And… if I wanted to do all that with the current implementations I had to rewrite all the code, and the way I found to do it was to start from scratch.

Hi @toni.sm

Thanks for your reply and clearing up the design methodology differences between skrl and stable-baselines. For sure, skrl looks very well-organized and modular, also when compared to stable-baselines. Thank you for the time you put into developing skrl and for sharing it with the community! I will be sure to test it out :)

Dear community,

skrl version 0.6.0 is now available (it is under active development. Bug detection and/or correction, feature requests and everything else are more than welcome: Open a new issue on GitHub! ). Please refresh your browser (Ctrl + Shift + R or Ctrl + F5) if the documentation is not displayed correctly

Added

  • Omniverse Isaac Gym environment loader
  • Wrap an Omniverse Isaac Gym environment
  • Save the best models during training
  • Omniverse Isaac Gym examples

Dear community,

skrl version 0.7.0 is now available (it is under active development. Bug detection and/or correction, feature requests and everything else are more than welcome: Open a new issue on GitHub! ). Please refresh your browser (Ctrl + Shift + R or Ctrl + F5) if the documentation is not displayed correctly

Added

  • A2C agent
  • Isaac Gym (preview 4) environment loader
  • Wrap an Isaac Gym (preview 4) environment
  • Support for OpenAI Gym vectorized environments
  • Running standard scaler for input preprocessing
  • Installation from PyPI (pip install skrl)

Now, with the implementation of the standard scaler (adapted from rl_games), better performance is achieved…

E.g, for the Ant environment

[ORANGE]: PPO agent with input preprocessor
[BLUE] PPO agent without input preprocessors

1 Like