SKRL: a modular reinforcement learning library with support for NVIDIA Omniverse Isaac Gym

Dear community,

skrl is an open-source modular library for Reinforcement Learning written in Python (using PyTorch) and designed with a focus on readability, simplicity, and transparency of algorithm implementation. In addition to supporting the OpenAI Gym and DeepMind environment interfaces, it allows loading and configuring NVIDIA Isaac Gym and NVIDIA Omniverse Isaac Gym environments, enabling agents’ simultaneous training by scopes (subsets of environments among all available environments), which may or may not share resources, in the same run

Please, visit the documentation for usage details and examples

https://skrl.readthedocs.io/en/latest/


The current version 0.6.0 is now available (it is under active development. Bug detection and/or correction, feature requests and everything else are more than welcome: Open a new issue on GitHub! ). Please refresh your browser (Ctrl + Shift + R or Ctrl + F5) if the documentation is not displayed correctly

This new version has focused on supporting the training and evaluation of reinforcement learning algorithms in NVIDIA Omniverse Isaac Gym

Added

  • Omniverse Isaac Gym environment loader
  • Wrap an Omniverse Isaac Gym environment
  • Save the best models during training
  • Omniverse Isaac Gym examples
4 Likes

Dear community,

skrl version 0.7.0 is now available (it is under active development. Bug detection and/or correction, feature requests and everything else are more than welcome: Open a new issue on GitHub! ). Please refresh your browser (Ctrl + Shift + R or Ctrl + F5) if the documentation is not displayed correctly

Added

  • A2C agent
  • Isaac Gym (preview 4) environment loader
  • Wrap an Isaac Gym (preview 4) environment
  • Support for OpenAI Gym vectorized environments
  • Running standard scaler for input preprocessing
  • Installation from PyPI (pip install skrl)

Now, with the implementation of the standard scaler (adapted from rl_games), better performance is achieved…

E.g, for the Ant environment

[ORANGE]: PPO agent with input preprocessor
[BLUE] PPO agent without input preprocessors

Exciting work, thanks!
Can you add some algorithms and environments for multi-agent reinforcement learning?

Hi @Mr.Fox

Thank you for giving the library a try.

The idea is to add more algorithms to the library gradually :)

Regarding environments, the development is focused on adding algorithms and functionalities to work with the relevant environments and interfaces in the RL field (such as Omniverse Isaac Gym, Isaac Gym, OpenAI Gym, or DeepMind, for example). Creating or providing environments as part of the library is not on my roadmap at the moment.

1 Like

Dear community,

skrl version 0.8.0 is now available (it is under active development. Bug detection and/or correction, feature requests, and everything else is more than welcome: Open a new issue on GitHub! ). Please refresh your browser (Ctrl + Shift + R or Ctrl + F5) if the documentation is not displayed correctly

Added

  • AMP agent for physics-based character animation
  • Manual trainer
  • Gaussian model mixin
  • Support for creating shared models
  • Parameter role to model methods
  • Wrapper compatibility with the new OpenAI Gym environment API (by @JohannLange)
  • Internal library colored logger
  • Migrate checkpoints/models from other RL libraries to skrl models/agents
  • Configuration parameter store_separately to agent configuration dict
  • Save/load agent modules (models, optimizers, preprocessors)
  • Set random seed and configure deterministic behavior for reproducibility
  • Benchmark results for Isaac Gym and Omniverse Isaac Gym on the GitHub discussion page
  • Franka Emika real-world example

Changed

  • Models implementation as Python mixin [breaking change]
  • Multivariate Gaussian model (GaussianModel until 0.7.0) to MultivariateGaussianMixin
  • Trainer’s cfg parameter position and default values
  • Show training/evaluation display progress using tqdm (by @JohannLange)
  • Update Isaac Gym and Omniverse Isaac Gym examples

Fixed

  • Missing recursive arguments during model weights initialization
  • Tensor dimension when computing preprocessor parallel variance
  • Models’ clip tensors dtype to float32

Removed

  • Parameter inference from model methods
  • Configuration parameter checkpoint_policy_only from agent configuration dict

As a showcase for the basic Franka Emika real-world example, a simulated version of the environment for Isaac Gym and Omniverse Isaac Gym are provided to support advanced implementations :)



3 Likes

Thank you for sharing this information, it is very helpful

Dear community,

skrl version 0.9.0 is now available (it is under active development. Bug detection and/or correction, feature requests, and everything else is more than welcome: Open a new issue on GitHub! ). Please refresh your browser (Ctrl + Shift + R or Ctrl + F5) if the documentation is not displayed correctly

Added

  • Support for Farama Gymnasium interface
  • Wrapper for robosuite environments
  • Weights & Biases integration (by @juhannc)
  • Set the running mode (training or evaluation) of the agents
  • Allow clipping of the gradient norm for DDPG, TD3, and SAC agents
  • Initialize model biases
  • Add RNN (RNN, LSTM, GRU, and any other variant) support for A2C, DDPG, PPO, SAC, TD3, and TRPO agents
  • Allow disabling training/evaluation progressbar
  • Farama Shimmy and robosuite examples
  • KUKA LBR iiwa real-world example

Changed

  • Forward model inputs as a Python dictionary [breaking change]
  • Returns a Python dictionary with extra output values in model calls [breaking change]
  • Adopt the implementation of terminated and truncated over done for all environments

Fixed

  • Omniverse Isaac Gym simulation speed for the Franka Emika real-world example
  • Call agents’ method record_transition instead of the parent method
    to allow storing samples in memories during the evaluation
  • Move TRPO policy optimization out of the value optimization loop
  • Access to the categorical model distribution
  • Call reset only once for Gym/Gymnasium vectorized environments

Removed

  • Deprecated method start in trainers

As a showcase, a basic real-world example is provided where a KUKA LBR iiwa robot is controlled through a direct API and via ROS/ROS2. In addition, its simulated version, in Omniverse Isaac Gym, is provided to support advanced implementations :)

Dear community,

skrl version 0.10.0 is now available (it is under active development. Bug detection and/or correction, feature requests, and everything else is more than welcome: Open a new issue on GitHub! ). Please refresh your browser (Ctrl + Shift + R or Ctrl + F5) if the documentation is not displayed correctly

This unexpected new version has focused on supporting the training and evaluation of reinforcement learning algorithms in NVIDIA Isaac Orbit

Added:

  • Isaac Orbit environment loader
  • Wrap an Isaac Orbit environment
  • Gaussian-Deterministic shared model instantiator

2 Likes

Dear developer,
I am currently using the multi-env framework with the skrl algorithm library. However, it’s strange that the performance of skrl is slightly worse than that of stable baselines3. I’m not sure if this is due to my configuration. Additionally, when clip_action=False, the actions outputted by skrl sometimes exceed the action space that I’ve defined. If clip_action=True, the actions will be clipped to the boundary values of -1 or 1. Could you please help me clarify these issues? Thank you very much.

Hi @2456496590

Can you please open a new discussion in the skrl repository to explore the topic there?
So we leave this post for announcements :)

1 Like

Dear community,

skrl version 1.0.0-rc.1 is now available (it is under active development. Bug detection and/or correction, feature requests, and everything else is more than welcome: Open a new issue on GitHub! ). Please refresh your browser (Ctrl + Shift + R or Ctrl + F5) if the documentation is not displayed correctly.

Among the major features of this new version are:

  • JAX support
  • Multi-agent training (the beginning).
  • Comprehensive documentation with new structure and theme

Visit the library documentation to start training your favorite task in Isaac Gym preview, Isaac Orbit or Omniverse Isaac Gym using PyTorch or JAX!

For questions and others, please open a new discussion in the skrl repository.
That way we leave this topic for announcements :)




The execution times used to construct this graph are the complete execution of the script
(including environment loading and, in the case of JAX, the jit compilation of XLA, for example).

2 Likes

Just came across this very very cool!