Train an agent with discrete action

Hello,

I want to train an agent with discrete action space, but I don’t know how to implement.
It seems that all samples in IsaacGym have continuous action space, so I am confused.
Is there any good example codes?

Thank you very much.

2 Likes

After trying to do the same unsuccessfully, I started debugging the errors which lead to inspecting the source files provided in the Isaac Gym implementation. Finally found the solution!

Mind that this solution is for a custom task class which derives from the RLTask class:

  1. Inside the initializer of your custom task, before calling RLTask.__init__(self, name, env), you should specify the action and observation spaces: self.action_space, self.observation_space.

NOTE: As per the a2c_common.py file, on line 893, you need to use a gym.spaces.Discrete or a gym.spaces.Tuple as an action space. It will not work with any other gym space and it will not work if you use gymnasium.spaces. It NEEDS to be gym.spaces.

  1. Inside your ...PPO.yaml config file, the algorithm name should be a2c_discrete and the model name should be discrete_a2c. Inside the network config, the space should be discrete. It can be continuous, discrete and multi_discrete.

NOTE: As per the torch_runner.py and model_builder.py files, the available algorithms, models and networks you can use in the config file are:

  • Algorithms: a2c_continuous, a2c_discrete and sac.

  • Models: discrete_a2c, multi_discrete_a2c, continuous_a2c, continuous_a2c_logstd, soft_actor_critic, central_value.

  • Networks: actor_critic, resnet_actor_critic, rnd_curiosity, soft_actor_critic.

I do not know if this is covered somehow in the documentation but I could not find it anywhere. If someone knows the links to the documentation that explain this, please share.

1 Like