Camera in rl tasks - essential feature for robotics

Omniverse Isaac Sim
1

Hi I as wanting to start using cameras for some rl manipulation and navigation tasks by training a network to locate objects, horizon etc…
can this be done in Isaac sim/OIGE to take advantage of parallelism speed advantages?
Ideally I would like per environment cameras that don’t see other environments - but this previously was not possible Independent Cameras in RL
are there any examples of walls to do this - can you generate one way walls that are only opaque from 1 direction or from only a particular camera?

is it possible to do this and are there any examples?
Is it possible to take some layers from an existing neural net as a base model to speed up training and how would I implement that?

2

Hi there. Maybe have a look at the ray-caster camera in Isaac Orbit (in the devel branch, not officially released yet).

3

@btx0424 I had a look at orbit again… it seems a whole new workflow to learn! are there any video tutorials - its unclear if it allows the parallel workflows from OIGE?

4

I think yes. It is different from the camera originally provided by Isaac Sim but instead implemented with NVIDIA Warp.

I have been using the new version of Orbit for a while but mainly on legged locomotion. But I am glad to share my experience and have further discussions if you would like.

1 Like
6

Hi there, Isaac Sim does not currently provide independent cameras that don’t see other environments. One way to workaround it is to build walls around the environments, which would just be large rectangle prims that block the views of other environments. Another alternative would be to place the environments far apart, or on different height levels.

1 Like
7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.