Replicator Composer: Scenario classes not showing up in segmentation view

(Similar to a previous issue, but now using version 2022.1.0)

I’m setting myself up for Panoptic segmentation. For that I need segmentation of all the classes I’m interested in, and instance segmentation of all instances (but with a way to find out what class they belong to.)

This could be user error of course, but I’m still stuck with:

  • Segmentation view correctly shows the object I’ve specified (see below).
  • Segmentation view does not show all the classes from the underlying scenario_model. I’ve tried with and without specifying a scenario_class_id
  • Instance view seems correct - showing both the object(s) I’ve added and the underlying scenario_model. However, without a full semantic view, I have no knowledge of what class an instance belongs to.
  • (Minor issue - the forklift object is treated differently in semantic vs. instance. In semantic view it’s one class, in instance view it’s two instances - the forklift truck and the actual fork assembly are different.) Also, in previous version the forklift truck got the same instance value as the floor patch near it - I still need to check that…

Following:

  • RGB
  • Semantic
  • Instance
  • yaml file (and invocation instructions)

forklift.yaml (1.4 KB)

(text of yaml pasted in…)

Simple, forklift only - no people

invoke with

./python.sh tools/composer/src/main.py --input */parameters/take1/simp-Forklift.yaml --output */dataset/take2-fork-1 --mount ~/composer-workspace --num-scenes 50

object_forklift:
obj_model: /Isaac/Props/Forklift/forklift.usd
obj_count: 1
obj_size_enabled: False
obj_scale: 1
obj_coord_camera_relative: True
obj_distance: 1400
obj_horiz_fov_loc: Uniform(-1,1)
obj_vert_fov_loc: 0.04 # floats a little, but none in the floor
obj_rot: (0, 0, Uniform(0, 360))
obj_class_id: 2
obj_physics: False # if True, person just falls to floot

colorful ceiling lights

lights:
light_count: Range(0, 2)
light_coord_camera_relative: False
light_coord: (Uniform(-200, 200), Uniform(-200, 200), 500)
light_color: Uniform((0, 0, 0), (255, 255, 255))
light_intensity: Uniform(0, 300000)
light_radius: 100

scenario_room_enabled: False
scenario_model: /Isaac/Environments/Simple_Warehouse/warehouse.usd

scenario_class_id: 0 # Tried both ways

camera

focal_length: 30
camera_coord: (900, Uniform(-200,200), 100) # 950 too far
camera_rot: (0, 0, Uniform(150,200))

output

output_dir: dataset
num_scenes: 10
img_width: 1280
img_height: 720

output types

rgb: True
depth: True
semantic_seg: True
instance_seg: True
groundtruth_visuals: True

model store

nucleus_server: localhost

sequential: False
sequence_step_count: 1
sequence_step_time: 0.1

Thanks for addressing the instance view problem (Replicator composer: instance view is incorrect (?) - #5 by jlafleche)

However, the show stopper is the segmentation view. The issue is that I wish to see a full segmentation view of all items in the scenario. Currently only the object I added and assigned an id to is shown. Without all the items, I can’t process the instance view (as I need a lookup to get the class, via, duh the segmentation view.)

Thx again!
p

Hello @peter.gaston, please find attached a new version of tools/composer/src/output/output.py to try where the semantic segmentation remapping function is commented out. Note that in so doing, we’re allowing all the scenario semantics to come through, including the forklift forks which have their own semantic label. I suspect this is not quite what you’re after either so please do reach out with feedback of further changes we can make to provide greater flexibility in the type of output generated.
output.py (11.1 KB)

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.