Isaac Sim 5.0.0 - Pose Estimation Data Generation

Omniverse Synthetic Data Generation (SDG)
1

Hi there,

I’m trying to generate pose estimation dataset. I have own scene and object in USD file. I uploaded usd file and added semantic but the camera doesn’t capture anything.

How should I solve this problem?

Thanks!

2

I’ve attached code below:

import omni.replicator.core as rep

from isaacsim.core.utils.stage import open_stage

from isaacsim.core.utils.prims import get_prim_at_path

from isaacsim.core.utils.semantics import add_labels

import omni.usd

usd_path = “/home/woojung/isaac/field/output/field_scene_fixed.usd”

open_stage(usd_path)

stage = omni.usd.get_context().get_stage()

with rep.new_layer():

camera = rep.create.camera( focus_distance=400.0, focal_length=24.0, clipping_range=(0.1, 10000000.0), name="Cam")

render_product = rep.create.render_product(camera, (1024, 1024))    

target_path = "/World/Strawberries/Strawberry_01/Geometry/Twig_d3/Flower_stalk_1/Receptacle" 

strawberry = stage.GetPrimAtPath(target_path)

add_labels(strawberry, labels= \["strawberry"\], instance_name= "class")

rgb_writer = rep.WriterRegistry.get("BasicWriter")

rgb_writer.initialize(

output_dir=“/home/woojung/debug/rgb”,

rgb=True

)

rgb_writer.attach(\[render_product\])

with rep.trigger.on_frame(max_execs=10):

with camera:

        rep.modify.pose(

position=rep.distribution.uniform((-500, 200, -500), (500, 500, 500)),

look_at=strawberry

        )

rep.orchestrator.run()

3

@w.son i am just another user, but can you confirm if this is only happening to your custom assets or including NVIDIA assets shipped with Isaac Sim? would you able to take a screenshot of what the USD stage looks like on the custom USD asset?

4

Hi @Simplychenable ,

I think it’s happening only for my case.

I’ve attached screenshots of what the stage, layer is and error logs

image
image556×467 32.4 KB

image
image556×155 7.87 KB

image

5

I tried to run the example code, but it still doesn’t include anything.

The code example is below:

import omni.replicator.core as rep

with rep.new_layer():

camera = rep.create.camera(position=(0, 0, 1000))



render_product = rep.create.render_product(camera, (1024, 1024))



torus = rep.create.torus(semantics=\[('class', 'torus')\] , position=(0, -200 , 100))



sphere = rep.create.sphere(semantics=\[('class', 'sphere')\], position=(0, 100, 100))



cube = rep.create.cube(semantics=\[('class', 'cube')\],  position=(100, -200 , 100) )

with rep.trigger.on_frame(max_execs=10):

with rep.create.group([torus, sphere, cube]):

        rep.modify.pose(

position=rep.distribution.uniform((-100, -100, -100), (200, 200, 200)),

scale=rep.distribution.uniform(0.1, 2))

# Initialize and attach writer

writer = rep.WriterRegistry.get("BasicWriter")



writer.initialize( output_dir="/home/woojung/debug", rgb=True,   bounding_box_2d_tight=True)



writer.attach(\[render_product\])



rep.orchestrator.preview()
6

i believe that’s the snippet from the tutorial found in the following page, correct?

if so, try using the modified snippet i have included in another post where i increased the size and added rotation to the camera that helped with the visibility of the objects:

7

if the scaling on the tutorial snippet worked for you, it’s likely not an issue with replicator itself. the console seems to only include warnings, which don’t always result in an error. i wonder whether the size of your custom asset or the distance of those assets from the camera could be contributing to the issue?

8

I tried the code that you provided. When the camera collect data, it includes objects and labels bounding box well. But the rgb images doesn’t include anything.

image
image1232×656 28.9 KB

image
image722×503 30.5 KB

image
image748×166 25.7 KB

9

set your viewport to use Default lighting rig before generating the render products:

image
image367×444 19.7 KB

otherwise, if you have an empty scene and are using Stage Lights lighting rig, you likely won’t be able to see anything since nothing will be lit and will appear completely black in your outputs

10

In my case, the images doesn’t appear black, images look like this.

image
image865×875 4.47 KB

I added scene or light and changed dfault lighting, but it doesn’t still work.

Also, there’s no error logs in this case.

11

that’s a new one. here’s a recap of what i did:

if you were to follow the steps exactly, do you still get blank outputs?

12

Thank you for your video. I did the same way you gave, but it still doesn’t include.

I don’t understand why my case doesn’t work.

image
image728×103 19.3 KB

I downloaded sdk kit and run the Issac Sim through ./isaac-sim.sh command. Is this problem?

13

hmm, i don’t think that’s an issue. have you tried this in other versions of Isaac Sim in the past where it worked? 4.5 or 5.1?

14

I tried it on Isaac Sim 4.5.0. Finally, it worked.

Thank you so much, I really appreciate it!

15

@w.son i wonder what it could be thats causing the issue on your end. just know that 4.5 is going to be EOL in a few release cycles, so if it’s a bug in 5.0, would be great if the dev can chime in here. or give 5.1 a shot since it’s more current.

16

@Simplychenable

I tried it on Isaac Sim 5.0.1 this time, It also happened problem. hmm..

17

what’s your GPU and driver version, by the way?

18

I have two GPUs, RTX 3090 and RTX 4090. And the driver version is 545.29.06

19

another thing to consider is getting the specific driver version suggested by the mods in another thread (linked below) and see if the issue persists in 5.0:

20

Hi,

I reinstalled NVDIA driver to 570.169. And the issue has been solved on Isaac Sim 5.0.1.

Thank you so much!