OmniGraph Warning: Required extended attribute outputs:samples is not resolved. while looping

I am using replicator to produce synthetic data. I am looping through a range and creating a new stage for each loop. This is so that I can get the exact same physics result for each seed. This works well some of the time. Some times I get the following warning which results in the objects not receiving the rigid bodies attributes. I know that the documentation says “Physics functionalities of Replicator API are currently only available on linux.” I am using windows. If there is a problem with windows, can you tell me if this is one of the things that does not work? The data generation works for most seeds and I do process the data on Linux as well, (docker in the cloud) I am still getting the same intermittent bad results on Linux. The seeds that are affected are random, (not the same one every time it’s run).

2023-02-27 23:13:45 [29,978ms] [Warning] [omni.graph.core.plugin] /Replicator/SDGPipeline/OgnSampleUniform_05: OmniGraph Warning: Required extended attribute outputs:samples is not resolved, compute skipped
                   (from compute() at line 208 in c:\users\mark\workspaceov\syntheticdata-ov\exts\theory.syntheticdata.replicator\app\extscache\omni.replicator.core-1.6.3+104.1.wx64.r.cp37\omni\replicator\core\ogn\OgnSampleUniformDatabase.py)
2023-02-27 23:13:45 [29,980ms] [Warning] [omni.graph.core.plugin] /Replicator/SDGPipeline/OgnWritePhysics: OmniGraph Warning: Required extended attribute inputs:values is not resolved, compute skipped
                   (from compute() at line 184 in c:\users\mark\workspaceov\syntheticdata-ov\exts\theory.syntheticdata.replicator\app\extscache\omni.replicator.core-1.6.3+104.1.wx64.r.cp37\omni\replicator\core\ogn\OgnWritePhysicsDatabase.py)

I have created a much smaller python file for you to replicate. It does not include seeding and other stuff that may not be relevant. The warning comes up at different loop indexes each time I run it.

from the command line in the folder that my extension resides I run the command:
app\omni.code.bat --no-window --exec "test_multi.py"

python file test_multi.py:

import asyncio

import omni.replicator.core as rep
import carb
import omni.kit


async def new_stage():
    carb.log_info("new stage")
    await omni.usd.get_context().new_stage_async()
    await omni.kit.app.get_app().next_update_async()
    stage = omni.usd.get_context().get_stage()
    stage.SetStartTimeCode(0)
    stage.SetEndTimeCode(100)


def create_replicator_nodes():
    with rep.new_layer():
        rep.create.cube(position=(0, 0, 0), semantics=[("class", "cube")])
        rep.create.cube(position=(0, 0, 0), semantics=[("class", "cube")])
        cubes = rep.get.prims(semantics=[("class", "cube")], path_pattern="/Replicator/*")

        camera = rep.create.camera(position=(0, 10, 0), rotation=(0, 0, 0))
        with rep.trigger.on_frame(num_frames=1):
            with cubes:
                ##################
                # rep.physics.rigid_body() produces error:
                # OmniGraph Error: WritePhysics Error: cannot reshape array of size 3 into shape (2,newaxis)
                # therefore have to create a ReplicatorItem class for velocity and angular_velocity
                ##################
                velocity = rep.distribution.uniform((0, 0, 0), (0, 0, 0))
                angular_velocity = rep.distribution.uniform((0, 0, 0), (0, 0, 0))
                rep.physics.rigid_body(velocity=velocity, angular_velocity=angular_velocity)

        render_product = rep.create.render_product(camera, (256, 192))
        render_products = [render_product]
        writer = rep.WriterRegistry.get("BasicWriter")
        writer.initialize(
            output_dir="out_multi",
            rgb=True,
            )
        writer.attach(render_products)


async def test_multi():
    # 100 frame delay to allow app to set itself up (FIX for OV Code startup behaviour)
    for i in range(100):
        await omni.kit.app.get_app().next_update_async()
    carb.settings.get_settings().set("/app/file/ignoreUnsavedOnExit", True)
    num_scenes = 10
    for a in range(num_scenes):
        carb.log_info(f"starting scene {a}")
        await new_stage()
        create_replicator_nodes()
        carb.log_info("starting run_until_complete_async")
        await rep.orchestrator.run_until_complete_async()
        carb.log_info(f"done scene {a}")

    carb.log_info("post quit")
    omni.kit.app.get_app().post_quit(0)


asyncio.ensure_future(test_multi())

Windows 11
RTX 3090
Code 2022.3
kit_20230227_161315.log (1.0 MB)

Log file attached

Thank you,

Mark Olson

Hello @mark148! I’ve forwarded to over to the dev team for further help!

1 Like

Hi Mark,

I ran this a few times on my linux machine and I got your Warning every run as well as the following error:

2023-03-07 19:21:31 [31,435ms] [Error] [asyncio] Task exception was never retrieved
future: <Task finished coro=<ScriptManager._pending_dirty_handler() done, defined at /home/hclever/.local/share/ov/pkg/deps/1eb06a429c5271adc9ef8e5307b0f59f/extscache/omni.kit.scripting-104.1.3+104.1.lx64.r.cp37/omni/kit/scripting/scripts/script_manager.py:250> exception=AttributeError("'NoneType' object has no attribute 'GetPrimAtPath'")>
Traceback (most recent call last):
  File "/home/hclever/.local/share/ov/pkg/deps/1eb06a429c5271adc9ef8e5307b0f59f/extscache/omni.kit.scripting-104.1.3+104.1.lx64.r.cp37/omni/kit/scripting/scripts/script_manager.py", line 259, in _pending_dirty_handler
    prim = stage.GetPrimAtPath(prim_path_added)
AttributeError: 'NoneType' object has no attribute 'GetPrimAtPath'

I’m going to try on a windows machine now (windows 10) and see what I get.

1 Like

Can you clarify:

The reason you are using

with rep.trigger.on_frame(num_frames=1):
and a for loop instead of something like

with rep.trigger.on_frame(num_frames=10): (or better yet with rep.trigger.on_time(interval=scene_length, num=10): )
with NO for loop

is because setting num_frames=1 is just a placeholder for how many times you want to run the entire operation in the for loop, so that each run is exactly the same regardless if num_frames=10000?

Otherwise why not run this? ( I get no errors with it)
test_multi_jf.py (2.2 KB)

Hello hclever,

thank you for your replies and for looking into this. Much appreciated!

I am using rep.trigger.on_time in my script to run physics for certain amount of time. I was just using on_frame as a simplified script as to not add any more physics stuff than needed to debug the warning.

The reason I am looping and creating a new stage each is so that I can reproduce the physics drop exactly for each seed.

For example I have 100 seeds to run, (1-100):
num_scenes = 100
scene_length = 1.0

When I find that seed 100 is having a visible error, for example an object is clipping another object and I want to recreate to debug…

Using the rep.trigger.on_time(interval=scene_length, num=num_scenes) method I have to run the whole script again waiting until seed 100 finishes to recreate exactly the same physics results.
I have a custom node which sets the seed each time the on_time node runs, (the set seed node runs right after the on_time node)

Using the loop, new_stage, rep.trigger.on_time(interval=scene_length, num=1) method results in the exact same physics result for the seed I am debugging. I can now just run seed 100 and the result will be exactly the same as when I ran the loop, new_stage with seeds 1 - 100.

This brings up another point related to predictable physics results:
running rep.trigger.on_time in the GUI, (not command line) results in the physics outcome being different from the images output. I have not recreated in a simpler script, but my rather large scripts have this result. Which makes it harder to debug physics drops using the GUI. I can post this in another thread as it is a different issue.

Thank you,

Mark Olson

Well this issue still present on windows but goes away with Code 2022.3.0 (has issue on 2022.3.1). weird, i’ll keep digging.

Shouldn’t you get the exact same physics results for each run, as long as you start each scene in num/num_frames the same way? Physics should be deterministic. Can you clarify what physics difference you see if you use the rep.trigger.on_time(interval=scene_length, num=num_scenes for each run?

-Henry

Hi Henry,

run the following code twice from the command line app\omni.code.bat --no-window --exec "test_multi.py", changing code as listed:

  1. as is
  2. uncomment # seed_list = [1, 2, 1] and change output_dir="out_multi_1" to another output

I would expect both the start frame and the end frame to be the same for each seed.
The start frame is the same but the final frame, (frame 0071 in this case) is not.

From this I conclude that the initial start situation, (objects, position and rotation) is exact, but the physics result at the final frame is different due to the stage, (cached prims, the first seed created some prims that have since been deleted for seed two, etc…) not being exact when running the second seed physics.

The first seed in the list is a throw away as it’s used for the preview.
Seed 1 in the first command line run will be rgb_0
Seed 1 in the second run will be rgb_1

import asyncio
from typing import List

import omni.replicator.core as rep
import carb
import omni.kit

import omni.graph.core as og


sequence = 0


@og.AutoFunc(module_name="omni.replicator")
def SetSeed() -> None:
    # Note 1: numSamples input currently required
    # Note 2: Only bundle output currently supported, this will be expanded in the future.

    # Use global to have access to a persistent `frame` variable
    seed_list = [1, 1, 2]
    # seed_list = [1, 2, 1]
    global sequence
    print(f"sequence: {sequence}")
    seed = seed_list[sequence]
    print(f"setting seed: {seed}")
    rep.set_global_seed(seed)
    sequence += 1


# Register functionality into replicator
def set_seed():
    return rep.utils.create_node("omni.replicator.SetSeed")


rep.randomizer.register(set_seed)


def create_replicator_nodes():
    SURFACE = 'omniverse://localhost/NVIDIA/Assets/Scenes/Templates/Basic/display_riser.usd'
    usd_file_names = ["003_cracker_box.usd", "004_sugar_box.usd", "005_tomato_soup_can.usd"]
    usd_paths = [
        f"omniverse://localhost/NVIDIA/Assets/Isaac/2022.2.1/Isaac/Props/YCB/Axis_Aligned/{f}" for f in usd_file_names
    ]
    num_scenes = 2
    scene_length = 3
    rep.settings.set_stage_up_axis("z")
    with rep.new_layer():
        rep.create.light(
            light_type="Sphere",
            temperature=6500,
            intensity=12000,
            position=(-100, -100, -100),
            scale=75,
            count=1
        )
        surface = rep.create.from_usd(SURFACE)
        with surface:
            rep.modify.pose(rotation=(90, 0, 0))
            rep.physics.collider()

        camera = rep.create.camera(position=(0, 10, 0), rotation=(0, 0, 0))
        with camera:
            rep.modify.pose(position=(50, 50, 250), look_at=surface)

        def env_props(mesh_count=50):
            instances = rep.randomizer.instantiate(
                usd_paths, size=mesh_count, mode="scene_instance", use_cache=True
            )
            with instances:
                rep.modify.pose(
                    position=rep.distribution.uniform((-5, -5, 50), (5, 5, 150)),
                    rotation=rep.distribution.uniform((0, -180, 0), (0, 180, 0)),
                    scale=100
                )

                rep.physics.rigid_body(
                    velocity=rep.distribution.uniform((0, 0, 0), (0, 0, 0)),
                    angular_velocity=rep.distribution.uniform((0, 0, 0), (0, 0, 0)))
            return instances.node
        rep.randomizer.register(env_props)

        count = rep.distribution.choice([15, 16, 17, 18])
        # seed_list = [1, 2, 3]
        with rep.trigger.on_time(interval=scene_length, num=num_scenes):
            with rep.utils.sequential():
                rep.randomizer.set_seed()
                rep.randomizer.env_props(mesh_count=count)
                # with camera:
                #     rep.modify.pose(position=rep.distribution.uniform((-50, 20, 100), (50, 50, 150)), look_at=surface)

        render_product = rep.create.render_product(camera, (256, 192))
        render_products = [render_product]
        writer = rep.WriterRegistry.get("BasicWriter")
        writer.initialize(
            output_dir="out_multi_1",
            rgb=True,
            )
        writer.attach(render_products)


async def test_multi():
    # 100 frame delay to allow app to set itself up (FIX for OV Code startup behaviour)
    for i in range(100):
        await omni.kit.app.get_app().next_update_async()
    carb.settings.get_settings().set("/app/file/ignoreUnsavedOnExit", True)
    create_replicator_nodes()
    await rep.orchestrator.run_until_complete_async()
    carb.log_info("post quit")
    omni.kit.app.get_app().post_quit(0)


asyncio.ensure_future(test_multi())

Are you sure about this? You are randomizing the initial position using:

position=rep.distribution.uniform((-5, -5, 50), (5, 5, 150)),

I changed this and made it bigger and it definitely alters the behavior seed to seed. The seed should only affect what is going on in replicator, since it is a replicator seed - it should not affect the physics. So if you don’t randomize the distribution it should have the same initial and final result.

That being said if you start everything at the same location with no randomization, it might have some floating point difference in positions in the initial frame that causes things to behave differently. The best way to check this is to print out the exact position of an object every step and check exactly what it is (better than rendering images of it)

I believe this is just documentation issue - it should work fine on both (as you saw, it works the same on both).

1 Like

I think I have it figured out.
You said, “The seed should only affect what is going on in replicator, since it is a replicator seed - it should not affect the physics”

This was not consistent with my outcomes when using rep.randomizer.instantiate. So I was puzzled. Then I did a test where I discovered that anytime an object is added using physics it changes the outcome. Even if the object is very far away from the other objects. When using instantiate there are cached objects which even though the current scene may not be using them they still affect the physics outcome.

I get an error when using use_cache=False in instantiate, so I’ll have to fix that.

I appreciate all of your posts that got me thinking…I have a path forward to fix my issue.

Thank you,

Mark Olson

btw,
If you rename a prim with physics on it, it will change the physics outcome.

Hi Mark,

I saw that you had things figured out so I put this issue on the back burner. Thanks for your in depth explanation – helpful to us in making our software better – I’m going to dig into this further in the coming weeks and see if I can find a better solution.

-Henry

1 Like

Warp How to add a two Collider object?
Why is Collider_01 not affected by particles?