Can't obtaing transform matrix applied by rep.modify.pose

Omniverse Synthetic Data Generation (SDG)
1

If I change the transform matrix of an object with rep.modify.pose I can’t obtain it back. I tried with different methods. Here I post 3 examples.

Example 1

from omni.isaac.kit import SimulationApp

simulation_app = SimulationApp(
    launch_config={"renderer": "RayTracedLighting", "headless": True}
)

import numpy as np
import omni.replicator.core as rep
from omni.isaac.core.utils import prims
import omni.isaac.core.utils.prims as prims_utils
import omni.usd


cone = rep.create.cone(position=(0.0, 0.0, 0.0), rotation=(0.0, 0.0, 0.0))
with cone:
    rep.modify.pose(
        position=rep.distribution.uniform((-1.0, -1.0, -1.0), (-0.5, -0.5, -0.5)),
        rotation=rep.distribution.uniform((90.0, 90.0, 90.0), (100.0, 100.0, 100.0)),
    )
prim_path = rep.utils.get_node_targets(cone.node, "inputs:prims")[0]
prim = omni.usd.get_context().get_stage().GetPrimAtPath(str(prim_path))
transform_matrix = np.array(omni.usd.get_world_transform_matrix(prim)).T
print(transform_matrix)
translation = prims_utils.get_prim_property(prim_path, "xformOp:translate")
quaternion = prims_utils.get_prim_property(prim_path, "xformOp:rotateXYZ")
print(translation, quaternion)
simulation_app.close()

Example 2

from omni.isaac.kit import SimulationApp

simulation_app = SimulationApp(
    launch_config={"renderer": "RayTracedLighting", "headless": True}
)

import numpy as np
import omni.replicator.core as rep
from omni.isaac.core.utils import prims
import omni.isaac.core.utils.prims as prims_utils
import omni.usd


prim_path = "/MyObjects/mycone"
prim = prims.create_prim(prim_path=prim_path, usd_path="/tmp/cone.usd")
cone = rep.get.prim_at_path(prim_path)
with cone:
    rep.modify.pose(
        position=rep.distribution.uniform((-1.0, -1.0, -1.0), (-0.5, -0.5, -0.5)),
        rotation=rep.distribution.uniform((90.0, 90.0, 90.0), (100.0, 100.0, 100.0)),
    )
transform_matrix = np.array(omni.usd.get_world_transform_matrix(prim)).T
print(transform_matrix)
translation = prims_utils.get_prim_property(prim_path, "xformOp:translate")
quaternion = prims_utils.get_prim_property(prim_path, "xformOp:orient")
print(translation, quaternion)
simulation_app.close()

Example 3

from omni.isaac.kit import SimulationApp

simulation_app = SimulationApp(
    launch_config={"renderer": "RayTracedLighting", "headless": True}
)

import numpy as np
import omni.replicator.core as rep
from omni.isaac.core.utils import prims
import omni.isaac.core.utils.prims as prims_utils
import omni.usd

The output of example 1 is:

[[1. 0. 0. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 0. 0. 1.]]
(0, 0, 0) (0, 0, 0)

The output of example 2 is:

[[1. 0. 0. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 0. 0. 1.]]
(0, 0, 0) (1, 0, 0, 0)

The output of example 3 is:

[[1. 0. 0. 0.]
 [0. 1. 0. 0.]
 [0. 0. 1. 0.]
 [0. 0. 0. 1.]]
None None

The file I used:
cone.usd (30.8 KB)

But if I use a different method to set the transform matrix with a different method it works. I show a modifed version of the example 2:

from omni.isaac.kit import SimulationApp

simulation_app = SimulationApp(
    launch_config={"renderer": "RayTracedLighting", "headless": True}
)

import numpy as np
import omni.replicator.core as rep
from omni.isaac.core.utils import prims
import omni.isaac.core.utils.prims as prims_utils
import omni.usd
import random
from pxr import Gf


prim_path = "/MyObjects/mycone"
prim = prims.create_prim(prim_path=prim_path, usd_path="/tmp/cone.usd")
cone = rep.get.prim_at_path(prim_path)
my_transform_matrix = np.eye(4)
my_transform_matrix[0, 3] = random.uniform(-1.0, -0.5)
my_transform_matrix[1, 3] = random.uniform(-1.0, -0.5)
my_transform_matrix[2, 3] = random.uniform(-1.0, -0.5)
transform = Gf.Transform(Gf.Matrix4d(my_transform_matrix.T))
rotation = transform.GetRotation()
quaternion = rotation.GetQuat()
translation = transform.GetTranslation()
prims_utils.set_prim_property(prim_path, "xformOp:translate", translation)
prims_utils.set_prim_property(prim_path, "xformOp:orient", quaternion)
transform_matrix = np.array(omni.usd.get_world_transform_matrix(prim)).T
print(transform_matrix)
translation = prims_utils.get_prim_property(prim_path, "xformOp:translate")
quaternion = prims_utils.get_prim_property(prim_path, "xformOp:orient")
print(translation, quaternion)
simulation_app.close()

The output is:

[[ 1.          0.          0.         -0.9777302 ]
 [ 0.          1.          0.         -0.92811792]
 [ 0.          0.          1.         -0.65131342]
 [ 0.          0.          0.          1.        ]]
(-0.9777302028157011, -0.9281179232026341, -0.6513134187091592) (1, 0, 0, 0)
2

Hello,

Sorry for the delay and for the confusion.

What Replicator is doing behind-the-scenes is setting up an OmniGraph Action Graph that will later execute during simulation.

So, somewhat unintuitively, when you created your cone object, it has not actually been modified yet during script execution, since the script is executed “all-at-once” and simulation/graph execution takes place after.

A “simple” way to see this is to print the transform of your cone in a second script execution, because in-between script executions, the OmniGraph has had time to execute (automatically) and set a value for your transform -which is what you see happen in the viewport, it just happens fast enough that it seems instant.

If you need to extract information from your replicator-created items within the same script execution, you can add-in a simulation step trigger such as rep.orchestrator.step() or also through OmniGraph directly via og.Controller.evaluate_sync().

Example script:

import numpy as np
import omni.replicator.core as rep
import omni.usd
from omni.usd._impl.utils import get_prim_at_path
from pxr import Sdf
import omni.graph.core as og

with rep.new_layer():
    cone = rep.create.cone(position=(1,1,1))
    prim_path = "/Replicator/Cone_Xform"
    cone_prim = get_prim_at_path(Sdf.Path(prim_path))
    transform_matrix = np.array(omni.usd.get_world_transform_matrix(cone_prim)).T
    print(transform_matrix) # print 1

    with cone:
        rep.modify.pose(
            position=rep.distribution.uniform((-1.0, -1.0, -1.0), (1.0, 1.0, 1.0)),
            rotation=rep.distribution.uniform((0.0, 0.0, 0.0), (100.0, 100.0, 100.0)),
        )

    transform_matrix = np.array(omni.usd.get_world_transform_matrix(cone_prim)).T
    print(transform_matrix)  # print 2

    og.Controller.evaluate_sync()
    print("After Sync")  # print 3
    transform_matrix = np.array(omni.usd.get_world_transform_matrix(cone_prim)).T
    print(transform_matrix)

My output:

# first print
[[1. 0. 0. 0.]
[0. 1. 0. 0.]
[0. 0. 1. 0.]
[0. 0. 0. 1.]]

# Second print
# In the script, the cone position "seems" like it has been modified, but as you can see it has not
# because the omni graph has not executed a simulation step yet
[[1. 0. 0. 0.]
[0. 1. 0. 0.]
[0. 0. 1. 0.]
[0. 0. 0. 1.]]

After Sync
[[ 0.92044801  0.3848541   0.06828459 -0.80145752]
[ 0.02861739  0.10787761 -0.99375222  0.25251219]
[-0.38981599  0.91665137  0.08828221  0.61443901]
[ 0.          0.          0.          1.        ]]

We know this is a bit unintuitive and cumbersome at the moment, and we are currently working on making API methods in Replicator that will make these types of queries about the state of the simulation easier to access.