Different human .usd can't generate skeleton annotation

Omniverse Synthetic Data Generation (SDG)
1

Dear OV users!
I’m trying to generate annotation data for skeleton information of humans.
Looks like if I use 1 or more Workers from Omniverse assets, it looks OK.
It works as well if I use the same “human” repeated multiple times:

HUMANS = [
    'omniverse://localhost/NVIDIA/Assets/Characters/Reallusion/Worker/Worker.usd',
    'omniverse://localhost/NVIDIA/Assets/Characters/Reallusion/ActorCore/Business_F_0002/business-f-0002.usd',
    'omniverse://localhost/NVIDIA/Assets/Characters/Reallusion/ActorCore/Party_M_0001/party-m-0001.usd',
    'omniverse://localhost/NVIDIA/Assets/Characters/Reallusion/ActorCore/Uniform_F_0001/uniform_f_0001.usd',
    'omniverse://localhost/NVIDIA/Assets/Characters/Reallusion/ActorCore/Uniform_M_0001/uniform_m_0001.usd'
]

But it doesn’t work if I use multiple “humans” at the same time (like a woker+businesswoman)

Here the log of the error:
log.txt (30.5 KB)

Error part from log.txt 
2024-01-24 03:11:16 [8,186ms] [Error] [omni.kit.app._impl] [py stderr]: /home/staff/.local/share/ov/pkg/deps/8021e278fce7781f5bb307a5b4b24ed4/extscache/omni.graph-1.82.1+818d8fa0.lx64.r.cp310/omni/graph/core/_impl/attribute_values.py:372: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  self.__attribute.set(new_value, on_gpu=on_gpu)

2024-01-24 03:36:13 [10,018ms] [Error] [omni.graph.core.plugin] /Render/PostProcess/SDGPipeline/RenderProduct_Replicator_skeleton_data: Assertion raised in compute - Unable to write from unknown dtype, kind=O, size=8
  File "/home/staff/.local/share/ov/pkg/deps/8021e278fce7781f5bb307a5b4b24ed4/extscache/omni.replicator.core-1.9.8+105.0.lx64.r.cp310/omni/replicator/core/ogn/python/_impl/nodes/OgnGetSkeletonData.py", line 448, in compute
    db.outputs.jointOcclusions = [s["joint_occlusions"] for s in skels_data]
  File "/home/staff/.cache/ov/ogn_generated/1.31.1/omni.replicator.core-1.9.8/omni.replicator.core/ogn/OgnGetSkeletonDataDatabase.py", line 668, in __setattr__
    super().__setattr__(item, new_value)
  File "/home/staff/.local/share/ov/pkg/deps/8021e278fce7781f5bb307a5b4b24ed4/extscache/omni.graph-1.82.1+818d8fa0.lx64.r.cp310/omni/graph/core/_impl/database.py", line 692, in __setattr__
    super().__setattr__(item, new_value)
  File "/home/staff/.cache/ov/ogn_generated/1.31.1/omni.replicator.core-1.9.8/omni.replicator.core/ogn/OgnGetSkeletonDataDatabase.py", line 418, in jointOcclusions
    data_view.set(value)
  File "/home/staff/.local/share/ov/pkg/deps/8021e278fce7781f5bb307a5b4b24ed4/extscache/omni.graph-1.82.1+818d8fa0.lx64.r.cp310/omni/graph/core/_impl/attribute_values.py", line 372, in set
    self.__attribute.set(new_value, on_gpu=on_gpu)

The following minimum the code is run with headless mode, but i have the same issue inside Omniverse Code using script/extension:
The code sample_generate_skeleton.py.txt (1.5 KB):

minimum code sample_generate_skeleton.py.

Run with:
/home/${USER}/.local/share/ov/pkg/code-2023.1.1/omni.code.sh --no-window --/omni/replicator/script=${SCRIPT_DIR}/sample_generate_skeleton.py

The code:

# /home/staff/dev/omniverse_test/sample_generate_skeleton.py
import os
import shutil
import omni.replicator.core as rep

PROJECT_FOLDER = os.path.dirname(os.path.realpath(__file__))
OUTPUT_DIR = os.path.join(PROJECT_FOLDER,"output_ov_report_20240124")
shutil.rmtree(OUTPUT_DIR,ignore_errors=True)

WORKER_SRC = 'omniverse://localhost/NVIDIA/Assets/Characters/Reallusion/Worker/Worker.usd'
BUSF_SRC = 'omniverse://localhost/NVIDIA/Assets/Characters/Reallusion/ActorCore/Business_F_0002/business-f-0002.usd'


with rep.new_layer():

    rep.settings.set_stage_up_axis("Z")

    # Display Worker
    worker = rep.create.from_usd(WORKER_SRC, semantics=[('class', 'worker')])
    with worker:
        rep.modify.pose(position=(30,0,0),rotation=(0,0,180))

    # Display Business Woman
    bus_f = rep.create.from_usd(BUSF_SRC, semantics=[('class', 'business_woman')])
    with bus_f:
        rep.modify.pose(position=(60,90,0),rotation=(90,0,90))

    light = rep.create.light(
        light_type="Sphere",temperature=10000,intensity=10000,
        position=(200,180,200),scale=100,count=1)
    camera = rep.create.camera(position=(100,500,100),rotation=(0,0,90))    
    render_product = rep.create.render_product(camera,(256,256))


    writer = rep.WriterRegistry.get("BasicWriter")
    writer.initialize(output_dir=OUTPUT_DIR,skeleton_data=True)
    writer.attach([render_product])

    rep.orchestrator.run(num_frames=1,start_timeline=False)
    # rep.orchestrator.step(rt_subframes=1)

I do have other things that bother me (but with lower priority):

  • headless mode stopped working for me on version code-2022.3.3 (it doesn’t generate data)
  • it runs indefinitely despite using .run(num_frames=1) or .step()
  • I did not find a way to randomize the usd url from rep.create.from_usd()
  • skeleton generated data is in a format incompatible with bodypose. I made myself a
    mapping (ov_bp_mapping.txt (1.4 KB))
    but an official way to interface with TAO should be the optimal way.

Any comment on my own code, my purpose or even as i posted my request is appreciated!

Reyemai

2

Just to let you know I still have the issue.
Hope in anyone who can comment on this.

Thank you!
Reyemai