Replicator empty data if Fabric Scene Delegate enabled

Omniverse Synthetic Data Generation (SDG)
1

When FSD is enabled, replicator returns an empty array.

Windows - Isaac Sim 2023.1.1
Replicator 1.10.20

Reproducing:

  1. In Isaac Sim enable FSD
import carb.settings as settings
import omni

fabric_enabled = settings.get_settings().get_as_bool("/app/useFabricSceneDelegate")

if not fabric_enabled:
    settings.get_settings().set_bool("/app/useFabricSceneDelegate", True)
    omni.kit.window.file.new()
    print("Fabric Scene Delegate enabled")
  1. Create a camera and render product and try to get the rgb data
import omni.replicator.core as rep
import asyncio

async def run():
    rep.settings.set_render_pathtraced(samples_per_pixel=64)

    camera = rep.create.camera()

    rp = rep.create.render_product(camera, (512, 512))

    annotator = rep.AnnotatorRegistry.get_annotator("rgb")
    annotator.attach(rp)

    await rep.orchestrator.step_async(rt_subframes=1)

    out = annotator.get_data(device="cuda")

    print(out)

task = asyncio.ensure_future(run())

If FSD is enabled then out is an empty array. Please let me know how to fix this.

1 Like
2

There are a few warnings the first time get_data is called:

2024-05-28 17:41:53  [Info] [omni.kit.app.plugin] Executing file: C:/Users/jreeves/AppData/Local/Temp/2/xxs4.0/script_1716918112.py
2024-05-28 17:41:53  [Info] [omni.kit.app.plugin] Found a script to execute: C:/Users/jreeves/AppData/Local/Temp/2/xxs4.0/script_1716918112.py
2024-05-28 17:41:53  [Warning] [omni.fabric.plugin] Warning: outputs:prims not found
2024-05-28 17:41:53  [Warning] [omni.fabric.plugin] 
2024-05-28 17:41:53  [Warning] [omni.fabric.plugin] No source has valid data array=0000022295FED378 usdValid=0 cpuValid=0
2024-05-28 17:41:53  [Info] [omni.graph.core.plugin] Resolving /Replicator/SDGPipeline/OgnSampleUniform.outputs:samples to float[3][]
2024-05-28 17:41:53  [Info] [omni.graph.core.plugin] Resolving /Replicator/SDGPipeline/OgnWritePrimAttribute.inputs:values to double[3][]
2024-05-28 17:41:53  [Warning] [omni.fabric.plugin] No source has valid data array=00000230A61ECBF0 usdValid=0 cpuValid=0
2024-05-28 17:41:53  [Info] [omni.usd] [MBVH]: enableHydraengineMasking: OFF, for indexForEngineInstance: 0
2024-05-28 17:41:53  [Info] [omni.usd] [MBVH]: raytracingMotion/enable: OFF, for indexForEngineInstance: 0
2024-05-28 17:41:53  [Info] [omni.replicator.core.scripts.annotators] Attaching rgb to render product(s) ['/Render/RenderProduct_Replicator']
2024-05-28 17:41:53  [Info] [carb] Plugin carb.scripting-python.plugin is already a dependency of omni.graph.core.plugin; not changing unload order
2024-05-28 17:41:53  [Info] [omni.replicator.core.scripts.orchestrator] Replicator Step
2024-05-28 17:41:53  [Info] [rtx.multigpumanager.plugin] View 0 (/Render/RenderProduct_omni_kit_widget_viewport_ViewportTexture_0) assigned to device 0
2024-05-28 17:41:53  [Info] [rtx.multigpumanager.plugin] View 0 (/Render/RenderProduct_omni_kit_widget_viewport_ViewportTexture_0) switching to 1x1 layout
2024-05-28 17:41:53  [Info] [rtx.multigpumanager.plugin] View 1 (/Render/RenderProduct_Replicator) assigned to device 0
2024-05-28 17:41:53  [Info] [rtx.multigpumanager.plugin] View 1 (/Render/RenderProduct_Replicator) switching to 1x1 layout
2024-05-28 17:41:53  [Warning] [rtx.raytracing.plugin] Samples per pixel clamped to 32! Set /rtx/pathtracing/clampSpp to override clamp value, or 0 to disable clamping.
2024-05-28 17:41:53  [Info] [omni.kit.menu.utils.scripts.utils] omni.kit.menu.utils.rebuild_menus
2024-05-28 17:41:54  [Warning] [omni.syntheticdata.plugin] SdRenderVarPtr missing valid input renderVar LdrColorSDhost
2024-05-28 17:41:54  [Info] [omni.kit.app._impl] [py stdout]: []
2024-05-28 17:41:54  [Info] [omni.kit.app._impl]

This one in particular may be relevant: [Warning] [omni.syntheticdata.plugin] SdRenderVarPtr missing valid input renderVar LdrColorSDhost since i’ve been able to trace the empty array all the way back to omni.replicatore.core.scripts.utils.annotator_utils in AnnotatorCache.get_data where the dataPtr is 0.

3

FSD also seems to affect rendering AOVs from movie capture. If FSD is on, movie capture will not output anything while rendering with the Render Product.

4

I’ve also tested on Ubuntu and get the same results. Any help would be appreciated.

5

Bumping. I haven’t had any luck getting this to work on my end.

7

Hi @josiah7 Unfortunately FSD is not supported with Replicator yet. I ran this by the devs and they were a little surprised to see someone using it. :)

For a little bit of extra info, its a feature that’s needed in omni.syntheticdata. There’s no stage with history in FSD and we have no way of retrieving data from a specific point in time.

8

Hi @pcallender thanks for checking. Is there a way around using replicator to capture synthetic data perhaps?

Currently looking to use this with Cesium but it looks like FSD could be really useful for large scale scenes and I can can think of a few other uses for it.