Path Tracing Matte Object Texture Randomizer Error

[Turowicz]

While trying to randomize the static background with Path Tracing turned on I am getting blue backgrounds every 2rd or 3rd frame. RTX Realtime shows it in the first couple rt_subframes only, but I would rather use path tracing.

...
# When the trigger executes, run the following logic
with_trigger:
  with.trigger:

    # Set background
    background:
      settings.carb_settings:
        setting: /rtx/post/backgroundZeroAlpha/backplateTexture
        value: 
          distribution.sequence:
            items: 
              - "omniverse://localhost/Library/CRH/Backgrounds/cam6/cam6_10.48.190.166_554_20240205_132913-00.01.56.594.jpeg"
              - "omniverse://localhost/Library/CRH/Backgrounds/cam6/cam6_10.48.190.166_554_20240206_074317-00.04.00.405.jpeg"
              - "omniverse://localhost/Library/CRH/Backgrounds/cam6/CRH NVR11_ch29_20240108064113_20240108072806-00.01.44.687.jpeg"
              - "omniverse://localhost/Library/CRH/Backgrounds/cam6/CRH NVR11_ch29_20240108064113_20240108072806-00.26.35.407.jpeg"
              - "omniverse://localhost/Library/CRH/Backgrounds/cam6/CRH NVR11_ch29_20240108064113_20240108072806-00.44.06.425.jpeg"
              - "omniverse://localhost/Library/CRH/Backgrounds/cam6/CRH NVR11_ch29_20240117124752_20240117134752-00.00.00.000.jpeg"
              - "omniverse://localhost/Library/CRH/Backgrounds/cam6/CRH NVR11_ch29_20240117124752_20240117134752-00.43.49.228.jpeg"
              - "omniverse://localhost/Library/CRH/Backgrounds/cam6/CRH NVR11_ch29_20240201142421_20240201151119-00.08.55.831.jpeg"
              - "omniverse://localhost/Library/CRH/Backgrounds/cam6/CRH NVR11_ch29_20240201142421_20240201151119-00.38.38.710.jpeg"
              - "omniverse://localhost/Library/CRH/Backgrounds/cam6/CRH NVR11_ch29_20240201142421_20240201151119-00.42.59.721.jpeg"

cc @dennis.lynch

[Turowicz]

Setting the interval on trigger to 2 and picking up every odd result rgb frame “solves” this issue.

[Turowicz]

cc @pcallender

[pcallender]

Thanks for posting this. I’ve reproduced the issue with the script below. I’ll make this into a ticket to track and see if we can get some clarity on what’s causing it.

import omni.replicator.core as rep
import carb
import carb.settings

test_textures_list = ['omniverse://localhost/NVIDIA/Materials/vMaterials_2/Ground/textures/aggregate_exposed_diff.jpg',
'omniverse://localhost/NVIDIA/Materials/vMaterials_2/Ground/textures/leaves_diff.jpg',
'omniverse://localhost/NVIDIA/Materials/vMaterials_2/Ground/textures/paving_stones_diff.jpg']

# Scene settings
rep.settings.set_stage_up_axis("Y")
rep.settings.set_stage_meters_per_unit(0.01)

# Remove the default light
default_light = rep.get.prims(path_pattern="/Environment/defaultLight")
with default_light:
    rep.modify.visibility(False)

camera_positions = [(1347,825,1440), (0, 825, 1440),(1440,825,0)]

# Create the lights
distance_light = rep.create.light(rotation=(400,-23,-94), intensity=10000, temperature=6500, light_type="distant")
cylinder_light = rep.create.light(position=(0,0,0),rotation=(0,-180,-180),light_type="disk")

# Create the environment
cone = rep.create.cone(position=(0,100,0), scale=2)
floor = rep.create.cube(position=(0,0,0), scale=(10,0.1,10))
wall1 = rep.create.cube(position=(-450,250,0), scale=(1,5,10))
wall2 = rep.create.cube(position=(0,250,-450), scale=(10,5,1))

#Create the replicator camera
camera = rep.create.camera(position=(1347,825,1440), look_at=(0,100,0), focus_distance=200,f_stop=8)

# Set the renderer to Path Traced
rep.settings.set_render_pathtraced(samples_per_pixel=64)

# Create the render product
render_product  = rep.create.render_product(camera, (1920, 1080))

# Initialize and attach writer
writer = rep.WriterRegistry.get("BasicWriter")
writer.initialize(output_dir="_composite_test", rgb=True)
writer.attach([render_product])

#Make the Compositing Post Process Enabled
rep.settings.carb_settings("/rtx/post/backgroundZeroAlpha/enabled", True)

# Render 10 frames, with 50 subframes
with rep.trigger.on_frame(num_frames=10, rt_subframes=50):
    rep.settings.carb_settings("/rtx/post/backgroundZeroAlpha/backplateTexture", rep.distribution.sequence(test_textures_list))
    with camera:
        rep.modify.pose(position=rep.distribution.sequence(camera_positions), look_at=(0,100,0))

[Turowicz]

Thanks for having a look. This is a minor issue for me as I have a workaround, but it would be nice to solve just to speed up the generation 2x (so I don’t have to delete every odd frame)

[Turowicz]

@pcallender what is the reason for hiding the default distant light and creating a new one? I always use the default one and modify it’s pose etc. Perhaps I’m doing it wrong?

[pcallender]

It was just part of the script I use for testing a lot, and that script has very specific lighting. Whether you use default or not is entirely up to you.

[Turowicz]

This is a possible reason for this crash: Crash

I am trying to trial-and-error what is causing the crash and so far its either light randomization or changing backplateTexture with a distribution. Will get back with results soon.

[Turowicz]

This seems related as I get blue background on every even frame like other issues here: Ghosts and other weird things

[user3362]

any updates on this issue?

[Turowicz]

@user3362 you can do a workaround by generating spearate scenario YAMLs per background with a bash script and then running those generations separately. That said, this issue is a pain in the ***.

[user3362]

Thanks, I’ve been having the same issue I’ll let you know if I find a better workaround. Very weird issue. You could automatically remove images over some threshold of those blue pixels and then regenerate that many but you’d probably have to do that recursively.

[Turowicz]

@pcallender any changes in 4.0.0 for this?

[Turowicz]

We no longer require to use textures as backgrounds so I am closing the issue. Most likely this issue has been mitigated in the 4.0.0 release by reintroduction of on_frame.rt_subframes parameter for path tracing.