Docker data generation: Replicator 1.6.3, Pathtraced WITHOUT motion blur? How?

Looks like this might work

#!/usr/bin/env bash
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
  /isaac-sim/runheadless.native.sh  --allow-root \
                                                               --/app/viewport/grid/enabled=false \
                                                               --/app/viewport/defaults/guide/grid/visible=false \
                                                               --no-window \
                                                                --/omni/replicator/script=${SCRIPT_DIR}render.py $ARG1 $ARG2 $ARG3

And on top of all of your patches, I will add the NATS queue management to consume a list of args on a number of pod workers.

The idea of shelling into a running container is not how I roll.

now getting the errror

motionblur-1  | 2024-05-16 23:22:00 [202,977ms] [Warning] [omni.isaac.kit.simulation_app] Please check to make sure no extra omniverse or pxr modules are imported before the call to SimulationApp(...)
motionblur-1  | 2024-05-16 23:22:00 [202,978ms] [Error] [asyncio] Task exception was never retrieved
motionblur-1  | future: <Task finished name='Task-44' coro=<Extension.run_script_async() done, defined at /isaac-sim/extscache/omni.replicator.core-1.10.20+105.1.lx64.r.cp310/omni/replicator/core/scripts/extension.py:197> exception=KeyError('EXP_PATH')>
motionblur-1  | Traceback (most recent call last):
motionblur-1  |   File "/isaac-sim/extscache/omni.replicator.core-1.10.20+105.1.lx64.r.cp310/omni/replicator/core/scripts/extension.py", line 221, in run_script_async
motionblur-1  |     spec.loader.exec_module(module)
motionblur-1  |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
motionblur-1  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
motionblur-1  |   File "/render.py", line 7, in <module>
motionblur-1  |     simulation_app = SimulationApp({"headless":True})
motionblur-1  |   File "/isaac-sim/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py", line 172, in __init__
motionblur-1  |     experience = f'{os.environ["EXP_PATH"]}/omni.isaac.sim.python.kit'
motionblur-1  |   File "/isaac-sim/kit/python/lib/python3.10/os.py", line 680, in __getitem__
motionblur-1  |     raise KeyError(key) from None
motionblur-1  | KeyError: 'EXP_PATH'

OK I have it working now

But the problem is as follows:

The pixel samples are not completed before the pixels are written to disk.

I have a huge number of samples per pixel but the result looks like there has only been less that 0.5 seconds spent on rendering each frame.

How can I fix this?

See how grainy it is?

And no I do not want to use optixDenoiser I was to sample the noise away until I have nothing but signal.

So if you want to see what I see

On a machine with Docker and NVIDIA Container Toolkit you just need open a shell in the folder with the docker-compose. issue the command

docker compose up —build

After a few minutes it will finish with the statement mission complete

Then you can look in the folder omniverse_data/output/… and look at the rgb.0000.png and other numbers to see the poor quality images

paging @pcallender I am not satisfied with the quality of the render, I still have a problem that needs to be solved.

note in the final solution the lines

from omni.isaac.kit import SimulationApp

simulation_app = SimulationApp({"headless": True})

were not needed because the isaac headless bootstrap script makes sure the app is running.

when you call it with the invocation for the replicator python script to be the entry point.

OK I have this integrated into my current k8s pipeline, but as you can see in the attached image the noise in the path traced render is a little too “salt and pepper” for my liking.

Additionally the first frame doesn’t load the roughness map correctly or something and ends up with chrome looking sails

see image attached.

So if you could look at the code at:

specifically render.py and suggest how to get rid or the first frame artefact and the noise, I would be in your debt.

This is why we have a denoiser. It works great. What reason would you have for turning it off ? If anything you can turn it down slightly. But you need to render way more samples then.

Hi @samuel.hodge Adding to what Richard mentioned …

The denoiser is pretty great. I use it for all my SDG needs. That said though, the settings can be tweaked with the following commands:

# Turn DeNoising on and off
rep.settings.carb_settings("/rtx/pathtracing/optixDenoiser/enabled", True)
# Set the Samples per pixel per Frame
rep.settings.carb_settings("/rtx/pathtracing/spp", 64)
# Max samples to accumulate per pixel
rep.settings.carb_settings("/rtx/pathtracing/totalSpp", 64)
# If Denoising True, then you can specify how much to blend with non-denoised - 1.0 is 100% non Denoised
rep.settings.carb_settings("/rtx/pathtracing/optixDenoiser/blendFactor", 0.0)

Try tinkering with the spp and totalSpp to get the desired quality. You can even blend the denoiser with non-denoised if you need. Let me know how this goes for you.

First frame texture loading is a known issue. Its to do with a lack of a reliable callback on texture load (if I recall correctly) I need to check on the status of that issue. I’ll pop back when I have something to report.

For convenience, I’m adding a script I use for testing pathtraced rendering quality:

import omni.replicator.core as rep

# Remove the default light
default_light = rep.get.prims(path_pattern="/Environment/defaultLight")
with default_light:
    rep.modify.visibility(False)
    
# Scene settings, different for isaacsim vs code
rep.settings.set_stage_up_axis("Y")
rep.settings.set_stage_meters_per_unit(0.01)

camera_positions = [(1347,825,1440), (0, 825, 1440),(1440,825,0)]

# Create the lights
distance_light = rep.create.light(rotation=(400,-23,-94), intensity=10000, temperature=6500, light_type="distant")
cylinder_light = rep.create.light(position=(0,0,0),rotation=(0,-180,-180),light_type="disk")

# Create the environment
cone = rep.create.cone(position=(0,100,0), scale=2)
floor = rep.create.cube(position=(0,0,0), scale=(10,0.1,10))
wall1 = rep.create.cube(position=(-450,250,0), scale=(1,5,10))
wall2 = rep.create.cube(position=(0,250,-450), scale=(10,5,1))

#Create the replicator camera
camera = rep.create.camera(position=(1347,825,1440), look_at=(0,100,0), focus_distance=200,f_stop=8)

# Set the renderer to Path Traced
rep.settings.set_render_pathtraced(samples_per_pixel=64)

# Create the render product
render_product  = rep.create.render_product(camera, (1920, 1080))

# Initialize and attach writer
writer = rep.WriterRegistry.get("BasicWriter")
writer.initialize(output_dir="_render_samples_test", rgb=True)
writer.attach([render_product])

# Turn DeNoising on and off
rep.settings.carb_settings("/rtx/pathtracing/optixDenoiser/enabled", True)
# Set the Samples per pixel per Frame
rep.settings.carb_settings("/rtx/pathtracing/spp", 64)
# Max samples to accumulate per pixel
rep.settings.carb_settings("/rtx/pathtracing/totalSpp", 64)
# If Denoising True, then you can specify how much to blend with non-denoised - 1.0 is 100% non Denoised
rep.settings.carb_settings("/rtx/pathtracing/optixDenoiser/blendFactor", 0.0)

# Render 3 frames, with 50 subframes
with rep.trigger.on_frame(num_frames=3):
    with camera:
        rep.modify.pose(position=rep.distribution.sequence(camera_positions), look_at=(0,100,0))

@pcallender did you download my reproduction data?

As for not using the denoiser is because I would like to sample the details that exists rather than hallucinating smoosh that is not real.

But I am happy to try both, the results I was getting with the denoiser were pretty poor.

In my sample code that I provided:

        if draft:
            rep.settings.set_render_rtx_realtime("RTXAA")
        else:
            rep.settings.set_render_pathtraced(SAMPLES_PER_PIXEL)
            settings.set_int("/rtx/pathtracing/spp", SAMPLES_PER_PIXEL)
            settings.set_float("/rtx/pathtracing/optixDenoiser/blendFactor",1.0)
            settings.set_bool("/omni/replicator/captureMotionBlur",0)
        with rep.trigger.on_frame(
            num_frames=NUM_FRAMES, interval=1, rt_subframes=SUBFRAME_SAMPLES
        ):
            pass
        for cur_frame in range(NUM_FRAMES):
            rep.orchestrator.step()
        rep.orchestrator.stop()
        rep.orchestrator.wait_until_complete()

Where the SAMPLES_PER_PIXEL was set to 2048 and 4096 but the quality didn’t improve AND the first frame looked different to the rest.

The full reproduction steps are possible using Docker and NVIDIA Container Runtime and issuing the command docker compose up --build and you will see that editing the variable that controls the SAMPLES_PER_PIXEL in the docker-compose.yml ie PIXEL_SAMPLES: 2048 has no effect on reducing the noise

Sam

Suddenly I can see again via shelling into the running container

and doing

tail -f /root/.nvidia-omniverse/logs/Kit/Isaac-Sim/2023.1/*.log

I was unable to see the stdout because it was being swallowed into this log and the logger from my render.py is visible again.

Full settings are here:
https://docs.omniverse.nvidia.com/py/replicator/1.10.10/source/extensions/omni.replicator.core/docs/API.html#module-omni.replicator.core.settings

I am used to better pixel reconstruction filters to choose from than

0=Box
1=Triangle
2=Gaussian
3=Uniform

and a kernel size in fractional pixel radius.

What happened to Mitchell, and Catmull-Rom?

Gaussian 2x2 is the best to prevent negative lobes but it is pretty soft

I am seeing the following error

2024-05-21 06:54:37 [422,662ms] [Error] [rtx.optixdenoising.plugin] [Optix] [DENOISER] Unable to load denoiser weights
2024-05-21 06:54:37 [422,662ms] [Error] [rtx.optixdenoising.plugin] optixDenoiserCreate(m_optixCtx, temporal ? OPTIX_DENOISER_MODEL_KIND_TEMPORAL_AOV : OPTIX_DENOISER_MODEL_KIND_AOV, &denoiserOptions, &denoiser) failed. Optix Error: OPTIX_ERROR_INTERNAL_ERROR.
Internal error

It seems what works on one machine with NVIDIA 545.x.y drivers under microk8s will not work on the production cluster with NVIDIA 535.x.y drivers. I will see what I can do to get the cluster upgraded.

errors.zip (555.3 KB)

Also I have not been informed about this line

2024-05-21 04:58:11 [394,808ms] [Warning] [rtx.raytracing.plugin] Samples per pixel clamped to 32! Set /rtx/pathtracing/clampSpp to override clamp value, or 0 to disable clamping

The mixed context of the messages are a little annoying too.

the setting applies to pathtracing but the error comes from the [rtx.raytracing.plugin] module.

this setting is only mentioned by @ahaidu here How to render the absence of a prim/mesh or part of a prim/mesh? - #9 by schneeweiss

Need to check:

https://docs.omniverse.nvidia.com/isaacsim/latest/known_issues.html#general

To see which of the errors are actually errors.

Is this documentation accurate?

https://docs.omniverse.nvidia.com/isaacsim/latest/installation/requirements.html#isaac-sim-driver-requirements

@pcallender you have told me that increasing the samples to 64 increases the quality and that is not true until clampSpp is also set to 64 or above.

I knew that setting the spp value to extreme values like 2048and 4096 because that is in my reproduction file that I delivered on Monday 20th May 2024 Adelaide time in the afternoon.

In your code example it would be no different to 32 for the value of spp or totalSpp

Not until clampSpp also comes into play, which is not in your sample code.

I had packaged up my runtime context, code and input data for NVIDIA staff benefit.

Please in the future take the advantage of seeing how the example actually runs rather than quoting a toy example

The perturbation of normals of a reflective ocean responds to sampling noise and the denoiser quite differently to a diffuse primitive so the comments about how the optix denoiser is fit for purpose will vary on the purpose for which it is being used.

This is the reason I supplied data that represents my use case to the NVIDIA team

I could have also shown an example of geometric aliasing with fur, hair or rope rigging of a vessel at sea, like fishing nets, that optix denoiser would have also been a poor choice.

Some times you just need to sample away antialiasing problems, but when it is capped/clamped hard at 32 samples in horizontal and vertical for path tracing there is no way this is going to reach an acceptable production quality for many visuals.

Thanks for your time all the same I am getting closer to a solution.

I have things working on one computer, my workstation, but it is not the same computational device(s) I want to employ for an extended period of time soon.