Docker data generation: Replicator 1.6.3, Pathtraced WITHOUT motion blur? How?

Omniverse Synthetic Data Generation (SDG)
1

Please find an archive of some scenes and Docker/code/docker compose stuff here:

Using docker compose with NVIDIA Container Toolkit on Linux

I have been able to generate motion blurred images using Path Tracing, which would be excellent, but ideally I want a really short shutter like 0.002 degrees out of 360 degrees.

There is now way I can see how to do this.

If I render with RTX Raytracing, there is no motion blur, but the antialiasing and light transport is a little crummy.

So if you download the data from the link.

All you need to do is

docker compose up --build

in the top level folder where the docker-compose.yml file resides along side the Dockerfile from a command prompt in Linux with NVIDIA Container Toolkit installed and Linux Driver 515+

Then after some time you can look in the subfolder ./omniverse_data/data/output/ ... and you will see the rgb.0000.png files as attached to this post

This is a subset of a larger project that procedurally generates these scenes and scales the output via a Kubernetes message queue.

But this encapsulates the component well enough to see the issue.

If you want to see it without the pathtracing edit the line in the docker-compose.yml file from “True” to “False” on the line that says RAYTRACED

I would be totally interested in anyone can get this to work on WSL2 also.

But I have not tried that.

See example image attached

rgb_0008
rgb_00081920Ă—1080 57.3 KB

3

Use CaptureOptions.ptmb_subframes_per_frame and set it to 0

5

We are going to get back to you, but you seem to be running an old version of Replicator. Is this the most recent version you can run through a docker, or is your actual docker out of date ?

6

where would I put this option in the render.py script?

7

there is a more recent synthetic-data-generator docker image which is from the 4th of May 2024, but that actually crashes hard, I prefer to get consistent images with the wrong motion blur rather than a crash on a beta product.

8

Here is the versions I have available

nvcr.io/nvidia/omniverse-replicator:1.6.3 motion blur issue

01/04/2023 7:09 AM

and

nvcr.io/nvidia/ov-synthetic-data-generation:0.0.16-beta Crashes hard

11/01/2023 8:29 AM

9

if you could update the render.py so it is pathtraced without motion blur I would be a very happy man.

10

It is hard to help you because you are doing everything through script commands. Most people just use the GUI.

11

If you know software engineering you know that the GUI is just calling an event within the API.

12

Here are the documents:

https://docs.omniverse.nvidia.com/py/replicator/1.6.4/source/extensions/omni.replicator.core/docs/API.html#omni.replicator.core.settings.set_render_pathtraced

13

Can you try the Isaac container? It seems to be the most reliable for SDG.

14

Hi @samuel.hodge

I realize you’re on replicator 1.6.3. I’m posting the solution for use with the latest IsaacSim release (2023.1.1) first, while I check with the engineers on what’s possible on the older version.

The setting is:
carb.settings.get_settings().set("/omni/replicator/captureMotionBlur", False)

This script produces these images for comparison:

import omni.replicator.core as rep
import carb.settings

with rep.new_layer():
    # Set the renderer to Path Traced
    rep.settings.set_render_pathtraced(samples_per_pixel=64)
    
    # Motion Blur Settings - set true false below
    carb.settings.get_settings().set("/omni/replicator/captureMotionBlur", True)
    
    render_product = rep.create.render_product('/World/Camera', (1920, 1080))
    
    # Initialize and attach writer
    writer = rep.WriterRegistry.get("BasicWriter")
    writer.initialize( output_dir="_MotionBlur_01", rgb=True)
    writer.attach([render_product])
    
    rep.trigger.on_frame(num_frames=20)

image
image1917Ă—581 43.5 KB

1 Like
15

Thanks for the fix.

I think the issues of taking on the other hurdles with Isaac Sim in a k8s / Docker deployment might shift the one line fix to days of work.

But it might be better than nothing

I see that the typical use case of Docker containers running IsaacSim are to use an Xorg session on the host system’s Vulkan drivers.

This is not where I want to go

So if we can get a nicer headless solution I would be much more grateful

The last updated Docker container that is truely headless is a lot older than the IsaacSim version, perhaps updating those images would be a better solution for me

16

If you can give me a Kubernetes deployment guide for an Isaac Sim container, yes this would be a viable solution all of the deployment guides are only well suited to a host system with Xorg and Vulkan installed. I think doing this on our host node running the k8s pods is going to be a lot of extra effort on my behalf. A solution which is more 1:1 equivalent with my current use case without bringing in more indirection would be acceptable for me.

17

Looking into

https://docs.omniverse.nvidia.com/isaacsim/latest/manual_standalone_python.html

and

https://docs.omniverse.nvidia.com/isaacsim/latest/installation/install_container.html

It looks like I might be able to fix this today.

18

Nope cannot get out of the gates, how can I run a script from within /isaac-sim/ directory of the following Dockerfile

FROM nvcr.io/nvidia/isaac-sim:2023.1.1
USER root
ENV BASE_PATH=/
ENV LC_ALL=C.UTF-8 \
    LANG=C.UTF-8
WORKDIR /
RUN mkdir /omniverse_data
WORKDIR /
COPY render.py render.py
COPY entrypoint.sh entrypoint.sh
ENTRYPOINT [ "/bin/bash",  "entrypoint.sh" ]

where entrypoint.sh is as follows

#!/usr/bin/env bash
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
/isaac-sim/python.sh ${SCRIPT_DIR}render.py /World/UAV/Payload/Camera_0 /omniverse_data/data/input/scenes/scene_0000.usd /omniverse_data/data/output/footage/scene_0000
echo "Mission Complete"

but was greeted with

motionblur-1  | Traceback (most recent call last):
motionblur-1  |   File "/render.py", line 6, in <module>
motionblur-1  |     import omni.replicator.core as rep
motionblur-1  | ModuleNotFoundError: No module named 'omni.replicator'

and docker-compose.yml

is

x-motionblur-config:
  &motionblur-config
  runtime: nvidia
  build:
    context: ./
    dockerfile: ./Dockerfile
    args:
      version: "${VERSION}"
  network_mode: "host"
  mem_limit: 320GB
  environment:
    CUDA_VISIBLE_DEVICES: 0
    NVIDIA_VISIBLE_DEVICES: all
    RAYTRACED: "True"
  shm_size: '300gb'

services:
  motionblur:
    << : *motionblur-config
    volumes:
      - ./omniverse_data/data:/omniverse_data/data
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [ gpu ]

So using Isaac Sim for Replicator seems to be a bad idea if you are using Docker.

19

Adding the following lines to the render.py script helps

from omni.isaac.kit import SimulationApp

simulation_app = SimulationApp({"headless": True})

but there is no output rendered, some of the code is run because newly dated folders exist.

20

Output is as follows from the bottom of a lot of noisy chatter

motionblur-1  | Starting kit application with the following args:  ['/isaac-sim/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py', '/isaac-sim/apps/omni.isaac.sim.python.kit', '--/app/tokens/exe-path=/isaac-sim/kit', '--/persistent/app/viewport/displayOptions=3094', '--/rtx/materialDb/syncLoads=True', '--/rtx/hydra/materialSyncLoads=True', '--/omni.kit.plugin/syncUsdLoads=True', '--/app/renderer/resolution/width=1280', '--/app/renderer/resolution/height=720', '--/app/window/width=1440', '--/app/window/height=900', '--/renderer/multiGpu/enabled=True', '--/app/fastShutdown=True', '--ext-folder', '/isaac-sim/exts', '--ext-folder', '/isaac-sim/apps', '--/physics/cudaDevice=0', '--portable', '--no-window', '--allow-root']
motionblur-1  | Passing the following args to the base kit application:  ['/World/UAV/Payload/Camera_0', '/omniverse_data/data/input/scenes/scene_0000.usd', '/omniverse_data/data/output/footage/scene_0000']
motionblur-1  | Warp 1.0.0-beta.2 initialized:
motionblur-1  |    CUDA Toolkit: 11.5, Driver: 12.2
motionblur-1  |    Devices:
motionblur-1  |      "cpu"    | x86_64
motionblur-1  |      "cuda:0" | NVIDIA GeForce RTX 4090 (sm_89)
motionblur-1  |    Kernel cache: /root/.cache/warp/1.0.0-beta.2
motionblur-1  | [78.917s] RTX ready
motionblur-1  | 2024-05-16 07:02:01 [93,952ms] [Warning] [omni.graph.core.plugin] Node type name '' is missing the unique namespace
motionblur-1  | 2024-05-16 07:02:01 [93,954ms] [Warning] [carb] [Plugin: omni.spectree.delegate.plugin] Module /isaac-sim/kit/exts/omni.usd_resolver/bin/libomni.spectree.delegate.plugin.so remained loaded after unload request
motionblur-1  | 2024-05-16 07:02:01 [93,956ms] [Warning] [omni.stageupdate.plugin] Deprecated: direct use of IStageUpdate callbacks is deprecated. Use IStageUpdate::getStageUpdate instead.
motionblur-1  | 2024-05-16 07:02:01 [93,957ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
motionblur-1  | 2024-05-16 07:02:01 [93,957ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
motionblur-1  | 2024-05-16 07:02:01 [93,957ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
motionblur-1  | 2024-05-16 07:02:01 [93,957ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
motionblur-1  | 2024-05-16 07:02:01 [93,957ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
motionblur-1  | 2024-05-16 07:02:01 [93,957ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
motionblur-1  | 2024-05-16 07:02:01 [93,957ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
motionblur-1  | 2024-05-16 07:02:01 [93,957ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
motionblur-1  | 2024-05-16 07:02:01 [93,957ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
motionblur-1  | 2024-05-16 07:02:01 [93,957ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
motionblur-1  | 2024-05-16 07:02:01 [93,957ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Annotators' for removal
motionblur-1  | 2024-05-16 07:02:01 [93,957ms] [Warning] [omni.graph.core.plugin] Could not find category 'Replicator:Core' for removal
motionblur-1  | 2024-05-16 07:02:01 [93,960ms] [Error] [omni.graph.core.plugin] N4omni5graph4core4NodeE::getPointerFromHandle - handle 31 not found
motionblur-1  | 2024-05-16 07:02:01 [93,961ms] [Warning] [carb.audio.context] 1 contexts were leaked
motionblur-1  | 2024-05-16 07:02:01 [94,001ms] [Warning] [carb] Recursive unloadAllPlugins() detected!
motionblur-1  | 2024-05-16 07:02:01 [94,008ms] [Warning] [omni.core.ITypeFactory] Module /isaac-sim/kit/exts/omni.activity.core/bin/libomni.activity.core.plugin.so remained loaded after unload request.
motionblur-1  | Mission Complete
motionblur-1 exited with code 0
21

Here’s my base dockerfile, maybe it will help.

FROM nvcr.io/nvidia/isaac-sim:2023.1.1

# Install tools
RUN apt update && apt install git vim -y

# Remove ROS/2 Bridge
RUN sed -i 's/ros_bridge_extension = "omni.isaac.ros2_bridge"/ros_bridge_extension = ""/g' /isaac-sim/apps/omni.isaac.sim.base.kit

# Toggle Grid Off
RUN sed -i '17i import omni.kit.viewport' /isaac-sim/extscache/omni.replicator.replicator_yaml-2.0.4+lx64/omni/replicator/replicator_yaml/scripts/replicator_yaml_extension.py
RUN sed -i '100i \ \ \ \ \ \ \ \ omni.kit.viewport.actions.actions.toggle_global_visibility(visible=False)' /isaac-sim/extscache/omni.replicator.replicator_yaml-2.0.4+lx64/omni/replicator/replicator_yaml/scripts/replicator_yaml_extension.py

Make sure to add your scripts to it and then run this command when shelled into it:

    /isaac-sim/runheadless.native.sh  --no-window \
                                      --/app/viewport/grid/enabled=false \
                                      --/app/viewport/defaults/guide/grid/visible=false \
                                      --/omni/replicator/replicatorYaml/yamlPath=$FILE

Please mind the above command is for YAML input. You need to use the appropriate one for Python input.

You don’t have to add your python scripts on docker build. You can mount a volume folder on your drive to the container during docker run.

22

This is the most useful help so far

Hopefully I can get some data generated over the weekend with path tracing and decent anti aliasing without too much motion blur and the annotations to do the job

It’s strange that the replicator and synthetic data docker images are so sparsely used

Oh well this patched workaround should work

As for mounting scripts for deployment building a container and putting it in a versioned container registry based on version controlled git artefacts gives me the CI/CD approach that I want to pick things apart at a later date forensically