Corrupted render when running SDG with multiple render products and multiple GPUs

Omniverse Isaac Sim
,
1

Isaac Sim Version

4.5.0
4.2.0
4.1.0
4.0.0
4.5.0
2023.1.1
2023.1.0-hotfix.1
Other (please specify):

Operating System

Ubuntu 22.04
Ubuntu 20.04
Windows 11
Windows 10
Other (please specify):

GPU Information

  • Model: 4x A6000
  • Driver Version: 560.35.03

Topic Description

Detailed Description

I’m trying to render two images for stereo cameras on a 4-GPU machine, but the output is corrupted. The issue resolves if I set /renderer/multiGpu/maxGpuCount=2.

It is shocking to me that such a simple toy example would break when it is claimed that there is multi-GPU support.

An additional issue is that there are only 2 data points rendered instead of the expected 3. This issue does not resolve even if multi-GPU rendering is disabled. UPDATE: I found that the rep.orchestrator.wait_until_complete() call must be placed after destroying render products for the last data to be written to the disk. This is either a bug or the tutorial code is incorrect. Getting Started Scripts — Isaac Sim Documentation

One more thing, the documentation here is incorrect.

/renderer/multiGpu/Enabled=true enables multiple GPUs for rendering

To disable multi-GPU, the command should be /renderer/multiGpu/enabled=false.

Steps to Reproduce

from isaacsim import SimulationApp

simulation_app = SimulationApp({"headless": True})
import omni.replicator.core as rep
from isaacsim.core.api import World
from omni.kit.viewport.utility import get_active_viewport

get_active_viewport().updates_enabled = False
rep.orchestrator.set_capture_on_play(False)


# rep.settings.set_render_pathtraced(8)
rep.settings.set_render_rtx_realtime("Off")

world = World(stage_units_in_meters=0.001, physics_prim_path="/PhysicsScene")
world.get_physics_context().enable_ccd(True)

writer = rep.writers.get("BasicWriter")
out_dir = "test_output"
writer.initialize(output_dir=out_dir, rgb=True)

camera = rep.create.stereo_camera(stereo_baseline=50, position=(0, 0, 800), look_at=(0, 0, 0), focal_length=60)
rps = rep.create.render_product(camera, (2880, 1860))

dome_light = rep.create.light(
    light_type="dome",
    intensity=1,
    exposure=10,
)

cube = rep.create.cube(scale=0.1)

world.reset()
writer.attach(rps)


num_data = 3
for i in range(num_data):
    rep.orchestrator.step(rt_subframes=20)

writer.detach()
for rp in rps:
    rp.destroy()
rep.orchestrator.wait_until_complete()

simulation_app.close()

Error Messages

None

Screenshots or Videos

Corrupted right image when using rtx realtime renderer. A portion of the image is overwritten by the left image content.

rtx
rtx2880×1860 76.9 KB

Corrupted right image when using path tracing renderer. A subset of pixels are not rendered at all and you can see the stripe like artifacts on the cube.
pathtrace
pathtrace2880×1860 254 KB

Additional Context

Here is the log file produced by using rtx realtime renderer.
kit_20250507_145246.log (803.7 KB)

What I’ve Tried

I tried to change the driver version to 535.247.01. The same issue persists.

Related Issues

I previously posted a similar issue. At that time I thought it is related to FXAA denoising. However after more testing I found that other rendering methods fail as well.

2

You are actually using the wrong recommended driver. Please try this one Driver Details | NVIDIA

Also, it has been documented that multi gpu ON, can cause that pattern. It is rare, but it can happen. If you do not have it with 1 gpu or 2 gpus, then I would stick with that for now. The error is very subtle and it depends on a lot of factors, some of which are beyond our control. You are using the realtime renderer, which is not ideal for final SDG rendering. Try switching over to the much superior “interactive / path tracing” renderer. Slower but far far more accurate. Plus it uses multi gpu much better.

3

Thank you for your reply!

The first image is produced by the realtime renderer and the second image is produced by the path tracing renderer. Therefore, neither works for me when using 4 GPUs.

Your link points to a Windows 11 driver, but I’m running it on Ubuntu.
I’ve tried to change the driver to latest 535 driver, 535.247.01, according to Isaac Sim Requirements — Isaac Sim Documentation

  • NVIDIA driver version 535.216.01 or later is recommended when upgrading to Ubuntu 22.04.5 kernel 6.8.0-48-generic or later.

But still no luck.

4

Is this rendered through the Replicator? What happens when you don’t do stereo cameras. Just normal? Can you provide the scene.

5

The scene is fully reproducible with the posted script. It is as simple as a cube, a dome light, and a pair of stereo cameras. It is rendered though Replicator.

The artifact issue does not happen if there is only one camera. The artifact always happens on the second camera.

6

Ah interesting. So the error must lie with the use of a second camera.

7

Seems like other users are experiencing similar issues. In their case, they seem to be using a single camera and got similar artifacts. They were also having the problem with A6000 graphics cards, I wonder if it is a coincidence.

8

This issue appears to be related. I followed the suggestion from that post and implemented a custom writer, but unfortunately it didn’t resolve the problem in my case.

9

Ok and just to recap, it works fine if you disable multi GPU correct ? We are about to release Isaac Sim 5.0, hopefully that fixes the issue for you. I will follow up. @pcallender for viz on this.