Replicator Composer Memory Leak

[_jon]

Hi all :)
According to the release notes of Isaac Sim 2023.1.0, “Replicator Composer has a memory leak when generating data”. Is this issue was also present in older versions of Isaac Sim (specifically 2022.2.0)?

Because this description would pretty much match what I’m seeing at the moment. In a standalone example, I’m doing the following:

import omni.isaac.core.utils.prims as prim_utils
import omni.replicator.core as rep

rgb_camera = prim_utils.create_prim(
        prim_path="/World/envs/env_0/Camera_RGB",
        prim_type="Camera",
        translation=(-0.01, 0.7, 1.68233),
        orientation=(0, 0, 0, -1),
        attributes={"focalLength": 40.0,}
    )

...

count = 0
while simulation_app.is_running():
    ...
    if count == 10:
        rgb_product = rep.create.render_product("/World/envs/env_0/Camera_RGB", resolution=(2048, 2048))
        rgb = rep.AnnotatorRegistry.get_annotator("rgb")
        rgb.attach([rgb_product])
        rep.orchestrator.step(pause_timeline=False)
        print(rgb.get_data())
    ...
    if count == 100:
        sim.reset()
        count = -1

    count += 1

In theory, this should only output one image at the (almost) start of the simulation. However, when i run this, as soon as the first image is taken, my FPS drops from 80 to 40 - and even further on every subsequent reset (to 20, to 9, to 7, …). Without the camera code, I’m getting stable 80 FPS.
It really looks like a memory leak too - my graphics card is maxed out and the used memory increases every time a new picture is taken. (The step in the GPU memory usage is exactly when the picture is taken).

This pretty much renders my simulation unusable. I’m running Isaac sim 202.2.0 on Ubuntu 22.04 on an RTX 3090 and an Intel Core i9 12th gen.

Is this the same memory leak? Or am I doing something else wrong? I can also provide more code or other details if this would help.

Thanks in advance :)

[jlafleche]

Hello @_jon, thank you for reaching out! We don’t believe you should be seeing memory leaks or performance deterioration as capture progresses, so this would certainly be unexpected.

On the FPS drop on first frame: In order to guarantee that no simulation frame is skipped, Kit is placed in a sync rendering mode, which will demonstrate as a FPS drop. In addition to this, a drop in FPS is expected whenever a new render product is created, proportional to its resolution.

On the subsequent FPS drops / VRAM consumption that you reported, I suspect this is the result of creating a new render product every 10 count. Though I don’t have the full context of your script, I would expect something like this would avoid creating unnecessary render products:

rgb_product = rep.create.render_product("/World/envs/env_0/Camera_RGB", resolution=(2048, 2048))
rgb = rep.AnnotatorRegistry.get_annotator("rgb")
rgb.attach([rgb_product])

count = 0
while simulation_app.is_running():
    ...
    if count == 10:
        rep.orchestrator.step(pause_timeline=False)
        print(rgb.get_data())
    ...
    if count == 100:
        sim.reset()
        count = -1

    count += 1

If this solution is not valid for your use case or you find that available VRAM and/or performance are still decreasing over time, please do let us know. Thank you!

[_jon]

Good morning @jlafleche :)

It really looks like the continued increase in memory usage was because of me creating a new rendering product, thank you! At least that is solved now.

For my specific use case, I really only need to render one camera frame at the start of the simulation. Having a rendering product ?running? continuously seems a bit like a waste in terms of resources (especially since I sacrifice about 50% of my simulation performance for this).

Is there a way to destroy the rendering product after I’ve saved the camera output? Or can I disable this sync rendering mode (since I don’t really care about the specific frame I’m capturing)? Or is there another way to do this?

Thank you again for your answer.

[jlafleche]

Hello @_jon,

Good questions! In the latest versions of Replicator you can now indeed easily destroy render products by calling <render_product>.destroy(). Async rendering mode resumes when you call rep.orchestrator.stop() but be aware that stopping replicator also means that on restart, the samplers will be reset. You can manually control sync/async behaviour by setting the appropriate flag as so:

# enable async
carb.settings.get_settings().set("/app/asyncRendering", True)

# disable async
carb.settings.get_settings().set("/app/asyncRendering", False)

[_jon]

Alright, thank you very much for the response - this looks exactly like what I’ve been looking for! I’ll try this as soon as I get the chance and report back on here.