Memory leaks when working with matplotlib or wandb

Omniverse Isaac Sim
1

Isaac Sim Version

4.5.0

Operating System

Ubuntu 20.04

GPU Information

  • Model: T4
  • Driver Version: 535.183.01

Topic Description

In the process of switching from Isaacsim 4.0.0 to to 4.5.0 I noticed that we suddenly have massive memory leaks with the new version.

I created a small repro at the end of this post.
And here the resulting memory footprint when I run the script with version 4.0 and 4.5 respectively:

image
image1490×1402 187 KB

I also observe new memory leaks from passing numpy arrays to wandb.Video though I can avoid that problem by creating a video file with ‘imageio’ and upload that to wandb.
So far I have not found a similar workaround for uploading plots, since simply creating them seems to already create some memory leakage.

Anyways, I believe that using these packages (wandb and matplotlib) creates memory leaks should be a major concern, regardless of possible work-arounds.

try:
    import isaacsim
    from isaacsim.simulation_app import SimulationApp  # isort: skip
except:
    from omni.isaac.kit import SimulationApp

simulation_app = SimulationApp({"headless": True})


def random_plot():
    import matplotlib.pyplot as plt
    import numpy as np

    fig = plt.figure()
    x = np.random.rand(10)
    y = np.random.rand(10)
    plt.scatter(x, y)
    plt.close()
    return fig


config = {
    "experiment_name": "MemroyLeakTest",
}

import wandb

wandb.init(
    config=config,
    name="MemoryLeakTest",
)

try:
    from isaacsim.core.api.world import World
except:
    from omni.isaac.core.world import World

world = World(stage_units_in_meters=1.0)

world.reset()

for i in range(1000):
    for j in range(100):
        world.step()
    print(i)
    # fig = random_plot()
    wandb.log(
        {
            "example_metric": i,
            "random_plot": wandb.Image(random_plot()),
        }
    )

wandb.finish()
world.stop()
simulation_app.close()
1 Like
3

Thank you for bringing this issue to our attention and providing a detailed reproduction script along with the memory usage graph.

An internal ticket has been created to track this issue. We will update this thread as soon as we have more information or a resolution. In the meantime, you might try using gc.collect().

4

Thank you for the update.

And you are right gc.collect() is enough to free the memory periodically. (Although it also costs quite a bit of time.)