Isaac Sim | Running script editor kills the GPU memory

[omers]

We are creating sensors and OmniGraph nodes using a python script.
Running the script kills the GPU’s memory.
Before running the script:

After:

Unfortunately, I can’t share the script because it is proprietary, but it basically creates 4 cameras, 2 imus, and the OmniGraph nodes for publishing the cameras to ROS2.

[Sheikh_Dawood]

Hi. Does this issue only occur in the script editor or also in command line?

It would be helpful if you have a non-proprietary sample python script with the issue so that we could reproduce it on our end.

Does the issue happen if you comment out the OmniGraph nodes?

A copy of the Isaac Sim logs would be helpful too.

[omers]

@Sheikh_Dawood Yes it occurs even without the OmniGraph nodes, here is an extracted example:

from omni.isaac.core.physics_context import PhysicsContext
from omni.isaac.core.prims import XFormPrim
from omni.isaac.sensor import IMUSensor, Camera

focal_length = 1.88
focal_distance = 300
PhysicsContext()

base_path = "/nakai_container/nakai_robot"
trunks_with_cameras = ["base", "trunk4"]

for trunk in trunks_with_cameras:
    right_mount_path = f"{base_path}/{trunk}_camera_link/{trunk}_camera_right_mount"
    left_mount_path = f"{base_path}/{trunk}_camera_link/{trunk}_camera_left_mount"
    
    XFormPrim(right_mount_path, name=f"{trunk}_camera_right_mount", translation=[0, -0.095, 0])
    right_camera = Camera(f"{right_mount_path}/{trunk}_camera_right", name="{link}_camera_right")
    right_camera.set_focal_length(focal_length) 
    right_camera.set_focus_distance(focal_distance)
    
    XFormPrim(left_mount_path, name=f"{trunk}_camera_left_mount")
    left_camera = Camera(f"{left_mount_path}/{trunk}_camera_left", name="{link}_camera_left")
    left_camera.set_focal_length(focal_length)
    left_camera.set_focus_distance(focal_distance)

    IMUSensor(f"{base_path}/{trunk}_camera_imu_link/imu")

[Sheikh_Dawood]

This could be related to ISAAC Sim just crashes abruptly - #9 by omers

It could have crashed because of insufficient VRAM. Try running the script with only 1 camera.

[omers]

@Sheikh_Dawood In this issue, the memory goes down just by running the above code in the script editor, without running the simulation. I also tried with a simulating a single stereo camera (left and right), seems like just creating this camera and its xforms takes aroung 1.8 GB of GPU memory.

[Sheikh_Dawood]

Thanks for the update.
How does the GPU memory usage looks as you add more nodes or cameras?

[omers]

The more I add the more it goes down.

[Sheikh_Dawood]

Ok. Can you help us detect if there is a memory leak?

Try start from a fresh boot and start Isaac Sim with the script with one camera. Check the GPU memory usage.
Next add more cameras and check GPU memory again.
Next close and reopen Isaac Sim and Run the script with one camera again and see the usage. It should be the same as before.

So far does it crash less with that one camera script?

[omers]

Every time I reopen Issac Sim, or even re-open the model without closing Isaac sim, the memory is freed. With only 1 camera it doesn’t crash, but it does still take a lot of memory.

By the way, if I run the script, save and open the saved file, memory usage is not so bad (300 mb). So seems like there is some kind of memory abuse by the python API.

[Sheikh_Dawood]

Sounds good. Yes, the team is working on optimizing performance but we have no ETA on this.

[omers]

@Sheikh_Dawood Thanks for the update.