Out of memory when loading scene

Omniverse Isaac Sim
1

Hi all,

I am getting an “out of memory error” when loalding assets in the scene. I am loading a lot of assets, but I thought this simulator was meant for that after all we can see entire cities loaded in the Siggraph demo. Is there something that I should set in the rendering options ?

In this scene I have about 3000 unique assets that weight around 1Mb each.
I am running on Ryzen 5900HX with 32Gb of RAM and a 3070 with 8Gb of RAM.

Thanks in advance,

Antoine

2021-08-19 15:07:17 [309,003ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2021-08-19 15:07:17 [309,003ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
World0.usd@ <0x7f2c7c007bd0>)
2021-08-19 15:06:43 [274,870ms] [Warning] [omni.usd] Warning: in _ReportErrors at line 2830 of /buildAgent/work/ca6c508eae419cf8/USD/pxr/usd/usd/stage.cpp -- Could not open asset @omniverse://localhost/Isaac/Props/lake/dem_chunks/2_converted/8144_stl.usd@ for reference on prim @anon:0x7f2c7c002210:World0.usd@,@anon:0x7f2c7c0047c0:World0-session.usda@</World/lake_bed/lake_bed_2319>. (recomposing stage on stage @anon:0x7f2c7c002210:World0.usd@ <0x7f2c7c007bd0>)

2021-08-19 15:07:17 [309,010ms] [Error] [gpu.foundation.plugin] AccelStruct creation failed for the device: 0.
2021-08-19 15:07:17 [309,010ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2021-08-19 15:07:17 [309,166ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2021-08-19 15:07:17 [309,167ms] [Error] [gpu.foundation.plugin] AccelStruct creation failed for the device: 0.
2021-08-19 15:07:17 [309,167ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2021-08-19 15:07:17 [309,167ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2021-08-19 15:07:17 [309,167ms] [Error] [gpu.foundation.plugin] AccelStruct creation failed for the device: 0.
2021-08-19 15:07:17 [309,168ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2021-08-19 15:07:17 [309,168ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2021-08-19 15:07:17 [309,168ms] [Error] [gpu.foundation.plugin] subAllocate() failed for device 0, Buffer size: 162816
2021-08-19 15:07:17 [309,168ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2021-08-19 15:07:17 [309,168ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2021-08-19 15:07:17 [309,168ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2021-08-19 15:07:17 [309,169ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2021-08-19 15:07:17 [309,169ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2021-08-19 15:07:17 [309,169ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2021-08-19 15:07:17 [309,169ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2021-08-19 15:07:17 [309,170ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2021-08-19 15:07:17 [309,170ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2021-08-19 15:07:17 [309,170ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2021-08-19 15:07:17 [309,171ms] [Error] [gpu.foundation.plugin] AccelStruct creation failed for the device: 0.
2021-08-19 15:07:17 [309,171ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2021-08-19 15:07:17 [309,171ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2021-08-19 15:07:17 [309,171ms] [Error] [gpu.foundation.plugin] AccelStruct creation failed for the device: 0.
2021-08-19 15:07:17 [309,171ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2021-08-19 15:07:17 [309,171ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2021-08-19 15:07:17 [309,172ms] [Error] [gpu.foundation.plugin] AccelStruct creation failed for the device: 0.
2021-08-19 15:07:17 [309,172ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2021-08-19 15:07:17 [309,172ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2021-08-19 15:07:17 [309,172ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2021-08-19 15:07:17 [309,172ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2021-08-19 15:07:17 [309,300ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2021-08-19 15:07:17 [309,300ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2021-08-19 15:07:17 [309,300ms] [Error] [gpu.foundation.plugin] subAllocate() failed for device 0, Buffer size: 370368
2021-08-19 15:07:17 [309,304ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2021-08-19 15:07:17 [309,304ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2021-08-19 15:07:17 [309,304ms] [Error] [gpu.foundation.plugin] subAllocate() failed for device 0, Buffer size: 318696
2021-08-19 15:07:17 [309,305ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2021-08-19 15:07:17 [309,305ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2021-08-19 15:07:17 [309,305ms] [Error] [gpu.foundation.plugin] subAllocate() failed for device 0, Buffer size: 368112
2021-08-19 15:07:17 [309,306ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2021-08-19 15:07:17 [309,306ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2021-08-19 15:07:17 [309,306ms] [Error] [gpu.foundation.plugin] subAllocate() failed for device 0, Buffer size: 142056
2

Hi Antoine,

We have seen some memory issues in the linux based apps which will be fixed for the next release.
Please try to load your scene in a windows based app such as Create, and see if you still see the memory issue.

kindly,
Liila

3

Hi Liila,

Thanks for the answer. I could not try on windows yet. The issue I am facing is that I use linux to build the USD file, and the code fails while I am adding assets to the scene. It fails after having loaded about 1800 unique assets or around 1.5Gb worth of USD file. Please note that if fails even if I disable the rendering.

The problem seems to be related to the number of unique assets one can load. When I was doing some debugging I was able to add over 3000 times the same asset without any issue.

I guess I could be doing something wrong when I add the assets to the scene. Could you let me know if the following code snippets are correct ?

CONFIG = {
    "experience": f'{os.environ["EXP_PATH"]}/omni.isaac.sim.python.kit',
    "renderer": "RayTracedLighting",
    "width": 128,
    "height": 72,
    "window_width": 192,
    "window_height": 108,
    "headless": True,
}

if __name__ == "__main__":

    omniverse_kit = OmniKitHelper(CONFIG)

    nucleus_server = utils.get_nucleus_server()
    asset_path = nucleus_server + "/Isaac/Props/lake/dem_chunks"

    stage = omni.usd.get_context().get_stage()
    prims = []
    utils.setup_cpu_physics(stage, "/World/physicsScene")
    X = pd.read_csv("/isaac-sim/python_samples/test2/parser.csv", header=None)
    M = pd.read_csv("/isaac-sim/python_samples/test2/files.txt", header=None, delimiter=" ")
    dct = {}
    for i, name in enumerate(M[1]):
        dct[name] = M[0][i]

    for y in range(99):
        for x in range(99):
            id = y*99 + x
            if int(X[1][X[0] == id]) == 1:
                position = Gf.Vec3d(19.8*x*100,19.8*y*100,0)
                prims = create_object("/World/lake_bed/lake_bed",stage,asset_path+"/"+dct[id],position=position,group=prims,scale=Gf.Vec3d(100,100,100), allow_physics=False)

    omni.usd.get_context().save_as_stage(nucleus_server + "/Users/test/large.usd", None)

    omniverse_kit.stop()
    omniverse_kit.shutdown()

def create_object(prefix, stage, path, position=Gf.Vec3d(0, 0, 0), rotation=Gf.Quaternion(1, Gf.Vec3d(0,0,0)), group=[], allow_physics=True, density=1000, scale=Gf.Vec3d(1.0,1.0,1.0)):
    """! Creates a 3D object from a USD file and adds it to the stage
    """
    prim_path = omni.usd.get_stage_next_free_path(stage, prefix, False)
    group.append(prim_path)
    obj_prim = stage.DefinePrim(prim_path, "Xform")
    obj_prim.GetReferences().AddReference(path)
    xform = UsdGeom.Xformable(obj_prim)

    xform = setScale(xform, scale)
    xform = setTransform(xform, rotation, position)

    if allow_physics:
        utils.setRigidBody(obj_prim, "convexHull", False)
        mass_api = UsdPhysics.MassAPI.Apply(obj_prim)
        mass_api.CreateDensityAttr(density)
    return group
4

The issue is related to how many unique mdl files are loaded, one way to reduce memory is to reuse the base mdl files across multiple assets.

Memory usage will be greatly reduced for the next release on linux. Shader caching for vulkan has been improved.

5

Thanks ! These assets should be texture free, they are only there for collisions. Is there a way that I could set their texture in python to be all the same ? Also do you have a rough ETA for the next release ?

Thanks,

Antoine

6

You can for example use OmniPBR or USDPreviewSurface and apply copies of that material to different objects. Copies of a material are effectively free memory wise if they do not have textures.

Next release is mid october if all goes well :)