Out of GPU memory but only if adding semantic labels

Omniverse Isaac Sim
1

Hello, I have a reasonably complex scene in a USD file. I can load it fine in the isaacsim GUI, and only about 8GB of the RTX 3090 24GB VRAM is used.

If in a python script I load it with omni.usd.get_context().open_stage(), then use add_update_semantics() to add labels to each object, and then try to load the world with World(), I get out of memory errors:

2024-10-02 00:41:58 [22,318ms] [Error] [carb.graphics-vulkan.plugin] Out of GPU memory allocating resource 'MemoryManager-allocate' [size: unknown]
2024-10-02 00:41:58 [22,319ms] [Error] [carb.graphics-vulkan.plugin] Failure injector rule to repro:
{
    debugName="MemoryManager-allocate",
}
2024-10-02 00:41:58 [22,319ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_OUT_OF_DEVICE_MEMORY
2024-10-02 00:41:58 [22,319ms] [Error] [carb.graphics-vulkan.plugin] vkAllocateMemory failed for flags: 0.
2024-10-02 00:41:58 [22,319ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-10-02 00:41:58 [22,319ms] [Error] [gpu.foundation.plugin] subAllocate() failed for device 0, Buffer size: 20908068
2024-10-02 00:41:58 [22,319ms] [Error] [rtx.scenedb.plugin] Out of device memory while allocating geometry buffers
2024-10-02 00:41:58 [22,320ms] [Error] [carb.graphics-vulkan.plugin] Out of GPU memory allocating resource 'MemoryManager-allocate' [size: unknown]
2024-10-02 00:41:58 [22,320ms] [Error] [carb.graphics-vulkan.plugin] Failure injector rule to repro:
{
    debugName="MemoryManager-allocate",
}

However, I noticed that if I do not add semantic labels, the world loads fine and the script runs completely fine.

Any idea why adding semantic labels would trigger memory issues?

Thanks