Unstable omniverse - memory issues

Omniverse General Discussion
, ,
1

Hi,

I just started using Omniverse open beta with a mind for testing it out and using it to simulate a robot with ROS for collecting sensor data, RGB and semantic data for training some neural nets. I have so far gotten a stage up with some assets, a burger robot and some RGB outputs into ROS. When it is running I love it.

However, Isaac sim and create seem to crash a fair bit and I think it is because they are sucking up all my RAM. I think this is the case because initially I was using 16Gb (the min) and kept having memory issues, I then upped my RAM to 32Gb and seem to get more run time as long as I don’t add too many assets - as I add them the RAM usage goes up but never comes down even after the model has been rendered. I intend on having a complicated scene likely imported from UE with a lot of assets and at this stage this probably isn’t possible.

Is there anything I can do to prevent the memory from filling up like this besides using more RAM?

I also tried using 2 GPUs but this just made the memory fill up faster.

I am using:

  • Ubuntu 20.04
  • 1-2 2080Ti
  • 16-32Gb Ram
  • NVIDIA-SMI 465.19.01 Driver Version: 465.19.01 CUDA Version: 11.3

I also have the cache app enabled.

Any help is appreciated because I am keen to keep using Omniverse.

Cheers!

2

Thanks Stefan for your interest in Omniverse and Isaac Sim.

When does it crash, at loading your big stage or running for a while then crashing or adding more assets crashing?

Are you exporting with the scene graph instancing enabled in the UE4 connector? Also see if not exporting materials helps.

On the OV side, you can try checking Disable Material Loading in the Render Settings UI.
If you want to do the same when launching or python, one tip is hovering over these settings, it’ll give you those option strings back. You can then pass “–/app/renderer/skipMaterialLoading” when launching the isaac-sim.sh

image
image682Ă—566 34 KB

This is how to set the same thing in Python.

import carb.settings
self._settings = carb.settings.get_settings()
# adjust couple of viewport settings
self._settings.set("/app/viewport/grid/enabled", True)

When you press F8 in Isaac Sim/Create, it’ll show some stats as well. Same with Window\Statistics. It’ll be good to see if the memory is used by geometry, textures, instances, etc.

image
image1306Ă—626 196 KB

There are different rendering modes (white mode, shaded wireframe, etc) after you load the stage, under the same RTX Real Time/Path-traced switch that might give you more running time.

For certain big stages, depending on their sizes, more RAM like 64GB might require. If crashing at the beginning, more VRAM is required or use the skipMaterialLoading. We’re constantly working on optimizing memory usage for future releases as well. Thanks.

3

Another option is to reduce the unique mdl files being used for materials.
Its better to use instances of the OmniPBR material so only one shader is compiled and everything else becomes an instance of that one compiled material, reducing CPU memory usage.

4

Hi, and Welcome to the dev Forum!

A few things to consider when dealing with large stages:

You can mitigate the RAM buildup by composing the stage in multiple USDs (for example everything that is static goes on a separate USD), then adding the subcomponents as a reference on a main usd scene, then set it to “Instanceable”, this will indicate to the parser that the structure of the usd inside that reference is unchangeable, and it will be pruned from processing (You will notice that that section will be grayed out in the stage editor, so you won’t be able to directly select sub-components or make changes whatsoever).

image
image434Ă—849 37.7 KB

Another benefit to the “Instanceable” option on referenced usds is that it allows for component reuse in the RAM. For example, if you have a same box or crate referenced multiple times in your composed scene, it will load it only once, and replicate it on the stage, referring to the same portion on the RAM.

1 Like
5

Hi @HaiLocLu, @Hammad_M and @rgasoto,

Thanks for your replys. I ended up upping my RAM to 64GB and the issue has resolved itself.

@HaiLocLu The issue was when I loaded a stage (rarely) when the textures were being loaded, and when I added new assets to a stage that had some already in it. The stage itself I would not say was large - see attached image.

2021-08-04_16-31
2021-08-04_16-311434Ă—877 75.4 KB

I grab some stats on the shown scene and had the following Tot GPU: 2946, Other: 1652, the rest < 500 but my systam RAM was ~25 Gb while stable.

I had not yet tried importing UE4 scenes at the time of the post but now have successfully - but with more RAM

@Hammad_M thanks for the tip with the materials

@rgasoto thanks for the USD and instanceable tips.

While playing with Isaac I have also noticed a couple of things that cause crashes:

  • Two view ports open and clicking on the Semantic (or Instance and possibly more I have not exhausted the options) Segmentation checkbox in Synthetic Data Sensors of the second view port will cause an instant shutdown of Isaac. This does not happen in view port 1 (even if view port 2 is up). I have not tried more view ports.
  • At times toggling instanceable.
  • I have had Isaac freeze indefinitely when clicking on the play button with a imported URDF robot. Restarts did not help the only thing I could figure was to delete the import and start again (the robot now does not cause a freeze). This has only happened once and I have not been able to replicate.

Looking forward to more updates and optimizations!