Isaac Sim Constantly Crashing on Google Cloud VM

Hello,

I’m running a remote client of Isaac Sim 2020.2.2 in a container on a Google Cloud VM. The machine has a single NVIDIA Tesla T4, 16 vCPUs, 60GB RAM, and a 100GB SSD.

The container has been crashing frequently. Sometimes I am able to replicate the crash with certain actions like moving certain prims around or running a simulation, but most of the time it is completely random. The remote client freezes for a few seconds before closing along with the entire container. Below is the error message when it crashes.

image

Here is the console error log from Google Cloud.

google-cloud-error-log.txt (1018.8 KB)

Would anyone be able to tell me what is going wrong and if there is a fix for it?

Thanks in advance!

Edit: Just a little more info - I went through most of the JetBot sample as well as played with the Universal Robots sample, and those ran without any issues. I started experiencing frequent crashes when I imported a STEP file of a robotic arm and began cleaning up the Stage (merging meshes, getting rid of empty prims, etc) and setting up the physics.

Hi, it looks like a crash in the physics collision. If possible could you provide the STEP file that was imported when the crash happens?

Please also provide some steps that could be used to reproduce the crash and does it happen with any assets in our samples.

Thanks.

It might partly be caused by the physics, but I also experienced crashes before even adding any physics components to the model.

This is the STEP file that I’ve been working with:
CR3 with grippers (1).step (22.5 MB)

I can’t seem to reproduce the crash with the exact same actions on the same parts anymore, but it still crashes randomly when I move prims around into other prims (in the Stage) and merge random prims.

I’ve gone through the Jetson Bot sample and played around with the UR10 sample, and neither of those crashed.

I was able to successfully import and work with the imported step file without any crashes.

Would you be able to provide an error log from before setting up any physx component to it to help us dial where the issue may be?

If you could provide us the faulty imported USD we may try to replicate the issue here from it too.

Hi, thank you for looking into this.

Here is the console log from Google Cloud and USD (flattened from the STEP file) that I’ve been using, and I made it crash just now by moving prims around randomly.

console-log-april-5-2021.txt (1018.7 KB)
cr3-no-physics.usd (2.0 MB)

The error message in my Ubuntu terminal is always the same:

I was able to reproduce it crashing with your provided USD file.

When saving flattened, did you use the main menu “save Flattened As”, or the built-in step importer “Save Flattened” Option?

The Step importer Save Flattened performs a cleanup on the materials layer instanceable components when flattening, which are the root of the issue you are seeing.

To properly finish a flattened import, scroll to the bottom of the step importer window, click on Finish Import, select the folder in which you want to save your USD, and then click on “Save Flattened”:

I would recommend you attempt to re-import the STEP file and finish using the “Save Flattened” option from the importer. That should stop you from seeing any errors.

Regardless of that, this crash will be dealt on for future releases. Thanks for providing us feedback!

I used the main menu “save Flattened As”, so that is probably why it is crashing. I’ll try to re-import and properly do the “Save Flattened” step!

Thank you for your help!