Isaac SDK Out of Memory Error

Hi,

I am trying to run the Isaac SDK tutorial “Training Object Detection from Simulation in Docker” (Training Object Detection from Simulation in Docker — ISAAC 2021.1 documentation) on my laptop running Ubuntu 18.04 with a Nvidia Geforce 1650 with 4Gb of GRAM and am getting an “OOM when allocating tensor with shape” error. When running nvidia-smi I can see that it is running out of GPU memory. I had a couple of questions:

  • Is it possible to tune the training parameters so it used less memory? Speed of execution is not important at all in this case.

  • I also have a Jetson AGX XAVIER developer kit. Would it be better to run the training on this? I am not sure if all the packages are available for ARM or just x86.

My end goal is to be able to do POSE estimation on the Jetson for new objects that are trained via 3D models and simulation.

If anyone has any hints or tips that would be great.

Thanks,
Ben.

4GB of memory may be too little to fit the entire training set into GPU memory. You could try reducing the number of training examples (it is likely trying to load all of them at once) or downrezing the training images so that they will fit. We do not recommend training on Jetson.