TensorFlow object detection inference out of memory

Robotics & Edge Computing Jetson & Embedded Systems Jetson Nano
1

I’m making an object detection tool using TensorFlow and a Jetson Nano. I have trained a R-FCN Resnet101 model on a CPU and was trying to do inference on a Jetson Nano. The inference uses about 4 GB of memory and my Nano has 3 GB of free memory, so when I run inference, the process starts with no error and after a while the board freezes and the process is killed (also the OS shows a message informing that the memory available is too low).

I’d like to know:

  1. Is there a way of increasing the memory available for inference?

  2. Does the swapfile works for inference?

  3. Does TensorRT decreases the memory consumption for inference?

2

Hi,

1. No. But Xavier has 8G memory and you can give it a try.

2. No. Swap only increase memory for CPU usage. It cannot be accessed by GPU.

3. Suppose yes. But I think your model is too complicated to use on the Nano.

Thanks.

3

Not the original poster of this thread but I can say that TensorRT can drastically decrease memory usage and inference time of your network.

I personally tested Yolo V3 with custom classes on a Jetson-TX2 (with 8GB of RAM) in two frameworks:

  1. Keras with TensorFlow
  2. TensorRT

I tested inference for batch size 1 and got the following results:

  • Using Keras with TensorFlow: the network used about 5GB of RAM and inference time was 400ms. Given the fact that I already had 1.5GB of RAM used by other processes, I was using almost all the memory I had and indeed I sometimes got an Out Of Memory error.
  • Using TensorRT: the network used about 1.5GB of RAM and inference time was 140ms with no memory issues. It even worked for me on a Jetson Nano (which has 4GB of RAM).

I don’t know how heavy R-FCN Resnet101 is in comparison to Yolo V3, but I hope I helped.

By the way, doesn’t Jetson Xavier have 16GB of RAM rather than 8?

4

Both is correct ,as there is a (GB and a 16GB version of the Xavier available:

5

Does this mean “object detection tool using TensorFlow and a Jetson Nano” not do’able on Jetson Nano at all?
Thx

6

Thank you all for the answers.

I used tf-trt engine in my RFC Resnet101 model and for inference, it took about 40 minutes to run, it took all the memory available and it didn’t work (it showed another memory error). A simpler model like SSD Mobilenet doesn’t work for my case, I need an assertiveness level and Mobilenet doesn’t come near the level I need (I don’t need FPS, I need something that makes the less mistakes possible).

What I’m seeing is that the Nano is still not an option if you need good assertiveness and a robust model. It’s fine for small models that use low memory but are not so reliable in its results.

I gave up on TensorFlow and I’m trying YOLO now. Let’s see if I can make it work.

7

Hi,

Resnet101 is a complicated model.

Would you mind to check how much memory it takes when you launch it on the desktop.
If the model doesn’t occupied more than 4G, you can try to convert it into the TensorRT to see if helps.

Thanks.