Tensorflow Model loading takes longer or not able to load

Autonomous Machines Jetson & Embedded Systems Jetson Nano
1

I am trying to run series of models as part of my application which are of following size on nano

I am using jetson nano with the DC barrel jack

Model 1 : 4.2 MB (Object Detection)
Model 2 : 3.2 MB (Landmarks)
Model 3,4: 9.2 Mb (MobileNet v2)
Model 5 : 6 Mb (facenet)

I was trying to load all these models in separate tensorflow session and that is where it fails to load or it will take usually more than 10-15 minutes. I often have to restart the board and after multiple attempts of restart it will run.

I usually get around 2-2.5 GB of free memory in RAM which can be seen using the command free but tensorflow report only less than what is shown using free command

I used following commands to increase the performance
sudo jetson_clocks
sudo nvpmodel -m 0

but these commands did not reduce the loading time.

Once it loads all the models then my application runs at quite good fps, but the loading time is unpredictable which is a big concern for us.

what could be the reason or if anybody have any suggestion kindly share

2

what error message do you get? if its the famous OOM resource exhausted for tf to load multiple models you need to specify gpu_fraction for the session, otherwise it will hog as much as it can and run out of GPU memory

see https://devtalk.nvidia.com/default/topic/1052263/out-of-memory-error-from-tensorflow-any-workaround-for-this-or-do-i-just-need-a-bigger-boat-/

3

Hi,

Could you try if this configure helps?

export TF_GPU_ALLOCATOR=CUDA_MALLOC

Thanks.

4

The issue has been resolved after i changed the protobuf which uses python implemention to cpp implementation.

5

Hi,

I faced the same issue and improved the performance as gopinath.r mentioned, used protobuf 3.6.1 version and used cpp implementation. Model loading time reduced significantly, from 120s to 0.8s. However, after mid of June, there was an issue for installing tensorflow in jetson due to pip3 upgrade. One related discussion is as below:

I upgraded pip3 under sudo, and fixed the tensorflow installation issue, but all previous performance optimization is gone. Model loading time goes back to 120s. Have you seen this issue? Could you provide some suggestions?

6

Hi,

Would you also try the the latest protobuf to see if helps?

sudo pip3 install -U protobuf

Thanks.

7

Hi Aastall,

Thanks for your reply. I already use latest protobuf, but loading time is still more than 120s.

8

Hi,

We want to feedback this issue to our internal team.
Would you mind to share the detail environment setup with us, JetPack, TensorFlow, protobuf, pip version? Both 120s and 0.8s.

Thanks.

9

Hi,

please see details below. seems only Jetpack and pip3 have different versions.
0.8s:Jetpack 4.2, Tensorflow 1.13.1, protobuf 3.6.1, pip3 9.0.1 (developed a few months ago)
120s: Jetpack 4.2.1 EA, Tensorflow 1.13.1, protobuf 3.6.1, pip3 19.1.1

Many thanks

10

Hi,

Do you have a subversion of the 0.8s TensorFlow package?
For example, the latest release subversion is nv19.5.

tensorflow_gpu-1.13.1+<b><u>nv19.5</u></b>-cp36-cp36m-linux_aarch64.whl

And we don’t release a package for JetPack4.2.1 yet.
The latest TensorFlow package is still built on JetPack4.2.

Could you try to reproduce this issue by replacing v4.2.1 back to v4.2?
Thanks.

11

Hi,

I have tried in JetPack 4.2, and model loading time still takes very long. Tensorflow version is 1.13.1 + nv19.5. pip3 version 19.2.1.