RAM Memory issue while using Tensorflow 1.13.0

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
1

Hi All,

I am currently working on an application which runs ssd mobilenet based Object detector in Nvidia AGX Xavier Developer kit, which is flashed with Jetpack 4.1.

The sofware specifications are as follows,

Ubuntu 18.04
tf = 1.13.0 from “pip install --pre --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v411 tensorflow-gpu”
CUDA 10.0
CUDNN 7.3.1.20

I have noticed that the algorithm is taking around 12GB out of 15.5GB RAM memory available in Xavier.

I have made use of “config = tf.ConfigProto(allow_soft_placement=True, log_device_placement=log_device)” and “config.gpu_options.allow_growth=allow_memory_growth” in the code to control the memory growth. But still I could not see any improvement in the memory allocation.

I have uninstalled tensorflow and installed it again too. But there is no much change.

The same algorithm allocates only about 2-3GB is GPU based Laptop.

Could someone explain on why there is a huge fluctuation in memory? Is this a bug or expected behavior with tensorflow 1.13? Are there any more changes to be done specific to xavier ?

Thanks in Advance !!!

Regards,
Niran

2

Hi,

You can try this configure to tell TensorFlow the expected fraction of memory to be allocated.

config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.4
session = tf.Session(config=config, ...)

Thanks.

3

Hi @AastaLLL,

Thank you for the response !!!

As you suggested, I added these lines of code in my algo. But I cant see any improvement. Not sure whether these lines of codes are really effective in my algorithm. I have added the below lines before session creation,

config = tf.ConfigProto(allow_soft_placement=True, log_device_placement=False)
config.gpu_options.per_process_gpu_memory_fraction = 0.4
config.gpu_options.allow_growth=True
session = tf.Session(config=config, ...)

Can you help me to confirm it.

Regards,
Niran

4

Hi,

Not sure why it doesn’t work on your environment.
Could you monitor the system with tegrastats and share the log with us?

sudo ./tegrastats

Thanks.

5

I’m encountering similar phenomenon with yolo but actually on TX2 not on Xavier.
The yolo I’ve used is here, https://github.com/YunYang1994/tensorflow-yolov3

I’ve successfully run yolo with JetPack 3.1 with tensorflow 1.4 without any problem.
But when I try to run yolo with JetPack 4.2 with tensorflow 1.13.1 it’d get killed 9/10 times.

Surely, tensorflow 1.13.1 seems to consume the memory aggressively.
One workaround is adding a swapfile. But that doesn’t satisfy my criteria because it gets slower.

I don’t think JetPack is the problem. Because I don’t believe L4T developers customized the later version of kernel to monitor the processes more strictly and kill the processes more often.
(JetPack 3.1 uses kernel version 4.4.15 and JetPack 4.2 uses kernel 4.9)

Thus, tensorflow 1.13.1 is very suspicious. But now I’m running out of idea.
What niranjana.arumugam8 suggests doesn’t work for my case neither.

6

Hi,

It is possible.
Since TensorFlow implementation changes quickly and it may occupy more memory to get a better performance.

Do you use our official TensorFlow version?
[url]https://devtalk.nvidia.com/default/topic/1038957/jetson-tx2/tensorflow-for-jetson-tx2-/post/5278617/#5278617[/url]

Thanks.

7

Hi,

Yes, I think I use the official TensorFlow 1.13.1 from Nvidia repository.
I followed the topic [url]https://devtalk.nvidia.com/default/topic/1038957/jetson-tx2/tensorflow-for-jetson-tx2-/post/5278617/#5278617[/url].

But I’m not sure TensorFlow 1.4 that I used was also the official.