Continuously decreasing available memory

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

If I run tensorflow just after start I have sufficient memory

I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:874] ARM64 does not support NUMA - returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: 
name: GP10B
major: 6 minor: 2 memoryClockRate (GHz) 1.3005
pciBusID 0000:00:00.0
Total memory: 7.67GiB
Free memory: 6.01GiB

But as I execute it multiple times it gradually decreases to the point tensorflow would not run.
For example in the next run:

I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:874] ARM64 does not support NUMA - returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: 
name: GP10B
major: 6 minor: 2 memoryClockRate (GHz) 1.3005
pciBusID 0000:00:00.0
Total memory: 7.67GiB
Free memory: 6.01GiB

Closing the terminal does not help and I have to restart the system.
Is it possible to clean up memory by using some command? This happens even if my programs
exits correctly. Can I run some command to clear the memory?

2

You might try running your program under Valgrind and see what it says:

valgrind --leak-check=yes <your_program> <program_arguments...>

It does sound like a memory leak, but that’s a bit too general…Valgrind might narrow it down.

3

Memory leaks that last after a process has exited are, by definition, kernel and/or driver bugs, and thus valgrind in userspace won’t find them.

4

Hi,

  1. The memory amount in comment #1 is the same. Please help us recheck the data.

  2. Could you also share the info of meminfo?

cat /proc/meminfo

Thanks.