GPU out of memory with TensorFlow and JetPack 4.2

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

Hi,

I’m trying to use my Jetson TX2 with Jetpack 4.2 with official TensorFlow but it seems very slow and I got the following errors:

2019-05-09 11:58:44.080454: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1,73GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-05-09 11:58:44.290104: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1,19GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-05-09 11:58:44.464373: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 610,31MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-05-09 11:58:44.768497: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 550,42MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-05-09 11:58:44.871052: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1,05GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-05-09 11:58:44.936249: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2,13GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-05-09 11:58:45.677759: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3,54GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.

How can I solve it?
Thanks

2

Hi,

Based on the log, you are running out of memory.
Please noticed that there are only 8G memory on the TX2.

ran out of memory trying to allocate 2,13GiB

You can also run tegrastats at the time to double confirm if the memory is fully allocated.

sudo tegrastats

Thanks.

3

having the same issue.
when I run tegrastats it shows 5155/7852MB before I run my object detection script. (I’m just running chromium.

Is this normal???
I’m having a 24" display connected with HDMI, could this be the problem?

when I close everything I get 14 FPS with SSD Mobilenet TensorRT FP32 batchsize 2.
If I run the same net wih FP16 bs 2 I have the same.
Does this mean, that I have some other bottleneck?

4

Hi,

Have you maximized the device performance first?

sudo jetson_clocks.sh

Thanks.

5

yes I did.

Is it normal, that FP 32 and 16 give the same results fps wise?

6

Hi,

Usually, fp16 should have a better performance.

It’s recommended to profile the application with nvvp first.
If the pipeline bottleneck is in the camera or display, the improvement of fp16 mode will be limited.

Thanks.