Hi,
The fix is available now.
Please check JetPack SDK | NVIDIA Developer
Hi,
JetPack 3.1 really can sovle this problem.
Thanks a lot.
Hi everyone,
Im trying to use my Jetson Tx2 with Jetpack 3.2 gpu with tensorflow wihch works but Its too slow and I got the following errors:
018-04-26 13:14:40.069245: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.53GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-04-26 13:14:40.078779: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 442.69MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-04-26 13:14:40.091886: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 798.28MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-04-26 13:14:40.110919: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.04GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-04-26 13:14:40.152810: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.07GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-04-26 13:14:40.331434: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.07GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-04-26 13:14:40.350748: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.07GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-04-26 13:14:40.408377: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.13GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-04-26 13:14:40.601111: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.13GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-04-26 13:14:40.634236: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.14GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
I sees there is RAM issue. Can I solve this with allocating swap like what you suggested?
Hi,
Please try the following configuration:
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.Session(config=config, ...)
If the error keeps, please share your tegrastats results with us.
sudo ./tegrastats
Thanks.
-
I am using JetPack3.2 and CUDA9
-
I try to deploy a Lightweight Mask RCNN Model on the Jetson.
On my host machine (with 4gb GPU mem) takes the model <2GB memory.
So a lot less then the 8GB available on the Jetson.
Still it gives me memory errors stating that too many resources were requested to launch.
-
Furthermore the model is deployable if i set CUDA_VISIBLE_DEVICES to ‘-1’ as stated in this thread https://devtalk.nvidia.com/default/topic/1033209/general-question-about-jetsons-gpu-cpu-shared-memory-usage/?offset=13#5265168
-
For me it seems that there is still a restriction / or general inability on the jetson to load larger memory chunks. In fact i was so far not able to load anything bigger than an SSD Model, which is at the end very poorly…
-
What are those 8GB? How can they be effectively used for inference? It seems they are good for nothing…
Hi,
It’s recommended to apply some experiment to figure out the real cause first.
Could you test the maximal accessible memory chunk with CUDA API directly?
[url]https://devtalk.nvidia.com/default/topic/1013464/jetson-tx2/gpu-out-of-memory-when-the-total-ram-usage-is-2-8g/post/5170376/#5170376[/url]
We can allocate around 7.xGb memory on TX2.
If you also can allocate 7Gb memory but hit out of memory issue with TensorFlow API.
This limitation may come from TensorFlow implementation rather than Jetson system.
Thanks.
is there any fix one this?
Hi, civilman
What is your error?
This is a pretty old issue and is already fixed in JetPack 3.1.
The original bug is that CUDA cannot allocate more than half of the physical memory since we separate it for CPU and GPU.
This issue won’t occur in our latest JetPack3.3.
Thanks.