TensorFlow memory growth: cannot allocate memory in static TLS block

Robotics & Edge Computing Jetson Systems Jetson Nano
1

I’ve tried two ways of limiting GPU growth: session config and at the GPU level

For both, I get this error:

ImportError: /usr/lib/aarch64-linux-gnu/libgomp.so.1: cannot allocate memory in static TLS block

Upon googling, I found people recommending setting this in bashrc: export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libgomp.so.1:/$LD_PRELOAD

This unfortunately did not help my problem
This is what I’m using
Python 3.6.9
Tensorflow 2.4.1 (also tried 2.5.0+nv21.8)
JetPack 4.6.2

Full stack trace:

2022-10-11 10:14:29.349921: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 449 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
1 Physical GPUs, 1 Logical GPUs
Traceback (most recent call last):
  File "/home/user/face.py", line 1, in <module>
    import cv2
  File "/usr/lib/python3.6/dist-packages/cv2/__init__.py", line 89, in <module>
    bootstrap()
  File "/usr/lib/python3.6/dist-packages/cv2/__init__.py", line 79, in bootstrap
    import cv2
ImportError: /usr/lib/aarch64-linux-gnu/libgomp.so.1: cannot allocate memory in static TLS block

Any ideas on how to fix this?

2

Managed to make tensorflow 2.5.0+nv21.8 work with this code:

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
config.gpu_options.per_process_gpu_memory_fraction = 0.4
session = tf.Session(config=config, ...)

2.4 would not work

4

Hi,

Please noted that there are some dependencies between the TensorFlow package and JetPack software.
If you are using JetPack 4.6.2, which is a minor update from JetPack 4.6.1, it’s recommended to install v2.7.0+nv22.1 instead:

https://developer.download.nvidia.com/compute/redist/jp/v461/tensorflow/

Thanks.

5

Thank you! I’ll do that.
Might be worth updating the official Tensorflow thread with a link for the 4.6 and 4.6.1 users for posterity!

6

Thanks.

Already add some information to the topic.