Jetson Nano problem in running tensorflow and using GPUs

Problem statement:
I’m investigating the performance of Jetson Nano for running some object detection model based on YOLO architecture to be used in a new product we develop.
I have followed the instruction on Nvidia website downloading the image and writing it to the SD card (JetPack 4.6 version is installed).
I have installed pip and tensorflow + some additional python libraries. Everything seems to work fine except the following issues:

  1. I’ve installed TensorFlow 2.6.0 in accordance with the description on Nvidia website.
    Seemingly, TensorFlow is installed properly as when I load the libraries there are no errors. However, when I load a (tf) model (which I’ve created on a different system and saved on an hdf5 file), python fires and exception. Needless to say that this exception is not occurring on the system created the model.

This is the command on the Jetson:
avner@avner-desktop:~/myPythonFiles$ python3 benchmarkObjectDetection.py

This is the exception:

2021-10-10 17:30:16.839340: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/api/keras/optimizers
Traceback (most recent call last):
File “benchmarkObjectDetection.py”, line 25, in
nn_loaded_model = tf.keras.models.load_model(filepath_for_model, compile = False) # IMPORTANT! the compile = False input is crucial for loading the model as the custom loss function is unknown in the compilation phase; we can only use the model for prediction this way.
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/lazy_loader.py”, line 62, in getattr
module = self._load()
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/lazy_loader.py”, line 45, in _load
module = importlib.import_module(self.name)
File “/usr/lib/python3.6/importlib/init.py”, line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 994, in _gcd_import
File “”, line 971, in _find_and_load
File “”, line 941, in _find_and_load_unlocked
File “”, line 219, in _call_with_frames_removed
File “”, line 994, in _gcd_import
File “”, line 971, in _find_and_load
File “”, line 941, in _find_and_load_unlocked
File “”, line 219, in _call_with_frames_removed
File “”, line 994, in _gcd_import
File “”, line 971, in _find_and_load
File “”, line 941, in _find_and_load_unlocked
File “”, line 219, in _call_with_frames_removed
File “”, line 994, in _gcd_import
File “”, line 971, in _find_and_load
File “”, line 955, in _find_and_load_unlocked
File “”, line 665, in _load_unlocked
File “”, line 678, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/usr/local/lib/python3.6/dist-packages/keras/init.py”, line 25, in
from keras import models
File “/usr/local/lib/python3.6/dist-packages/keras/models.py”, line 20, in
from keras import metrics as metrics_module
File “/usr/local/lib/python3.6/dist-packages/keras/metrics.py”, line 26, in
from keras import activations
File “/usr/local/lib/python3.6/dist-packages/keras/activations.py”, line 20, in
from keras.layers import advanced_activations
File “/usr/local/lib/python3.6/dist-packages/keras/layers/init.py”, line 23, in
from keras.engine.input_layer import Input
File “/usr/local/lib/python3.6/dist-packages/keras/engine/input_layer.py”, line 21, in
from keras.engine import base_layer
File “/usr/local/lib/python3.6/dist-packages/keras/engine/base_layer.py”, line 43, in
from keras.mixed_precision import loss_scale_optimizer
File “/usr/local/lib/python3.6/dist-packages/keras/mixed_precision/loss_scale_optimizer.py”, line 18, in
from keras import optimizers
File “/usr/local/lib/python3.6/dist-packages/keras/optimizers.py”, line 26, in
from keras.optimizer_v2 import adadelta as adadelta_v2
File “/usr/local/lib/python3.6/dist-packages/keras/optimizer_v2/adadelta.py”, line 22, in
from keras.optimizer_v2 import optimizer_v2
File “/usr/local/lib/python3.6/dist-packages/keras/optimizer_v2/optimizer_v2.py”, line 37, in
“/tensorflow/api/keras/optimizers”, “keras optimizer usage”, “method”)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/monitoring.py”, line 361, in init
len(labels), name, description, *labels)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/monitoring.py”, line 135, in init
self._metric = self._metric_methods[self._label_length].create(*args)
tensorflow.python.framework.errors_impl.AlreadyExistsError: Another metric with the same name already exists.

  1. I made a small test to check if TensorFlow (2.6.0) uses the GPUs (as it should by default if Nvidia GPUs are installed; this is true for the new TensorFlow versions…) by running a simple mathematical operation using TensorFlow. I’m getting the following warning which basically says that the GPUs are not used (the calculation is done with the CPU and returned as a tensor results):

This is the command on the Jetson:
avner@avner-desktop:~/myPythonFiles$ python3 test_tensorflow_on_jetson.py

This is the return warnings:

2021-10-11 09:32:58.899943: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1017] ARM64 does not support NUMA - returning NUMA node zero
2021-10-11 09:32:59.279358: W tensorflow/stream_executor/platform/default/dso_loader.cc:65] Could not load dynamic library ‘libcutensor.so.1’; dlerror: libcutensor.so.1: cannot open shared object file: No such file or directory
2021-10-11 09:32:59.507354: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1835] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at GPU 支持  |  TensorFlow for how to download and setup the required libraries for your platform.
Skipping registering GPU devices…
tf.Tensor(0.13533528, shape=(), dtype=float32)

I would very much appreciate any help to solve these issues.
Thanks a lot!

1 Like

hi @avnersafrani2018

Can you please try working with Tensor flow v2.5.0 instead. You can refer the documentation here on installation steps for any desired version Installing TensorFlow For Jetson Platform :: NVIDIA Deep Learning Frameworks Documentation

Thank you

Hi,
Thanks for getting back to me. I’ve already done this and in addition installed vn container. It works very well.
Thanks.
Avner

בתאריך יום ג׳, 19 באוק׳ 2021, 9:50, מאת Ankithsingh Work via NVIDIA Developer Forums ‏<nvidia@discoursemail.com>: