Problem in tensorflow on jetson nano 2gb

Okay… So I’m trying to run this repo on my jetson nano 2gb…

I followed this official documentation for installation for tensorflow on jetson nano, while I followed Install TensorFlow 2, this official documentation to run tensorflow on windows…

So after setting the dependencies on windows…the started to run smoothly with no problem.

But after setting the dependencies on jetson nano… the code showed this error…
(1st run)

 python3 object_tracker.py --weights ./checkpoints/yolov4-tiny-416 --model yolov4 --tiny --video 0
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (933) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1
Frame #:  1
2021-02-09 21:38:01.527621: F tensorflow/core/kernels/resize_bilinear_op_gpu.cu.cc:493] Non-OK-status: GpuLaunchKernel(kernel, config.block_count, config.thread_per_block, 0, d.stream(), config.virtual_thread_count, images.data(), height_scale, width_scale, batch, in_height, in_width, channels, out_height, out_width, output.data()) status: Internal: too many resources requested for launch
Fatal Python error: Aborted

Thread 0x0000007fa7355010 (most recent call first):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py", line 60 in quick_execute
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 550 in call
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1924 in _call_flat
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/saved_model/load.py", line 106 in _call_flat
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1722 in _call_with_flat_signature
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1673 in _call_impl
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1655 in __call__
  File "object_tracker.py", line 125 in main
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 251 in _run_main
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 303 in run
  File "object_tracker.py", line 238 in <module>
Aborted (core dumped)

But as soon as this error came…I ran the code again (just to be sure) and this is the error now…
2nd run

 python3 object_tracker.py --weights ./checkpoints/yolov4-tiny-416 --model yolov4 --tiny --video 0
Traceback (most recent call last):
  File "object_tracker.py", line 238, in <module>
    app.run(main)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 303, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "object_tracker.py", line 73, in main
    saved_model_loaded = tf.saved_model.load(FLAGS.weights, tags=[tag_constants.SERVING])
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/saved_model/load.py", line 603, in load
    return load_internal(export_dir, tags, options)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/saved_model/load.py", line 633, in load_internal
    ckpt_options)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/saved_model/load.py", line 131, in __init__
    self._restore_checkpoint()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/saved_model/load.py", line 344, in _restore_checkpoint
    restore_ops = position.restore_ops()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/tracking/base.py", line 395, in restore_ops
    self._checkpoint.restore_saveables(tensor_saveables, python_saveables))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/tracking/util.py", line 297, in restore_saveables
    validated_saveables).restore(self.save_path_tensor, self.options)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saving/functional_saver.py", line 340, in restore
    restore_ops = restore_fn()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saving/functional_saver.py", line 316, in restore_fn
    restore_ops.update(saver.restore(file_prefix, options))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saving/functional_saver.py", line 111, in restore
    restored_tensors, restored_shapes=None)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saving/saveable_object_util.py", line 125, in restore
    restored_tensor = array_ops.identity(restored_tensor)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/dispatch.py", line 201, in wrapper
    return target(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/array_ops.py", line 287, in identity
    ret = gen_array_ops.identity(input, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 3992, in identity
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run Identity: Dst tensor is not initialized. [Op:Identity]

Firstly, the code was working on the laptop perfectly,but on jetson nano its showing this error…secondly…why am I getting two different errors?

What I suspect is maybe the code isn’t compactible with the tensorflow on nvidia or there is a memory problem…

Please any help is appreciated…
Also if u want to have memory logs for the 1st and the 2nd run, i have logged that…
Thank you…

Maybe you don’t have enough free RAM?

do u want to see the memory log I looged using tegrastats for both runs…?

Not sure, probably you have the same problem as described here: Aborted Error while running detect.py with yolov4-tiny-trt model in Jetson Nano · Issue #235 · hunglc007/tensorflow-yolov4-tflite · GitHub

1 Like

Oh my god…it worked…U are a saviour…Thank you!!

1 Like