CUDA_ERROR_LAUNCH_FAILED on Jetson Nano (4GB), Tensorflow 2.5.0, Python 3.6.9

Hi, I have a custom face verification siamese network based on VGG Face that keeps failing on start with the following error:

2021-08-09 00:22:50.224593: W tensorflow/core/common_runtime/bfc_allocator.cc:467] *************************************************************************************_______________
2021-08-09 00:22:50.346121: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:818] failed to record completion event; therefore, failed to create inter-stream dependency
2021-08-09 00:22:50.346237: E tensorflow/stream_executor/cuda/cuda_driver.cc:1175] failed to enqueue async memcpy from host to device: CUDA_ERROR_LAUNCH_FAILED: unspecified launch failure; GPU dst: 0xf0097bc00; host src: 0x20175600; size: 4=0x4
2021-08-09 00:22:50.361214: E tensorflow/stream_executor/stream.cc:356] Error recording event in stream: Error recording CUDA event: CUDA_ERROR_LAUNCH_FAILED: unspecified launch failure; not marking stream as bad, as the Event object may be at fault. Monitor for further errors.
2021-08-09 00:22:50.361288: E tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_LAUNCH_FAILED: unspecified launch failure
2021-08-09 00:22:50.361381: F tensorflow/core/common_runtime/device/device_event_mgr.cc:221] Unexpected Event status: 1
Aborted (core dumped)

I installed tensorflow from this documentation here:
https://docs.nvidia.com/deeplearning/frameworks/install-tf-jetson-platform/index.html

This works on my Mac OSX machine but refuses to run on the Jetson Nano. I’m unsure if this is relevant but I was first getting an OOM error and I have inserted the following lines of code below my imports:

device = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(device[0], True)
tf.config.experimental.set_virtual_device_configuration(device[0], [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=1024)])

I have also linked a pastebin and attached a textfile that has the full output on it:

CUDA Error in Tensorflow for Jetson Nano.txt (23.5 KB)


I have tried to run cuda-memcheck but it keeps failing (I think) due to a “syntax error” which is not a syntax error. The error is below:

========= CUDA-MEMCHECK
  File "main.py", line 46
    text = f"Last seen at {str(times)[:-10]}"
                                            ^
SyntaxError: invalid syntax
========= ERROR SUMMARY: 0 errors

I’d appreciate any help on this as I am at my wits end now! I saw some posts on the forum that were talking about a Python, CUDA and CUDNN version mismatch, however this Jetson is running a stock Jetpack 4.6, with tensorflow installed from the official Nvidia website so I’m hoping its not that.

I tried to install a Python 3.7 and 3.8 version however tensorflow isn’t available (unless you build by source which I am not too confident in doing at the moment but if thats the only solution then will do!)

Hope someone can help!

Hi,

Since JetPack 4.6 is just released, I’m afraid that you are facing some compatibility issues.
Please make sure that you are using v46 download link.

$ sudo pip3 install --pre --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v46 tensorflow

Thanks.

Hi AastaLLL,

Thanks for replying! I was pretty sure I downloaded the v46 link but I tried it again nonetheless. This was the output for that command:

Looking in indexes: https://pypi.org/simple, https://developer.download.nvidia.com/compute/redist/jp/v46
Requirement already satisfied: tensorflow in /usr/local/lib/python3.6/dist-packages (2.5.0+nv21.7)
Requirement already satisfied: keras-preprocessing~=1.1.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (1.1.2)
Requirement already satisfied: absl-py~=0.10 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (0.13.0)
Requirement already satisfied: numpy~=1.19.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (1.19.4)
Requirement already satisfied: protobuf>=3.9.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (3.17.3)
Requirement already satisfied: tensorflow-estimator<2.6.0,>=2.5.0rc0 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (2.5.0)
Requirement already satisfied: wrapt~=1.12.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (1.12.1)
Requirement already satisfied: keras-nightly~=2.5.0.dev in /usr/local/lib/python3.6/dist-packages (from tensorflow) (2.5.0.dev2021032900)
Requirement already satisfied: h5py~=2.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (2.10.0)
Requirement already satisfied: opt-einsum~=3.3.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (3.3.0)
Requirement already satisfied: astunparse~=1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (1.6.3)
Requirement already satisfied: tensorboard~=2.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (2.6.0)
Requirement already satisfied: flatbuffers~=1.12.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (1.12)
Requirement already satisfied: gast==0.4.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (0.4.0)
Requirement already satisfied: termcolor~=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (1.1.0)
Requirement already satisfied: grpcio~=1.34.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (1.34.1)
Requirement already satisfied: wheel~=0.35 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (0.36.2)
Requirement already satisfied: six~=1.15.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (1.15.0)
Requirement already satisfied: typing-extensions~=3.7.4 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (3.7.4.3)
Requirement already satisfied: google-pasta~=0.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow) (0.2.0)
Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard~=2.5->tensorflow) (0.6.1)
Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard~=2.5->tensorflow) (1.34.0)
Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard~=2.5->tensorflow) (49.6.0)
Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard~=2.5->tensorflow) (2.0.1)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from tensorboard~=2.5->tensorflow) (0.4.5)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard~=2.5->tensorflow) (3.3.4)
Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard~=2.5->tensorflow) (2.26.0)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard~=2.5->tensorflow) (1.8.0)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow) (4.2.2)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow) (4.7.2)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow) (0.2.8)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.5->tensorflow) (1.3.0)
Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.6/dist-packages (from markdown>=2.6.8->tensorboard~=2.5->tensorflow) (4.6.3)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.6/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow) (0.4.8)
Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow) (2.0.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow) (1.22)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow) (2.6)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow) (2018.1.18)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.5->tensorflow) (3.1.1)
Requirement already satisfied: dataclasses in /usr/local/lib/python3.6/dist-packages (from werkzeug>=0.11.15->tensorboard~=2.5->tensorflow) (0.8)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata->markdown>=2.6.8->tensorboard~=2.5->tensorflow) (3.5.0)

I ran my program again and got the same errors :(

Thanks

Hi,

total_region_allocated_bytes_: 1073741824 memory_limit_: 1073741824 available bytes: 0 curr_region_allocation_bytes_: 2147483648

Based on the error, it might be an OOM issue.
But TensorFlow usually reports an OOM error rather than a CUDA launch failure.

To verify it further, would you mind checking the system status at the same time?

$ sudo tegrastats

And share memory usage with us?

Thanks.