Unknown: Failed to get convolution algorithm on Tx1

when i run model.predict on python3.6.9 tenforflow 1.14.0 on Tx1, ( but it works on my PC computer)

i got the error as following:

i do not know how to do , please give me some suggestion, thanks.


Using TensorFlow backend.
2020-02-21 15:16:41.816608: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2020-02-21 15:16:41.817660: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x15d267f0 executing computations on platform Host. Devices:
2020-02-21 15:16:41.817734: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): ,
2020-02-21 15:16:41.851982: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2020-02-21 15:16:41.923552: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2020-02-21 15:16:41.923859: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x16efc0c0 executing computations on platform CUDA. Devices:
2020-02-21 15:16:41.923925: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2020-02-21 15:16:41.924565: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2020-02-21 15:16:41.924704: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2020-02-21 15:16:41.924800: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2020-02-21 15:16:41.931383: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2020-02-21 15:16:41.952524: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2020-02-21 15:16:41.997926: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2020-02-21 15:16:42.027057: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2020-02-21 15:16:42.046516: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2020-02-21 15:16:42.090524: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-02-21 15:16:42.091611: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2020-02-21 15:16:42.092135: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2020-02-21 15:16:42.092499: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-02-21 15:16:42.092779: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2020-02-21 15:16:48.269050: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-02-21 15:16:48.269122: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2020-02-21 15:16:48.269158: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2020-02-21 15:16:48.274550: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2020-02-21 15:16:48.274901: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2020-02-21 15:16:48.275255: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 325 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
/home/pi/.local/lib/python3.6/site-packages/keras/engine/saving.py:341: UserWarning: No training configuration found in save file: the model was not compiled. Compile it manually.
warnings.warn('No training configuration found in save file: ’
WARNING:tensorflow:From /home/pi/.local/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /home/pi/.local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

winList,winPosList: (500, 3, 160, 200) (500,)
2020-02-21 15:17:04.194672: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2020-02-21 15:17:05.602767: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-02-21 15:17:10.609638: E tensorflow/stream_executor/cuda/cuda_dnn.cc:319] Loaded runtime CuDNN library: 7.3.1 but source was compiled with: 7.5.0. CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2020-02-21 15:17:10.636918: E tensorflow/stream_executor/cuda/cuda_dnn.cc:319] Loaded runtime CuDNN library: 7.3.1 but source was compiled with: 7.5.0. CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
Traceback (most recent call last):
File “pocWuhu_first_nnGW_linux3.py”, line 70, in
runMain_predict_GW_linux()
File “pocWuhu_first_nnGW_linux3.py”, line 47, in runMain_predict_GW_linux
predictResult=nnPredict(ds,featureExtractorModel,gwModel)
File “pocWuhu_first_nnGW_linux3.py”, line 26, in nnPredict
embedding=featureExtractorModel.predict(ds)
File “/home/pi/.local/lib/python3.6/site-packages/keras/engine/training.py”, line 1462, in predict
callbacks=callbacks)
File “/home/pi/.local/lib/python3.6/site-packages/keras/engine/training_arrays.py”, line 324, in predict_loop
batch_outs = f(ins_batch)
File “/home/pi/.local/lib/python3.6/site-packages/tensorflow/python/keras/backend.py”, line 3292, in call
run_metadata=self.run_metadata)
File “/home/pi/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1458, in call
run_metadata_ptr)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
(0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node conv1d_1/convolution}}]]
[[embedding/BiasAdd/_545]]
(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node conv1d_1/convolution}}]]
0 successful operations.
0 derived errors ignored.
pi@zya-desktop:~/eric/MFL$


pi@zya-desktop:~/eric/MFL$ pip3 list
Package Version


absl-py 0.9.0
apturl 0.5.2
asn1crypto 0.24.0
astor 0.8.1
beautifulsoup4 4.6.0
blinker 1.4
Brlapi 0.6.6
certifi 2018.1.18
chardet 3.0.4
cryptography 2.1.4
cupshelpers 1.0
defer 1.0.6
distro-info 0.18ubuntu0.18.04.1
feedparser 5.2.1
gast 0.2.2
google-pasta 0.1.8
graphsurgeon 0.3.2
grpcio 1.26.0
h5py 2.10.0
html5lib 0.999999999
httplib2 0.9.2
idna 2.6
Keras 2.3.1
Keras-Applications 1.0.8
Keras-Preprocessing 1.1.0
keyring 10.6.0
keyrings.alt 3.0
language-selector 0.1
launchpadlib 1.10.6
lazr.restfulclient 0.13.5
lazr.uri 1.0.3
louis 3.5.0
lxml 4.2.1
macaroonbakery 1.1.3
Mako 1.0.7
Markdown 3.1.1
MarkupSafe 1.0
numpy 1.18.1
oauth 1.0.1
oauthlib 2.0.6
pip 19.3.1
protobuf 3.11.2
pycairo 1.16.2
pycrypto 2.6.1
pycups 1.9.73
pygobject 3.26.1
PyJWT 1.5.3
pymacaroons 0.13.0
PyNaCl 1.1.2
pyRFC3339 1.0
python-apt 1.6.5+ubuntu0.2
python-debian 0.1.32
pytz 2018.3
pyxdg 0.25
PyYAML 3.12
requests 2.18.4
requests-unixsocket 0.1.5
scipy 1.3.3
SecretStorage 2.3.1
setuptools 44.0.0
simplejson 3.13.2
six 1.11.0
ssh-import-id 5.7
system-service 0.3
systemd-python 234
tensorboard 1.14.0
tensorflow-estimator 1.14.0
tensorflow-gpu 1.14.0+nv19.10
tensorrt 5.0.6.3
termcolor 1.1.0
ubuntu-drivers-common 0.0.0
uff 0.5.5
unattended-upgrades 0.1
unity-scope-calculator 0.1
unity-scope-chromiumbookmarks 0.1
unity-scope-colourlovers 0.1
unity-scope-devhelp 0.1
unity-scope-firefoxbookmarks 0.1
unity-scope-manpages 0.1
unity-scope-openclipart 0.1
unity-scope-texdoc 0.1
unity-scope-tomboy 0.1
unity-scope-virtualbox 0.1
unity-scope-yelp 0.1
unity-scope-zotero 0.1
urllib3 1.22
wadllib 1.3.2
webencodings 0.5
Werkzeug 0.16.0
wheel 0.30.0
wrapt 1.11.2
xkit 0.0.0
zope.interface 4.3.2
WARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.
You should consider upgrading via the ‘pip install --upgrade pip’ command.
pi@zya-desktop:~/eric/MFL$

the model is resCNN2D

Hi,

The issue is from the TensorFlow package:

Loaded runtime CuDNN library: 7.3.1 but source was compiled with: 7.5.0. CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a

This indicates that your cuDNN package is not compatible with the one TensorFlow is built with.
Please noticed that you will need to use the same JetPack as TensorFlow built with.

It’s recommended to give JetPack4.3 and our official release a try:
https://docs.nvidia.com/deeplearning/frameworks/install-tf-jetson-platform/index.html

Thanks.