Hello, I have a python script, a tensorflow network for image classification, it can run with cpu, but can’t with gpu. My device is 2080ti, system is ubuntu 18.04.
this is the error:
WARNING:tensorflow:From /home/fjuser/.conda/envs/ENetbylxb_py3.6/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
2022-06-01 13:34:30.684410: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2022-06-01 13:34:31.322219: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-01 13:34:31.322590: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x559dfba32290 executing computations on platform CUDA. Devices:
2022-06-01 13:34:31.322605: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): NVIDIA GeForce RTX 2080, Compute Capability 7.5
2022-06-01 13:34:31.323985: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3000000000 Hz
2022-06-01 13:34:31.324470: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x559dfab59050 executing computations on platform Host. Devices:
2022-06-01 13:34:31.324508: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2022-06-01 13:34:31.324578: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: NVIDIA GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.71
pciBusID: 0000:01:00.0
totalMemory: 7.79GiB freeMemory: 7.69GiB
2022-06-01 13:34:31.324589: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2022-06-01 13:34:31.324995: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-06-01 13:34:31.325004: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2022-06-01 13:34:31.325023: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2022-06-01 13:34:31.325082: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7477 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)
Epoch 1/10
2022-06-01 13:43:54.934052: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2022-06-01 13:43:54.934143: W ./tensorflow/stream_executor/stream.h:2099] attempting to perform DNN operation using StreamExecutor without DNN support
2022-06-01 13:43:54.936390: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2022-06-01 13:43:54.937281: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
Traceback (most recent call last):
File "train/train.py", line 190, in <module>
workers=1)
File "/home/fjuser/.conda/envs/ENetbylxb_py3.6/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 737, in fit
initial_epoch=initial_epoch)
File "/home/fjuser/.conda/envs/ENetbylxb_py3.6/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1426, in fit_generator
initial_epoch=initial_epoch)
File "/home/fjuser/.conda/envs/ENetbylxb_py3.6/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_generator.py", line 191, in model_iteration
batch_outs = batch_function(*batch_data)
File "/home/fjuser/.conda/envs/ENetbylxb_py3.6/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1191, in train_on_batch
outputs = self._fit_function(ins) # pylint: disable=not-callable
File "/home/fjuser/.conda/envs/ENetbylxb_py3.6/lib/python3.6/site-packages/tensorflow/python/keras/backend.py", line 3076, in __call__
run_metadata=self.run_metadata)
File "/home/fjuser/.conda/envs/ENetbylxb_py3.6/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1439, in __call__
run_metadata_ptr)
File "/home/fjuser/.conda/envs/ENetbylxb_py3.6/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InternalError: cudnn PoolForward launch failed
[[{{node max_pooling2d/MaxPool}}]]
[[{{node loss/softmax_loss/broadcast_weights/assert_broadcastable/is_valid_shape/has_valid_nonscalar_shape/has_invalid_dims/concat}}]]
and my cuda is:
cudatoolkit 10.0.130 0 anaconda
cudnn 7.6.0 cuda10.0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
tensorflow’s version is:
tensorflow-estimator 1.13.0 pypi_0 pypi
tensorflow-gpu 1.13.2 pypi_0 pypi