Tensorflow1.14 is not working with RTX3090 inside the Docker container of CUDA10.0.
I created a program for reinforcement learning, and it works in the Docker container I created on GTX1080Ti.
I move on to a new PC with RTX3090, but it doesn’t work.
The program takes 20 minutes to start itself then error codes are shown.
On GTX1080Ti PC, it works without waiting a long time.
Note that some CUDA10.0 samples work on RTX3090 PC inside the docker container.
The error codes as follows
2020-12-01 08:12:06.722138: E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED
2020-12-01 08:12:06.722611: I tensorflow/stream_executor/stream.cc:4838] [stream=0x555d1dba6960,impl=0x555d1dd5fe80] did not memzero GPU location; source: 0x7f1872ffbd20
2020-12-01 08:12:06.722627: I tensorflow/stream_executor/stream.cc:315] did not allocate timer: 0x7f1872ffbd30
2020-12-01 08:12:06.722633: I tensorflow/stream_executor/stream.cc:1839] [stream=0x555d1dba6960,impl=0x555d1dd5fe80] did not enqueue ‘start timer’: 0x7f1872ffbd30
2020-12-01 08:12:06.722642: I tensorflow/stream_executor/stream.cc:1851] [stream=0x555d1dba6960,impl=0x555d1dd5fe80] did not enqueue ‘stop timer’: 0x7f1872ffbd30
2020-12-01 08:12:06.722651: F tensorflow/stream_executor/gpu/gpu_timer.cc:65] Check failed: start_event_ != nullptr && stop_event_ != nullptr
Aborted (core dumped)
GPU Type: GTX1080TI(works), RTX3090(doesn’t work)
Nvidia Driver Version: 418.40.04(1080Ti PC), 455.45.01(3090 PC)
CUDA Version:10.1(1080Ti PC host), 11.1(3090 PC host), 10.0(docker)
CUDNN Version: 7
Operating System + Version: Ubuntu18.04(1080Ti PC), Ubuntu 20.04(3090 PC), Ubuntu 18.04(docker)
Python Version (if applicable): 2.7
TensorFlow Version (if applicable): 1.14
Baremetal or Container (if container which image + tag): I used it as a base image; nvidia/cuda:10.0-cudnn7-devel-ubuntu18.04