cuDNN failed to initialize

I am trying to train VGGNET model on a custom dataset, I have experimented with the model on colab but now I need to run the same code on local machine.

I tried to run the code with Tensorflow GPU 1.15 and CUDA 10.0 (colab I am using has Tensorflow GPU 1.15 and CUDA 10.0), but the code is giving this error

...
  (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[{{node block1_conv1/convolution}}]]
	 [[loss/mul/_169]]
...

Things I tried

  • Tried different version of cuDNN 7.6.4 and 7.4.2
  • Tried downgrading Tensorflow GPU to 1.14 from 1.15

OS: Ubuntu 18.04
Libraries: CUDA 10.0, cuDNN 7.4.2, Nvidia driver 440.64, Tensorflow GPU 1.15

Seems like there were some issues with RTX Cards, I had to add few lines of code to make it work

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
tf.Session(config=config)