I am experiencing slow training of neural networks when training on the GPU. I know this because I have another, inferior, hardware to compare this to.
The inferior hardware is a laptop (Lenovo X1 Carbon) connected to an eGPU containing a 1080TI. The superior hardware is a desktop with an Intel i9-10940x processor with the ASUS PRO WS X299 SAGE II motherboard, which contains 3 GPUs – two 2080TIs and one 1080TI.
When training on the desktop, I am getting 2-4x slower speed as compared to training on the laptop+eGPU. I used nvidia-smi to make sure the GPUs were being used and they were.
The software for both systems is the same – Ubuntu 18.04, and I installed PyTorch using Anaconda3 in exactly the same way. This gives me an conda environment with PyTorch 1.7.1, CUDA 11, and cuDNN 8.0.5. I also tested this on Tensorflow code and got the same result – the desktop training was much slower.
Any help would be much appreciated! Thank you!
Edit: Here is the Tensorflow code I’m using to test this:
import tensorflow as tf from tensorflow import keras from tensorflow.keras import layers import time model = keras.Sequential( [ layers.Dense(2, activation="relu", name="layer1"), layers.Dense(3, activation="relu", name="layer2"), layers.Dense(1, name="layer3"), ] ) inp = tf.ones((100000, 3)) labels = tf.zeros((100000, 1)) model.compile( optimizer=keras.optimizers.RMSprop(), loss=keras.losses.MeanSquaredError(), metrics=[keras.metrics.MeanSquaredError()], ) tic = time.perf_counter() model.fit( inp, labels, batch_size=1000, epochs=int(500) ) toc = time.perf_counter()