I need to have more GPU memory for my deep learning projects. I currently have multiple GTX 1080Ti with 11GB RAMs. But, my training data load has already exceeded 11GB GPU RAM so I need to find a way to extend GPU RAMs. I have been wondering that NVLink can combine and accumulate CUDA cores and GPU RAMs. Can NVLink combine 2x GPUs and increase a number of cuda cores and RAM size as 1x GPU?
It won’t behave exactly as if it were 1 GPU. For example, a CUDA kernel launched on one GPU won’t automatically “use the CUDA cores” of the other GPU.
However, scaling of DL training doesn’t really happen this way anyway. If you are using a DL framework like Tensorflow, training scaling understands that there are multiple GPUs and knows how to use them.
In this context, the NVLink bridge would help this sort of distributed training on 2 GPUs.