Train on large image resolution with multi-gpu

AI & Data Science Deep Learning (Training & Inference) Frameworks
1

Hello,

I need to train a neural network on large image resolution (>= 2000 * 2000). The problem is with a A100 40Go I can only train in 900 * 900 resolution. Even with 4 * A100 40 Go, each GPU needs to support 1 batch. So I am always limited with 900 * 900 resolution.

How I can train on large image resolution without crop / resize or patch my dataset ?

NVlink could do this ? Or horovold ? Or the maximum resolution is limited by the current GPU RAM, like A100 80Go ? We can’t make a 1 batch fit more 80Go rigth now ?

Thanks for your response.

2

Training is often done at the lower resolutions - can you share a little more information about your project, and the dataset that you are using and what is driving the requirement for the high resolution - and any other information you think may help us - help you. If the reason is that your data set is high-res - then a pre-processing transformation step may be an option.
Many Thanks,

3

We are using the SPADE framework from NVidia, and we need to generate high resolution images. Pre-processing will reduce the quality of the generated image.

Create a patch also. The generator can’t make a spatial consistency image.

For that, we are looking for the best way to create 1 batch with the maximum resolution. Are we limited to the RAM on 1 GPU ? Are we limited to 80Go (A100 80Go) ?

Thanks for your reply.

Best regards,