I need to train a neural network on large image resolution (>= 2000 * 2000). The problem is with a A100 40Go I can only train in 900 * 900 resolution. Even with 4 * A100 40 Go, each GPU needs to support 1 batch. So I am always limited with 900 * 900 resolution.
How I can train on large image resolution without crop / resize or patch my dataset ?
NVlink could do this ? Or horovold ? Or the maximum resolution is limited by the current GPU RAM, like A100 80Go ? We can’t make a 1 batch fit more 80Go rigth now ?
Thanks for your response.