glFinish stall when using a lot of texture memory (caused by driver freeing up mem?)

Here’s a quick problem description:

I am developing an application which uses a lot of GPU memory for texture data.
I have a set of texture arrays (say 20 ) that I update each frame with data located in a single big fat PBO ( say around 1.5GB for some cases, could be 6GB for other cases).
I’m binding these textures to a moderately complex shader for rendering.
The texture data are typically RGTC2 and S3TC_DXT1.
I’m using Nvidia 970 GTX, driver 372.54

Now here’s the problem:

If I use a dataset larger than some size (e.g. 1.5GB), then occasionally I get a stall in the glFinish() part of SDL_GL_SwapWindow (I’ve verified that by adding glFinish outside of the SwapWindow call, and the CPU stall happens there)

If I disable PBO->texture updates, the stall still happens
If I disable setting the RGTC2 texture as a uniform in the shader the stall goes away
If I disable setting the S3TC_DXT1 texture as a uniform in the shader , the stall still happens
It does not matter how much I use the texture in the shader – a single fetch suffices to cause a stall

And the funny bit:
I’ve added out of curiosity some memory reporting code that I found for Nvidia, using GL_GPU_MEM_INFO_CURRENT_AVAILABLE_MEM_NVX
Immediately after the stall, the reported available GPU memory is 300MB more (using one of the datasets), so the stall looks like it’s caused by the driver freeing up memory

Does anybody have any idea about why and how that could happen, or even better, how to prevent such a stall?

Thanks.

Why are you calling glFinish ?