Activating GPU scaling leads to touch input being scaled as well

Windows 10 computer with 3 monitors attached: one touch screen with native resolution 1280x1024 and 2 identical monitors with native resolution 1920x1080. The touch screen is set as Clone Source in the Nvidia Control Panel, and the 2 other monitors are cloned with it. The 2 monitors should display the exact same image as the touch screen (not considering differences in hardware pixel size), i.e they should display the 1280x1024 image centered with black bands around it to fill the rest of the screen. But the monitors do not have a ‘non-scaling’ mode where they just center the resolution which comes in so using the Nvidia Control Panel they are set to no scaling, and perfoming scaling on the GPU.

Visually this does exactly what it should do, but somehow it makes touch input on the touch screen unusable since that input gets scaled as well, likely by a factor 1920/1280 horizontally and 1080/1024 vertically though I did not measure it exactly. What happens is that touching the center of the screen is ok, but the further away one taps from the center, the further the tap gets registered, likely such that when I tap the first pixel, it acts as if I’m tapping the first pixel on a screen which would be 1920 pixels wide instead of 1280. This should make it clear (done in Paint for simplicity, but every other application behaves the same):

The black points are drawn with the mouse and number 1 is placed on the physical center of the screen i.e. around pixel 640x512. The red points are the positions where touch input was registered when touching those black points on the touch screen (I offsetted the red points a bit vertically so they are easier to see). Vertically a similar scaling occurs but it’s much less obvious, undoubtedly because vertically the difference between the 2 resolutions is small. Turning off GPU scaling makes the problem go away (but makes the 2 monitors display the original image stretched out so not usable for our case), so does turning off the 2 non-touch monitors.

What would cause this? Seems like a mix between what the NVidia driver is doing and something in Windows getting changed. Is there a way to avoid this?