As per driver 415.22, no news. What makes it worse is that those 4% are probably observed while the gpu is at P0. While idling at P8, the gpu load is up to tenfold, 40-50%. Proportional to pixel count.
@aplattner found the cause (Force Composition Pipeline in nvidia-settings was turned on). Turning this off resolved the behaviour. Was not happening until I did a major upgrade from Ubuntu 22.04 to 23.10.
I attached the nvidia-bug-report.log generated while the issue was active to Bug ID: 4341092
Thanks very much - thought my system was haunted - scary, ooo!