Very High Xorg CPU usage using latest Drivers with an RTX 3080

After recently acquiring an RTX 3080, I switched to the Nvidia proprietary driver. The immediate thing that I noticed was the abnormally high Xorg CPU usage (around 15-30% idle, 70-80% when moving/resizing windows as reported by various utilities like htop), which of course doesn’t happen on Windows.
My immediate thought was that something was wrong with my specific setup, so I booted a fresh PopOS live environment with the latest Nvidia driver, but that didn’t fix it, Xorg was still reporting a really high CPU usage. Here are some other mitigation steps I took:

Drivers tried so far:

  • 465.31 (New Features Branch)
  • 460.84 (Production Branch)

Things tried so far:

  • Using Force Composition Pipeline/Force Full Composition Pipeline
  • Various nvidia-settings settings/setups, with or without multiple screens
  • Different compositor settings, with and without XRender backend
  • Different Desktop Environement/Window Manager, Like Gnome, KDE, XFCE and almost bare bone Xorg setup launched with startx and with a minimal window manager like OpenBox
  • 3 Different Linux based distro (Arch Linux, Void Linux (Clean Install)l, PopOS (Clean Install))
  • Drivers from both the distro package manager and the standalone nvidia installer

This issue seems to be related to my specific GPU (an RTX 3080 FE), since after trying on someone else machine equipped with an RTX 2070 Super and the 460.84 drivers, the Xorg usage doesn’t goes higher than around 5%.

I have also included here a bug report, ran with startx -- -logverbose 6 from a fresh Void Linux install

nvidia-bug-report.log.gz (272.8 KB)

Thanks in advance,