For nvidia2nvidia PRIME, you don’t need the modesetting driver, just driver 470+, see
https://forums.developer.nvidia.com/t/ubuntu-20-04-not-able-to-run-6-7-monitor-setup-with-one-x-screen/200904/2?u=generix
though in that case, pci2pci transfers are involved instead of pci2sysmem in the intel case so iommu device isolation (needed for the passthrough on the host) might interfere.