Virtual Machines using separate GPU's

Hi, I’ve got a machine with 4 (Titan X) GPUs connected in SLI running Pop! OS 19.04; and I’ve installed NVIDIA proprietary driver 430.34.

I’m trying configure multiple virtual machines that can each utilize one of the GPU’s. So far I’ve got GPU 0 outputting correctly as the primary monitor and GPU 1 connected to the second screen. I can move my mouse to the second screen, but the mouse turns to an “X” and I’m not able to drag any windows to it or get the OS to recognize it otherwise.

I also have VirtualBox configured to run another instance of Pop!.

I’m not sure about what to do next. Is the best next step to setup PCI passthrough to get each VM it’s own dedicated GPU?