Technically the 1080 (or any GeForce) as a passthrough device is not supported, however there are flags that can be passed to certain hypervisors to disable checks done by the driver to enable the card to work under the guest OS. Not sure if you can passthrough more than 1 physical device in your hypervisor implementation, so I cannot answer the rest of that question fully.
In order to support any NVIDIA card on bare metal, yes, drivers need to be installed. Unfortunately I don’t have first hand experience of this in Linux. I do know that in Windows it works perfectly fine, so theoretically it should be possible in Linux.
If I had to take a guess (perhaps someone can correct me) it might be enough to run the installer as explained here:
which does not replace existing OpenGL libraries (yes I know optimus is not the same as 2 GPUs, but similar enough)
Full readme is here: ftp://download.nvidia.com/XFree86/Linux-x86_64/375.39/README/index.html
I am also assuming you are running the latest closed-source AMD driver to run OpenCL code, in which case you are halfway there:
Confirmation/Edit: Link here:
says it is possible, and indeed the advice is the same as above – to the NVIDIA runfile with the --no-opengl-files option.
I believe the same can be accomplished if installing nvidia drivers via Ubuntu’s package manager, (apt-get install nvidia-xxx) and afterwards using update-alternatives to pick a GL provider, but I am unsure if this will break your AMD drivers installation.
If it will work, It should be able to be accomplished by using (ONLY) step 7 of this page:
That being said, I am not certain that installing via apt-get and doing the above will make it work – I can only confirm that at least 2 other people had success installing the native AMD driver and installing the NVIDIA driver .run file afterwards with the --no-opengl-files options, but felt I should mention another possible (yet, untested) way.