Implement MMIO Filer for NVSwitch - A100 HGX 8GPU System

Accelerated Computing DGX User Forum
1

For full passthrough virtualization … i could see the hypervisor needs to implement MMIO filtering for NVSwithc in fabric manager manual for other requiremnts.

Please let me know how to implement it ?

Thanks

2

Hi @booyeonjeong , although DGX A100 doesn’t support virtualization, I understand you’re asking more generically about HGX A100 based systems. Your best path would be to contact the NVIDIA Solutions Architect (SA) or Field Application Engineer (FAE) that you’ve been working with directly, so we can help you.

ScottE

3

I’m unable to find anything about MMIO filtering, which is necessary if you want as much isolation as possible. Is this only possible with certain hypervisors? Additional info is needed.

4

As ScottEllis mentioned, It would be better to contact your system supplier … then NVIDIA FAE can help how to block disabled NVLink connection on each GPU and Switch in full passthrough virtualization.

5

We encounter a GPU passthrough problem that p2p links are disonnected and in NS state. Maybe that is the problem you are to solve. A simple solution is an undocumented option for FM:
DISABLE_DEGRADED_MODE=1
Add it to /usr/share/nvidia/nvswitch/fabricmanager.cfg and restart FM service to see if passed through GPU are connected by
nvidia-smi topo -p2p n
If it works, the interconnect state should be OK.