I need some guidance / pointers on how to accomplish an OVS-DOCA / vDPA setup I need to implement.
We are using host systems with ConnectX-6 Dx NICs. These systems run Rocky 9 Linux. We need to host around 200 VMs on this server.
We’d like to take advantage of as much hardware offloading that the CX6 can do for us. Going through all of the extensive documentation it appears DOCA-Networking profile contains everything we need (ovs, dpdk, vDPA and other driver components). We’ve installed all of this successfully and things appear to be happy.
What is not clear to me is exactly how this would be configured. The documentation seems to discuss vDPA in such a way that we enable SR-IOV VFs, create vDPA interfaces on them, and pass a vDPA interface, per vm, into that vm where it appears as a virtio device. OVS-DOCA then seems to know how to handle the data plane offload.
Seems straight forward except that we need 200 VMs and, AFAIK, 127 is the absolute max for VFs (on the nic) and there are likely other system (PCI related) limits as well.
We do not need the absolute performance of passing VFs into each VM. But we would like to benefit from as much data plane offload to the card as we can as well as vxlan and, hopefully, LACP offloading (link from the CX6 to our switch). We do use live migration as well, so this another constraint about passing VFs into the VMs.
So my questions… is what I described even possible? And, if so, can someone point me in the right direction as to how this might be accomplished? I am tempted to think it may be as simple as creating a vDPA interface and adding that to OVS (rather than passing into the VM) and, presumably, OVS-DOCA understands this and will be able to offload the ovs dataplane as well as do the vxlan / LACP offloads?
Any pointers at all are most appreciated. If someone has an example config, that would be to die for! :)
Thanks!