I’m looking the best value of max_vfs on SR-IOV configuration of virtualization host. I find following statement on ConnectX-5 EN brochure.
SR-IOV: Up to 1K Virtual Functions
SR-IOV: Up to 16 Physical Functions per host
Let’s say that the virtualization host is equipped with total of 40 cores (80 threads). Should I use 1024 or 64 (1024/16) for max_vfs? If not, what is the optimal one for that environment? How to utilize all ConnectX-5 EN VF? How to utilize all ConnectX-5 EN PF?
Let’s say that I’m using latest Centos 7 (currently v7.5 April 2018 built). What is the best number PF and VF should I use and why? Any pros and cons consideration for having too many or little number of PF and VF?
Regarding the number as mentioned on the firmware of ConnectX-5 EN
The maximum Virtual Functions (VF) per port is 64. Note: When increasing the number of VFs, the following limitations must be taken into consideration: server_total_bar_size >= (num_pfs)(2log_pf_uar_bar_- size + 2log_vf_uar_bar_sizetotal_vfs) server_total_msix >= (num_pfs)*(num_pf_msix + num_vfs_msix *total_vfs)
Note: For the maximum number of VFs supported by your driver, please refer to your drivers’ Release Notes or User Manual.
You can refer to the MLNX_EN driver as well :
I suggest to open a support case with Mellanox at firstname.lastname@example.org mailto:email@example.com asking for recommended performance tuning for such scenario.
Please note that such question is related to the firmware and OS capabilities,
You can refer to the latest firmware of ConnectX-5 EN release notes page 21 :
The current firmware support:
Reduced firmware’s memory consumption to increase the supported number of VFs per PF to up to 100.
There is not optimal recommendation, it always depends on the Server type capabilities with SR-IOV and the firmware of the adapter.
It seem that document valid for VPI cards, we talk about EN cards in this discussion. Even if it is valid for EN cards, I only find the compatibility related matter regarding to the number of configured PF and VF. I still need the performance related matter of that configuration.
Do you mean that following statement on page 3 of http://www.mellanox.com/related-docs/prod_adapter_cards/PB_ConnectX-5_EN_Card.pdf http://www.mellanox.com/related-docs/prod_adapter_cards/PB_ConnectX-5_EN_Card.pdf is not achievable?
– SR-IOV: Up to 1K Virtual Functions
– SR-IOV: Up to 16 Physical Functions per host
If we do some calculation, above statement seem related to 64 VFs per PF (1K/16). Is it true?
Let’s say it is true and there is only 1 PF I can be used and compatible with OS, should I use 64 VFs configuration for performance? How is about using lower VF numbers (32, 16, 8, 4, 2 or other) regarding to performance? When should I use small number, when should I use big number?